@pessimisttech interesting development — before you chase the interrupt angle any further, here's the same grep from my working rig (same GPU, xcp-ng 8.3, 38 t/s inference):
92: 0 0 0 0 0 0 0 0 xen-pirq -msi amdgpu
Zero. My card has never received an interrupt either. So scratch next thoughts on chasing ARI and nopv — zero GPU interrupts appears to be the normal (if ugly) state for these cards under 8.3 HVM passthrough, and everything runs on polling instead: ROCm compute completion is handled in userspace via HSA signals, and the kernel fences ride the fallback timer. That's why I see the same "Fence fallback timer expired" spam you do, permanently and harmlessly.
The difference between our rigs is in what happens next. On mine, the fences complete and polling just notices them late. On yours, they genuinely never complete — your dmesg shows signaled seq=52, emitted seq=54 and a failed ring reset before the MODE1 reset wipes VRAM (that "VRAM is lost" line is your model evaporating while llama-server waits). Your SDMA engine is actually stalling mid-transfer, which is a different beast from a missed notification.
With interrupts eliminated as the difference, the biggest remaining gap between our setups is the guest software stack, and it's a big one. I'm on Slowroll's stock 7.0.12 kernel with the in-tree amdgpu driver and current firmware. You're on Ubuntu's 6.8.0-134 — a kernel base from before this card existed — with AMD's DKMS 6.16.13 driver layered on top, and your boot log shows the seams: "SMU driver if version not matched" and a MES firmware version workaround note. That's roughly two years of RDNA4 enablement between our guests. I'd suspect that long before the hardware or the hypervisor.
Two ways to test it cheaply:
Close the gap in place: update the amdgpu firmware in your guest to current (the /lib/firmware/amdgpu files in a 6.8-era package predate this card's tuning) and move to the newest HWE kernel available. Fair warning that Ubuntu's HWE channel seems to top out well short of what I'm running, so this narrows the gap rather than closes it — but firmware alone might be enough if that's where the stall lives.
Cleaner and probably faster: spin up a throwaway openSUSE Slowroll (or Tumbleweed) VM, attach one GPU, install docker + the same lemonade container, and run the same load. That's my exact known-good recipe, and it's a proper controlled test — same host, same BIOS, same GPU, same container, only the guest stack changes. If it works there, you've found your answer and can decide whether to migrate or keep fighting Ubuntu. If it still fails on Slowroll, then it's genuinely host-side and we've got a much sharper bug report for the Vates folks.
@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.
R
