RDNA 4 GPU Passthrough

[ravenet]

@PessimistTech said:

@ravenet
Thanks for the ideas!

Here's the outcome of the tests
ps -eo pid,stat,wchan:32,cmd | grep llama:

62 Rl   -                                /root/.cache/lemonade/bin/llamacpp/rocm-nightly/llama-server -m /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --ctx-size 262144 --port 8001 --jinja --metrics --mmproj /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/mmproj-F16.gguf --reasoning-format auto --no-webui --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 --chat-template-kwargs {"preserve_thinking":true}

sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence":
dmesgout.txt

Same overall results... I even re-ran the same tests after removing the 2nd GPU from the PCI passthrough settings and still the same...

I also seem to run into an issue where the GPU tends to get stuck after some of these tests. Even rebooting gets stuck waiting for llama server processes to stop and I end up having to force power off. Not sure if that is related as I get the same output on the first test, but it may be worth noting.

found it — or at least found a very large problem. Your kernel command line has amdgpu.msi=0:

Command line: BOOT_IMAGE=/vmlinuz-6.8.0-134-generic root=... ro amdgpu.msi=0 console=tty1 console=ttyS0

That disables MSI interrupts for the GPU driver entirely, and your dmesg shows exactly what that costs: fence fallback timer messages every half-second on both cards from the moment the driver loads, before any workload runs. The driver is surviving on a polling fallback instead of interrupts. My card throws those messages in occasional bursts under load (lost interrupts); yours throws them constantly (no interrupts). Under load that degrades into what's in your log: sdma0 ring timeout → ring reset fails → full MODE1 GPU reset → "VRAM is lost" → your loaded model evaporates while llama-server keeps waiting. It also explains your stuck reboots — the traces at the end show unkillable fence waits in the reset path.
Remove amdgpu.msi=0 from grub, update-grub, reboot the VM, then verify interrupts are actually flowing:

grep -i amdgpu /proc/interrupts

You should see MSI/MSI-X vectors for the amdgpu devices with counts that climb while a model loads. Then watch dmesg during a load — occasional fence fallback bursts are survivable (I get them too under Xen), but the constant 0.5s drumbeat should be gone.

Out of curiosity — did you add msi=0 while chasing the "no interrupts registered" issue from your first post? If you re-enable MSI and interrupts still don't flow, that's the real bug and exactly what the Vates folks will want to see, especially since you still have the debug Xen + iommu=debug boot — xl dmesg from dom0 while the VM is loading a model would show whether MSIs are being injected.

One more thing for after interrupts are fixed: your llama-server command shows --ctx-size 262144. A 256K context KV cache on top of that model is a huge allocation for a 32GB card and can silently spill into GTT and crawl. I'd validate at something modest first (8192), confirm it serves, then step the context up while watching VRAM. That's the approach I'm using on mine.

R

[PessimistTech]

@ravenet ooh, thanks for the callout, that is a config that is left over from some earlier debugging. I'll remove that and try again.

I had a similar thought about the model size of the qwen 3.6 model (chosen intially for running on both GPUs). I also tested with llama3.2 for a much smaller test.

[PessimistTech]

@ravenet welp, removed the amdgpu.msi=0 option, and I am seeing 0 interrupts in the /proc/interrupts file, and overall behavior is still the same...

106:          0          0          0          0          0          0  xen-pirq    -msi       amdgpu

[PessimistTech]

for future reference here is the xl dmesg at this point
xldmesg.txt

[ravenet]

@pessimisttech interesting development — before you chase the interrupt angle any further, here's the same grep from my working rig (same GPU, xcp-ng 8.3, 38 t/s inference):

 92:          0          0          0          0          0          0          0          0 xen-pirq    -msi       amdgpu

Zero. My card has never received an interrupt either. So scratch next thoughts on chasing ARI and nopv — zero GPU interrupts appears to be the normal (if ugly) state for these cards under 8.3 HVM passthrough, and everything runs on polling instead: ROCm compute completion is handled in userspace via HSA signals, and the kernel fences ride the fallback timer. That's why I see the same "Fence fallback timer expired" spam you do, permanently and harmlessly.

The difference between our rigs is in what happens next. On mine, the fences complete and polling just notices them late. On yours, they genuinely never complete — your dmesg shows signaled seq=52, emitted seq=54 and a failed ring reset before the MODE1 reset wipes VRAM (that "VRAM is lost" line is your model evaporating while llama-server waits). Your SDMA engine is actually stalling mid-transfer, which is a different beast from a missed notification.

With interrupts eliminated as the difference, the biggest remaining gap between our setups is the guest software stack, and it's a big one. I'm on Slowroll's stock 7.0.12 kernel with the in-tree amdgpu driver and current firmware. You're on Ubuntu's 6.8.0-134 — a kernel base from before this card existed — with AMD's DKMS 6.16.13 driver layered on top, and your boot log shows the seams: "SMU driver if version not matched" and a MES firmware version workaround note. That's roughly two years of RDNA4 enablement between our guests. I'd suspect that long before the hardware or the hypervisor.
Two ways to test it cheaply:

Close the gap in place: update the amdgpu firmware in your guest to current (the /lib/firmware/amdgpu files in a 6.8-era package predate this card's tuning) and move to the newest HWE kernel available. Fair warning that Ubuntu's HWE channel seems to top out well short of what I'm running, so this narrows the gap rather than closes it — but firmware alone might be enough if that's where the stall lives.

Cleaner and probably faster: spin up a throwaway openSUSE Slowroll (or Tumbleweed) VM, attach one GPU, install docker + the same lemonade container, and run the same load. That's my exact known-good recipe, and it's a proper controlled test — same host, same BIOS, same GPU, same container, only the guest stack changes. If it works there, you've found your answer and can decide whether to migrate or keep fighting Ubuntu. If it still fails on Slowroll, then it's genuinely host-side and we've got a much sharper bug report for the Vates folks.

@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

R

[TeddyAstie]

@ravenet said:
@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

TL;DR This is likely a known issue, and we're currently discussing on a potential fix.

There is a known bug regarding MSI on AMDGPU when using "hvm_pirq" ("use event channels for device interrupts"), see
https://lore.kernel.org/xen-devel/20240110095311.61809-1-roger.pau@citrix.com/
https://github.com/QubesOS/qubes-issues/issues/7971

The easy fix is to disable "HVM PIRQ" feature (which has been made disabled by default in Xen 4.19), but on the XCP-ng side, this is configured by XAPI Project which needs to be updated as well to apply this change (which appears to be a bit tricky for various reasons).

neowutran created this issue in QubesOS/qubes-issues

closed MSI problems with `amdgpu` driver on recent kernels #7971

[PessimistTech]

@TeddyAstie Is this specific to the lack of interrupts?

[PessimistTech]

@ravenet well... thanks for the idea. I went ahead and tried spinning up an opensuse slowroll VM. I installed docker and the amdgpu firmware/kernel drivers, and tried running lemonade-server and ollama and same behavior. So that definitely seems to point to something weird going on on my host rather than the guest software stack...

[ravenet]

@PessimistTech You can try disabling ARI in your bios as a quick next step. I'm out for a few days so will attempt to assist further when I'm back if issues continue

[PessimistTech]

@ravenet Sure, gave that a whirl and no change, though I did notice some nvtop weirdness as it would show some load, but most of the wait time there was actually no load on the GPUs instead of the constant load matching the model being loaded. For some historical context ARI support was initially disabled in the bios when I started this thread. That was on the list of things I enabled when I started seeing some success with ollama (something in the changes since has broken ollama now too, but there was at least some forward progress after enabling). dmesg output overall looked the same, but I did see this output on the console (and in dmesg) that seemed interesting. Not 100% sure at this point if this was in the previous dmesg outputs or not, but may be worth sharing. EDIT: looks like this may actually be new... I looked back through the past dm dmesg outputs and I did not see this output.

[  108.547683] amdgpu 0000:00:09.0: MES(0) failed to respond to msg=REMOVE_QUEUE
[  108.547729] amdgpu 0000:00:09.0: failed to remove hardware queue from MES, doorbell=0x1202
[  108.547746] amdgpu 0000:00:09.0: MES might be in unrecoverable state, issue a GPU reset
[  108.547774] amdgpu 0000:00:09.0: Failed to evict queue 2
[  108.547789] amdgpu 0000:00:09.0: Failed to evict process queues
[  108.547803] amdgpu: Failed to quiesce KFD
[  108.547870] amdgpu 0000:00:09.0: GPU reset begin!. Source:  3
[  109.656850] amdgpu 0000:00:09.0: Failed to remove queue 0
[  109.657324] amdgpu 0000:00:09.0: Dumping IP State
[  109.756265] amdgpu 0000:00:09.0: Dumping IP State Completed
[  112.047695] amdgpu 0000:00:09.0: MODE1 reset
[  112.047797] amdgpu 0000:00:09.0: GPU mode1 reset
[  112.054505] amdgpu 0000:00:09.0: GPU smu mode1 reset
[  113.075393] amdgpu 0000:00:09.0: GPU reset succeeded, trying to resume
[  113.090354] amdgpu 0000:00:09.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000).
[  113.092905] amdgpu 0000:00:09.0: [drm] AMDGPU device coredump file has been created
[  113.092913] amdgpu 0000:00:09.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[  113.092917] amdgpu 0000:00:09.0: VRAM is lost due to GPU reset!
[  113.092921] amdgpu 0000:00:09.0: PSP is resuming...
[  114.997591] amdgpu 0000:00:09.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed
[  115.091125] amdgpu 0000:00:09.0: RAP: optional rap ta ucode is not available
[  115.091130] amdgpu 0000:00:09.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
[  115.091134] amdgpu 0000:00:09.0: SMU is resuming...
[  115.091375] amdgpu 0000:00:09.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0)
[  115.432371] amdgpu 0000:00:09.0: SMU is resumed successfully!
[  115.445556] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0x4000000
[  115.445731] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0xc000000
[  115.750240] amdgpu 0000:00:09.0: [drm] DMUB hardware initialized: version=0x0A000800

As an update for other things I have tried, in order to eliminate hardware issues, or bios settings I tried installing proxmox and spinning up a VM there with both GPUs passed through and it worked just fine... With that feedback I did a fresh install of XCP-NG 8.3 and spun up a fresh VM using the same steps as I used on proxmox and still no dice. This leads me to believe the issue is somewhere in the XCP-NG passthrough stack with my specific hardware...