RDNA 4 GPU Passthrough

PessimistTech

@yannsionneau said:

segment bus device function address of the device

Hi all, thanks for your reponses!

the below is done with the assumption that dom0 is just the host cli. If that is incorrect please let me know and I can update this info:

dom0: lspci -vvv -nn -s 03:00.0:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device [1849:5413]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 120
Region 0: Memory at 7f000000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at 7f800000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at 3000 [size=256]
Region 5: Memory at f2200000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at f2280000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed unknown, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [200 v1] #15
Capabilities: [240 v1] Power Budgeting <?>
Capabilities: [270 v1] #19
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [2d0 v1] Process Address Space ID (PASID)
PASIDCap: Exec+ Priv+, Max PASID Width: 10
PASIDCtl: Enable- Exec- Priv-
Capabilities: [320 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [410 v1] #26
Capabilities: [450 v1] #27
Capabilities: [500 v1] #2a
Kernel driver in use: pciback

VM lspci -vvv -nn -s 03:00.0:
00:08.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device [1849:5413]
Physical Slot: 8
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0
Interrupt: pin A routed to IRQ 17
Region 0: Memory at 800000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at c300 [size=256]
Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 32GT/s, Width x16
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn+
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver in use: amdgpu
Kernel modules: amdgpu

dom0 xl dmesg output: xldmesg.txt
dom0 dmesg output: dom0dmesg.txt
vm dmesg output: vmdmesg.txt

I believe this is all the output requested (I only provided the lspci output for a single GPU as they are identical GPUs , but happy to provide all the outputs if desired (will note I am not entirely sure I selected the exact GPU in both contexts due to the mismatched IDs in vm vs dom0, not sure that is an issue but may be worth calling out)

edit: all output was taken with the VM running (but at idle). I did apply the grub changes to use the -d.gz file and iommu=debug as well so ideally that output should be included)

ravenet

@PessimistTech

I am running a radeon ai Pro 9700 on xcp-ng 8.3 passed through to an opensuse Slowroll, with rocm and lemonade docker. Gemma 4 31B-it-MTP gets 38tps

did you pass through as pcie passthrough with both the video and audio, or just the 'gpu passthrough' option in xoa? I did the 2 devices pcie passthrough
Make sure you have Resizable BAR enabled in bios or it just will not work

R

PessimistTech

@ravenet glad to hear you were able to get it working!
Yes I passed through both the audio device and video device via PCI passthrough. I have also checked and ensured Resizeable BAR is enabled in bios.

Did you do anything to set it up directly on the host? Or did you just use the PCI device passthrough options in XO?

PessimistTech

Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.

EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

ravenet

@PessimistTech said:

Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.

EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

Do you have examples of workloads that are struggling still?
and no, didn't do anything special on the xcp host that I'm aware of. Was already using it to passthrough a radeon pro 7500 to a windows vm for revit clarity automation server. It's been a year so I'll triple check if anything special was done back then.

PessimistTech

@ravenet sure, the workloads I have been testing with include ollama, vLLM, and lemonade server (all via docker for easier setup, but I do have all the rocm/AMD drivers and what not setup on the VM directly). Ollama works just fine now, but lemonade server still hangs and vLLM fails with the same errors on startup.

Cool, thanks for info/checking!

ravenet

@PessimistTech make sure lemonade is set to use rocm as backend. It was slow and unusable from vulkan

docker compose exec lemonade lemonade config set llamacpp.backend=rocm

does it finish loading model then just respond horribly slowly, or does it not complete the model load?
did you setup as straight docker, or as a docker compose? I did mine as a compose, but shouldn't matter outside of how the config is laid out

is your guest seeing the whole mem on gpu and the PCIE link speed?

sudo lspci -vv -d 1002: | grep -E "Region [0-9]|LnkSta:"
        Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
        Region 2: Memory at 1800000000 (64-bit, prefetchable) [size=256M]
        Region 4: I/O ports at c100 [size=256]
        Region 5: Memory at f0800000 (32-bit, non-prefetchable) [size=512K]
                LnkSta: Speed 32GT/s, Width x16
        Region 0: Memory at f08c0000 (32-bit, non-prefetchable) [size=16K]
                LnkSta: Speed 32GT/s, Width x16]

You'll also want to make sure AtomicOP is working, below is example command and result

sudo lspci -vv -d 1002: | grep -iE "AtomicOp"

                         AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
                         AtomicOpsCtl: ReqEn-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                         AtomicOpsCtl: ReqEn-

PessimistTech

@ravenet I have not tried with the rocm backend yet, but I will give that a whirl. When using the GPUs it never completes the model load.

For testing I was just using the docker run command from the lemonade server docs.

docker run -d \
	--name lemonade-server \
	-p 13305:13305 \
	-v lemonade-cache:/root/.cache/huggingface \
	-v lemonade-llama:/opt/lemonade/llama \
	-v lemonade-recipe:/root/.cache/lemonade \
	--device=/dev/kfd \
	--device=/dev/dri \
	ghcr.io/lemonade-sdk/lemonade-server:latest

Command output:

lspci -vv -s 00:08.0 | grep -E "Region [0-9]|LnkSta:"
	Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
	Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at c300 [size=256]
	Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
		LnkSta:	Speed 32GT/s, Width x16

lspci -vv -s 00:08.0 | grep -iE AtomicOP
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
			 AtomicOpsCtl: ReqEn+

ravenet

@PessimistTech

switch to rocm-nightly, as that's required to bring support for our ai pro 9700

docker exec lemonade-server lemonade config set llamacpp.backend=rocm
docker exec lemonade-server lemonade config set rocm_channel=nightly
docker restart lemonade-server

The second line is important. This GPU is gfx1201 and the rocm-stable channel doesn't have HIP kernels for it yet. If you switch to rocm backend without setting the channel to nightly, it silently falls back to CPU with no error — you'll just see terrible speeds and an idle GPU. Ask me how I know.

Restart is needed for the backend change to take effect, and the config survives restarts since the docs' run command already mounts the lemonade-recipe volume where it lives.

One more thing I'd add to your docker run: the --init flag, right after docker run -d. Without an init process in the container, a failed/hung model load can leave the container wedged in D-state and you end up force-killing it. I keep it permanently in my compose (init: true) after learning this the hard way.

Good news is your lspci output looks healthy — Region 0 at 32G in the guest and AtomicOpsCtl ReqEn+ means your BAR and atomics are fine, which was the failure mode on my end originally. So this is likely just backend/kernel config, not the passthrough itself.

After the restart you can confirm it's actually on GPU by watching VRAM fill during model load:

watch -n1 'cat /sys/class/drm/card*/device/mem_info_vram_used'

If that stays flat while a model loads, you're on CPU fallback.

R

PessimistTech

@ravenet well, applied those updates and still the same results unfortunately...

I am definitely seeing load on the GPU, so it is able to start the process, lemonade server just sits waiting logging
2026-07-08 20:06:06.477 [Debug] (WrappedServer) Still waiting for llama-server...
while the GPUs are sitting there with the model in mem (or at least the correct amount of mem for the model size being consumed).

looks like the rocm backend is using version b1292 which does note the correct gfx1201 model in the release notes.

ravenet

@PessimistTech
Try pass through just 1 gpu. See where that gets you

Edit:
actually, possibly faster way to test the single-GPU theory without touching passthrough — just hide one card from ROCm:

docker run -d \
    --name lemonade-server \
    --init \
    -e HIP_VISIBLE_DEVICES=0 \
    -p 13305:13305 \
    -v lemonade-cache:/root/.cache/huggingface \
    -v lemonade-llama:/opt/lemonade/llama \
    -v lemonade-recipe:/root/.cache/lemonade \
    --device=/dev/kfd \
    --device=/dev/dri \
    ghcr.io/lemonade-sdk/lemonade-server:latest

If it loads and serves with one GPU visible, the problem is apparently cross-GPU peer-to-peer under Xen. With two passed-through devices, llama.cpp splits the model and the first cross-device sync tries P2P DMA through the virtual root complex — ROCm reports peer access as available but the transaction never completes, so the fence waits forever. Model in VRAM, llama-server alive but frozen. That would explain why it dies after the memory is allocated rather than during load. (Related: even on my single card I get bursts of sdma0 fence fallback messages under Xen from lost interrupts — harmless for me, but on a P2P path a lost fence is fatal.)

Two things to grab while it's hung, so we know for sure where it's stuck:


docker exec lemonade-server ps -eo pid,stat,wchan:32,cmd | grep llama

If llama-server shows state D with a wchan in amdgpu fence/ring wait, it's wedged in the kernel driver, not userspace. Also check guest dmesg during the hang for ring timeout / fence messages:

sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence"

If single-GPU works and you want both cards back, a couple of things to try before giving up on dual:

-e HSA_ENABLE_SDMA=0 — routes copies through shader blits instead of the SDMA engines; it's the standard workaround for ROCm fence hangs inside VMs
keep the split on one device via lemonade: docker exec lemonade-server lemonade config set llamacpp.args="--split-mode none" then restart — both GPUs visible but no cross-device traffic

Worst case, two VMs with one GPU each works around the P2P problem entirely and you can load-balance in front.

Fingers crossed

R

ravenet

@pessimisttech one thing I realized we never actually confirmed — we've only seen BAR info for one of your cards (you ran -s 00:08.0). Can you run it against all AMD devices in the guest:

sudo lspci -vv -d 1002: | grep -E "Region 0|LnkSta:"

and check that both GPUs show Region 0 at [size=32G]?

Reason I ask: hvmloader sizes the guest MMIO space automatically, and it clearly handles one 32GB BAR fine (both our single-card outputs prove that). But two cards need 64GB+ of BAR space and I've never seen anyone confirm the auto-sizing copes with that. If your second GPU got squeezed down to 256M, that's the exact failure mode I originally had on my single card before enabling Resizable BAR — driver wedges in D-state with no error, model appears to load, llama-server never comes up. It would also explain the "some workloads fine, some not" pattern: anything that only touches GPU1 (like ollama, which apparently defaults to a single card when the model fits) works, anything that splits across both hangs.

If GPU2 does show a small Region 0, there's a VM param worth trying:

xe vm-param-set uuid=<vm-uuid> platform:mmio_hole_size=137438953472

(128GB, needs the VM fully shut down and cold booted, not rebooted). Fair warning that I haven't tested this myself — my single card worked with host BIOS ReBAR alone and hvmloader did the rest — so treat it as an experiment. It's also historically a below-4GB hole setting and these BARs live in high address space, so it may turn out to be the wrong knob entirely. If it doesn't change anything, that's useful info too.

If both GPUs already show 32G, then BAR sizing is ruled out and it's back to the cross-GPU angle — in which case the single-GPU test result (HIP_VISIBLE_DEVICES=0) plus the ps/wchan output while hung would be the thing to look at, and probably worth summoning @teddyastie with the findings since P2P DMA between two passthrough devices under Xen is hypervisor territory, not config.

R

PessimistTech

@ravenet
Thanks for the ideas!

Here's the outcome of the tests
ps -eo pid,stat,wchan:32,cmd | grep llama:

62 Rl   -                                /root/.cache/lemonade/bin/llamacpp/rocm-nightly/llama-server -m /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --ctx-size 262144 --port 8001 --jinja --metrics --mmproj /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/mmproj-F16.gguf --reasoning-format auto --no-webui --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 --chat-template-kwargs {"preserve_thinking":true}

sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence":
dmesgout.txt

Same overall results... I even re-ran the same tests after removing the 2nd GPU from the PCI passthrough settings and still the same...

I also seem to run into an issue where the GPU tends to get stuck after some of these tests. Even rebooting gets stuck waiting for llama server processes to stop and I end up having to force power off. Not sure if that is related as I get the same output on the first test, but it may be worth noting.

ravenet

@PessimistTech said:

@ravenet
Thanks for the ideas!

Here's the outcome of the tests
ps -eo pid,stat,wchan:32,cmd | grep llama:
62 Rl   -                                /root/.cache/lemonade/bin/llamacpp/rocm-nightly/llama-server -m /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --ctx-size 262144 --port 8001 --jinja --metrics --mmproj /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/mmproj-F16.gguf --reasoning-format auto --no-webui --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 --chat-template-kwargs {"preserve_thinking":true}
sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence":
dmesgout.txt

Same overall results... I even re-ran the same tests after removing the 2nd GPU from the PCI passthrough settings and still the same...

I also seem to run into an issue where the GPU tends to get stuck after some of these tests. Even rebooting gets stuck waiting for llama server processes to stop and I end up having to force power off. Not sure if that is related as I get the same output on the first test, but it may be worth noting.

found it — or at least found a very large problem. Your kernel command line has amdgpu.msi=0:

Command line: BOOT_IMAGE=/vmlinuz-6.8.0-134-generic root=... ro amdgpu.msi=0 console=tty1 console=ttyS0

That disables MSI interrupts for the GPU driver entirely, and your dmesg shows exactly what that costs: fence fallback timer messages every half-second on both cards from the moment the driver loads, before any workload runs. The driver is surviving on a polling fallback instead of interrupts. My card throws those messages in occasional bursts under load (lost interrupts); yours throws them constantly (no interrupts). Under load that degrades into what's in your log: sdma0 ring timeout → ring reset fails → full MODE1 GPU reset → "VRAM is lost" → your loaded model evaporates while llama-server keeps waiting. It also explains your stuck reboots — the traces at the end show unkillable fence waits in the reset path.
Remove amdgpu.msi=0 from grub, update-grub, reboot the VM, then verify interrupts are actually flowing:

grep -i amdgpu /proc/interrupts

You should see MSI/MSI-X vectors for the amdgpu devices with counts that climb while a model loads. Then watch dmesg during a load — occasional fence fallback bursts are survivable (I get them too under Xen), but the constant 0.5s drumbeat should be gone.

Out of curiosity — did you add msi=0 while chasing the "no interrupts registered" issue from your first post? If you re-enable MSI and interrupts still don't flow, that's the real bug and exactly what the Vates folks will want to see, especially since you still have the debug Xen + iommu=debug boot — xl dmesg from dom0 while the VM is loading a model would show whether MSIs are being injected.

One more thing for after interrupts are fixed: your llama-server command shows --ctx-size 262144. A 256K context KV cache on top of that model is a huge allocation for a 32GB card and can silently spill into GTT and crawl. I'd validate at something modest first (8192), confirm it serves, then step the context up while watching VRAM. That's the approach I'm using on mine.

R

PessimistTech

@ravenet ooh, thanks for the callout, that is a config that is left over from some earlier debugging. I'll remove that and try again.

I had a similar thought about the model size of the qwen 3.6 model (chosen intially for running on both GPUs). I also tested with llama3.2 for a much smaller test.

PessimistTech

@ravenet welp, removed the amdgpu.msi=0 option, and I am seeing 0 interrupts in the /proc/interrupts file, and overall behavior is still the same...

106:          0          0          0          0          0          0  xen-pirq    -msi       amdgpu

PessimistTech

for future reference here is the xl dmesg at this point
xldmesg.txt

ravenet

@pessimisttech interesting development — before you chase the interrupt angle any further, here's the same grep from my working rig (same GPU, xcp-ng 8.3, 38 t/s inference):

 92:          0          0          0          0          0          0          0          0 xen-pirq    -msi       amdgpu

Zero. My card has never received an interrupt either. So scratch next thoughts on chasing ARI and nopv — zero GPU interrupts appears to be the normal (if ugly) state for these cards under 8.3 HVM passthrough, and everything runs on polling instead: ROCm compute completion is handled in userspace via HSA signals, and the kernel fences ride the fallback timer. That's why I see the same "Fence fallback timer expired" spam you do, permanently and harmlessly.

The difference between our rigs is in what happens next. On mine, the fences complete and polling just notices them late. On yours, they genuinely never complete — your dmesg shows signaled seq=52, emitted seq=54 and a failed ring reset before the MODE1 reset wipes VRAM (that "VRAM is lost" line is your model evaporating while llama-server waits). Your SDMA engine is actually stalling mid-transfer, which is a different beast from a missed notification.

With interrupts eliminated as the difference, the biggest remaining gap between our setups is the guest software stack, and it's a big one. I'm on Slowroll's stock 7.0.12 kernel with the in-tree amdgpu driver and current firmware. You're on Ubuntu's 6.8.0-134 — a kernel base from before this card existed — with AMD's DKMS 6.16.13 driver layered on top, and your boot log shows the seams: "SMU driver if version not matched" and a MES firmware version workaround note. That's roughly two years of RDNA4 enablement between our guests. I'd suspect that long before the hardware or the hypervisor.
Two ways to test it cheaply:

Close the gap in place: update the amdgpu firmware in your guest to current (the /lib/firmware/amdgpu files in a 6.8-era package predate this card's tuning) and move to the newest HWE kernel available. Fair warning that Ubuntu's HWE channel seems to top out well short of what I'm running, so this narrows the gap rather than closes it — but firmware alone might be enough if that's where the stall lives.

Cleaner and probably faster: spin up a throwaway openSUSE Slowroll (or Tumbleweed) VM, attach one GPU, install docker + the same lemonade container, and run the same load. That's my exact known-good recipe, and it's a proper controlled test — same host, same BIOS, same GPU, same container, only the guest stack changes. If it works there, you've found your answer and can decide whether to migrate or keep fighting Ubuntu. If it still fails on Slowroll, then it's genuinely host-side and we've got a much sharper bug report for the Vates folks.

@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

R

TeddyAstie

@ravenet said:
@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

TL;DR This is likely a known issue, and we're currently discussing on a potential fix.

There is a known bug regarding MSI on AMDGPU when using "hvm_pirq" ("use event channels for device interrupts"), see
https://lore.kernel.org/xen-devel/20240110095311.61809-1-roger.pau@citrix.com/
https://github.com/QubesOS/qubes-issues/issues/7971

The easy fix is to disable "HVM PIRQ" feature (which has been made disabled by default in Xen 4.19), but on the XCP-ng side, this is configured by XAPI Project which needs to be updated as well to apply this change (which appears to be a bit tricky for various reasons).

PessimistTech

@TeddyAstie Is this specific to the lack of interrupts?