@ravenet Sure, gave that a whirl and no change, though I did notice some nvtop weirdness as it would show some load, but most of the wait time there was actually no load on the GPUs instead of the constant load matching the model being loaded. For some historical context ARI support was initially disabled in the bios when I started this thread. That was on the list of things I enabled when I started seeing some success with ollama (something in the changes since has broken ollama now too, but there was at least some forward progress after enabling). dmesg output overall looked the same, but I did see this output on the console (and in dmesg) that seemed interesting. Not 100% sure at this point if this was in the previous dmesg outputs or not, but may be worth sharing. EDIT: looks like this may actually be new... I looked back through the past dm dmesg outputs and I did not see this output.

[  108.547683] amdgpu 0000:00:09.0: MES(0) failed to respond to msg=REMOVE_QUEUE
[  108.547729] amdgpu 0000:00:09.0: failed to remove hardware queue from MES, doorbell=0x1202
[  108.547746] amdgpu 0000:00:09.0: MES might be in unrecoverable state, issue a GPU reset
[  108.547774] amdgpu 0000:00:09.0: Failed to evict queue 2
[  108.547789] amdgpu 0000:00:09.0: Failed to evict process queues
[  108.547803] amdgpu: Failed to quiesce KFD
[  108.547870] amdgpu 0000:00:09.0: GPU reset begin!. Source:  3
[  109.656850] amdgpu 0000:00:09.0: Failed to remove queue 0
[  109.657324] amdgpu 0000:00:09.0: Dumping IP State
[  109.756265] amdgpu 0000:00:09.0: Dumping IP State Completed
[  112.047695] amdgpu 0000:00:09.0: MODE1 reset
[  112.047797] amdgpu 0000:00:09.0: GPU mode1 reset
[  112.054505] amdgpu 0000:00:09.0: GPU smu mode1 reset
[  113.075393] amdgpu 0000:00:09.0: GPU reset succeeded, trying to resume
[  113.090354] amdgpu 0000:00:09.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000).
[  113.092905] amdgpu 0000:00:09.0: [drm] AMDGPU device coredump file has been created
[  113.092913] amdgpu 0000:00:09.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[  113.092917] amdgpu 0000:00:09.0: VRAM is lost due to GPU reset!
[  113.092921] amdgpu 0000:00:09.0: PSP is resuming...
[  114.997591] amdgpu 0000:00:09.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed
[  115.091125] amdgpu 0000:00:09.0: RAP: optional rap ta ucode is not available
[  115.091130] amdgpu 0000:00:09.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
[  115.091134] amdgpu 0000:00:09.0: SMU is resuming...
[  115.091375] amdgpu 0000:00:09.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0)
[  115.432371] amdgpu 0000:00:09.0: SMU is resumed successfully!
[  115.445556] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0x4000000
[  115.445731] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0xc000000
[  115.750240] amdgpu 0000:00:09.0: [drm] DMUB hardware initialized: version=0x0A000800

As an update for other things I have tried, in order to eliminate hardware issues, or bios settings I tried installing proxmox and spinning up a VM there with both GPUs passed through and it worked just fine... With that feedback I did a fresh install of XCP-NG 8.3 and spun up a fresh VM using the same steps as I used on proxmox and still no dice. This leads me to believe the issue is somewhere in the XCP-NG passthrough stack with my specific hardware...

@ravenet well... thanks for the idea. I went ahead and tried spinning up an opensuse slowroll VM. I installed docker and the amdgpu firmware/kernel drivers, and tried running lemonade-server and ollama and same behavior. So that definitely seems to point to something weird going on on my host rather than the guest software stack...

@TeddyAstie Is this specific to the lack of interrupts?

for future reference here is the xl dmesg at this point
xldmesg.txt

@ravenet welp, removed the amdgpu.msi=0 option, and I am seeing 0 interrupts in the /proc/interrupts file, and overall behavior is still the same...

106:          0          0          0          0          0          0  xen-pirq    -msi       amdgpu

@ravenet ooh, thanks for the callout, that is a config that is left over from some earlier debugging. I'll remove that and try again.

I had a similar thought about the model size of the qwen 3.6 model (chosen intially for running on both GPUs). I also tested with llama3.2 for a much smaller test.

@ravenet
Thanks for the ideas!

Here's the outcome of the tests
ps -eo pid,stat,wchan:32,cmd | grep llama:

62 Rl   -                                /root/.cache/lemonade/bin/llamacpp/rocm-nightly/llama-server -m /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --ctx-size 262144 --port 8001 --jinja --metrics --mmproj /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/mmproj-F16.gguf --reasoning-format auto --no-webui --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 --chat-template-kwargs {"preserve_thinking":true}

sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence":
dmesgout.txt

Same overall results... I even re-ran the same tests after removing the 2nd GPU from the PCI passthrough settings and still the same...

I also seem to run into an issue where the GPU tends to get stuck after some of these tests. Even rebooting gets stuck waiting for llama server processes to stop and I end up having to force power off. Not sure if that is related as I get the same output on the first test, but it may be worth noting.

@ravenet well, applied those updates and still the same results unfortunately...

I am definitely seeing load on the GPU, so it is able to start the process, lemonade server just sits waiting logging
2026-07-08 20:06:06.477 [Debug] (WrappedServer) Still waiting for llama-server...
while the GPUs are sitting there with the model in mem (or at least the correct amount of mem for the model size being consumed).

looks like the rocm backend is using version b1292 which does note the correct gfx1201 model in the release notes.

@ravenet I have not tried with the rocm backend yet, but I will give that a whirl. When using the GPUs it never completes the model load.

For testing I was just using the docker run command from the lemonade server docs.

docker run -d \
	--name lemonade-server \
	-p 13305:13305 \
	-v lemonade-cache:/root/.cache/huggingface \
	-v lemonade-llama:/opt/lemonade/llama \
	-v lemonade-recipe:/root/.cache/lemonade \
	--device=/dev/kfd \
	--device=/dev/dri \
	ghcr.io/lemonade-sdk/lemonade-server:latest

Command output:

lspci -vv -s 00:08.0 | grep -E "Region [0-9]|LnkSta:"
	Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
	Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at c300 [size=256]
	Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
		LnkSta:	Speed 32GT/s, Width x16

lspci -vv -s 00:08.0 | grep -iE AtomicOP
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
			 AtomicOpsCtl: ReqEn+

@ravenet sure, the workloads I have been testing with include ollama, vLLM, and lemonade server (all via docker for easier setup, but I do have all the rocm/AMD drivers and what not setup on the VM directly). Ollama works just fine now, but lemonade server still hangs and vLLM fails with the same errors on startup.

Cool, thanks for info/checking!

Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.

EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

@ravenet glad to hear you were able to get it working!
Yes I passed through both the audio device and video device via PCI passthrough. I have also checked and ensured Resizeable BAR is enabled in bios.

Did you do anything to set it up directly on the host? Or did you just use the PCI device passthrough options in XO?

@yannsionneau said:

segment bus device function address of the device

Hi all, thanks for your reponses!

the below is done with the assumption that dom0 is just the host cli. If that is incorrect please let me know and I can update this info:

dom0: lspci -vvv -nn -s 03:00.0:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device [1849:5413]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 120
Region 0: Memory at 7f000000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at 7f800000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at 3000 [size=256]
Region 5: Memory at f2200000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at f2280000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed unknown, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [200 v1] #15
Capabilities: [240 v1] Power Budgeting <?>
Capabilities: [270 v1] #19
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [2d0 v1] Process Address Space ID (PASID)
PASIDCap: Exec+ Priv+, Max PASID Width: 10
PASIDCtl: Enable- Exec- Priv-
Capabilities: [320 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [410 v1] #26
Capabilities: [450 v1] #27
Capabilities: [500 v1] #2a
Kernel driver in use: pciback

VM lspci -vvv -nn -s 03:00.0:
00:08.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device [1849:5413]
Physical Slot: 8
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0
Interrupt: pin A routed to IRQ 17
Region 0: Memory at 800000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at c300 [size=256]
Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 32GT/s, Width x16
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn+
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver in use: amdgpu
Kernel modules: amdgpu

dom0 xl dmesg output: xldmesg.txt
dom0 dmesg output: dom0dmesg.txt
vm dmesg output: vmdmesg.txt

I believe this is all the output requested (I only provided the lspci output for a single GPU as they are identical GPUs , but happy to provide all the outputs if desired (will note I am not entirely sure I selected the exact GPU in both contexts due to the mismatched IDs in vm vs dom0, not sure that is an issue but may be worth calling out)

edit: all output was taken with the VM running (but at idle). I did apply the grub changes to use the -d.gz file and iommu=debug as well so ideally that output should be included)

@yannsionneau Thank you!

Hi all,
I am trying to pass through 2 AMD Radeon Ai Pro 9700 GPUs to a ubuntu VM. I have gottent to the point where I have the GPUs passed through, I can see them fine in the VM (Ie rocm-info/rocm-smi/nvtop all show the GPUs and current status without isssue), and I can even start workloads against them and see load start on the GPUs, but the workloads always time out or fail as they don't seem to get a response back or something along those lines.

In my digging so far I have seen that no interrupts are being registered, and as far as I can tell this seems to be an issue with the XCP-NG -> VM layer in the repsponse (as I cannot fine any issues, errors or misconfigs on the VM other than the timeouts).

Has anyone tried/had success with newer RDNA 4 GPUs? Any suggestions on things to try next? I'm at a bit of a loss here...