RDNA 4 GPU Passthrough
-
@PessimistTech
That looks a bit odd indeed.In addition to what proposed @yannsionneau, can you also give the output of :
xl dmesg(in Dom0)dmesg(in Dom0)dmesg(in the guest)
So that we can try to pin-point what may be happening.
-
@yannsionneau Thank you!
-
segment bus device function address of the device
Hi all, thanks for your reponses!
the below is done with the assumption that dom0 is just the host cli. If that is incorrect please let me know and I can update this info:
dom0: lspci -vvv -nn -s 03:00.0:
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device [1849:5413]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 120
Region 0: Memory at 7f000000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at 7f800000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at 3000 [size=256]
Region 5: Memory at f2200000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at f2280000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed unknown, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed unknown, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [200 v1] #15
Capabilities: [240 v1] Power Budgeting <?>
Capabilities: [270 v1] #19
Capabilities: [2a0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Capabilities: [2d0 v1] Process Address Space ID (PASID)
PASIDCap: Exec+ Priv+, Max PASID Width: 10
PASIDCtl: Enable- Exec- Priv-
Capabilities: [320 v1] Latency Tolerance Reporting
Max snoop latency: 0ns
Max no snoop latency: 0ns
Capabilities: [410 v1] #26
Capabilities: [450 v1] #27
Capabilities: [500 v1] #2a
Kernel driver in use: pcibackVM lspci -vvv -nn -s 03:00.0:
00:08.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
Subsystem: ASRock Incorporation Device [1849:5413]
Physical Slot: 8
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
Latency: 0
Interrupt: pin A routed to IRQ 17
Region 0: Memory at 800000000 (64-bit, prefetchable) [size=32G]
Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at c300 [size=256]
Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 32GT/s, Width x16
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-
AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
AtomicOpsCtl: ReqEn+
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Kernel driver in use: amdgpu
Kernel modules: amdgpudom0 xl dmesg output: xldmesg.txt
dom0 dmesg output: dom0dmesg.txt
vm dmesg output: vmdmesg.txtI believe this is all the output requested (I only provided the lspci output for a single GPU as they are identical GPUs , but happy to provide all the outputs if desired (will note I am not entirely sure I selected the exact GPU in both contexts due to the mismatched IDs in vm vs dom0, not sure that is an issue but may be worth calling out)
edit: all output was taken with the VM running (but at idle). I did apply the grub changes to use the -d.gz file and iommu=debug as well so ideally that output should be included)
-
I am running a radeon ai Pro 9700 on xcp-ng 8.3 passed through to an opensuse Slowroll, with rocm and lemonade docker. Gemma 4 31B-it-MTP gets 38tps
did you pass through as pcie passthrough with both the video and audio, or just the 'gpu passthrough' option in xoa? I did the 2 devices pcie passthrough
Make sure you have Resizable BAR enabled in bios or it just will not workR
-
@ravenet glad to hear you were able to get it working!
Yes I passed through both the audio device and video device via PCI passthrough. I have also checked and ensured Resizeable BAR is enabled in bios.Did you do anything to set it up directly on the host? Or did you just use the PCI device passthrough options in XO?
-
Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.
EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...
-
Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.
EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...
Do you have examples of workloads that are struggling still?
and no, didn't do anything special on the xcp host that I'm aware of. Was already using it to passthrough a radeon pro 7500 to a windows vm for revit clarity automation server. It's been a year so I'll triple check if anything special was done back then. -
@ravenet sure, the workloads I have been testing with include ollama, vLLM, and lemonade server (all via docker for easier setup, but I do have all the rocm/AMD drivers and what not setup on the VM directly). Ollama works just fine now, but lemonade server still hangs and vLLM fails with the same errors on startup.
Cool, thanks for info/checking!
-
@PessimistTech make sure lemonade is set to use rocm as backend. It was slow and unusable from vulkan
docker compose exec lemonade lemonade config set llamacpp.backend=rocmdoes it finish loading model then just respond horribly slowly, or does it not complete the model load?
did you setup as straight docker, or as a docker compose? I did mine as a compose, but shouldn't matter outside of how the config is laid outis your guest seeing the whole mem on gpu and the PCIE link speed?
sudo lspci -vv -d 1002: | grep -E "Region [0-9]|LnkSta:" Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G] Region 2: Memory at 1800000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at c100 [size=256] Region 5: Memory at f0800000 (32-bit, non-prefetchable) [size=512K] LnkSta: Speed 32GT/s, Width x16 Region 0: Memory at f08c0000 (32-bit, non-prefetchable) [size=16K] LnkSta: Speed 32GT/s, Width x16]You'll also want to make sure AtomicOP is working, below is example command and result
sudo lspci -vv -d 1002: | grep -iE "AtomicOp" AtomicOpsCap: 32bit+ 64bit+ 128bitCAS- AtomicOpsCtl: ReqEn- AtomicOpsCap: 32bit- 64bit- 128bitCAS- AtomicOpsCtl: ReqEn- -
@ravenet I have not tried with the rocm backend yet, but I will give that a whirl. When using the GPUs it never completes the model load.
For testing I was just using the docker run command from the lemonade server docs.
docker run -d \ --name lemonade-server \ -p 13305:13305 \ -v lemonade-cache:/root/.cache/huggingface \ -v lemonade-llama:/opt/lemonade/llama \ -v lemonade-recipe:/root/.cache/lemonade \ --device=/dev/kfd \ --device=/dev/dri \ ghcr.io/lemonade-sdk/lemonade-server:latestCommand output:
lspci -vv -s 00:08.0 | grep -E "Region [0-9]|LnkSta:" Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G] Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at c300 [size=256] Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K] LnkSta: Speed 32GT/s, Width x16lspci -vv -s 00:08.0 | grep -iE AtomicOP AtomicOpsCap: 32bit+ 64bit+ 128bitCAS- AtomicOpsCtl: ReqEn+ -
switch to rocm-nightly, as that's required to bring support for our ai pro 9700
docker exec lemonade-server lemonade config set llamacpp.backend=rocm docker exec lemonade-server lemonade config set rocm_channel=nightly docker restart lemonade-serverThe second line is important. This GPU is gfx1201 and the rocm-stable channel doesn't have HIP kernels for it yet. If you switch to rocm backend without setting the channel to nightly, it silently falls back to CPU with no error — you'll just see terrible speeds and an idle GPU. Ask me how I know.
Restart is needed for the backend change to take effect, and the config survives restarts since the docs' run command already mounts the lemonade-recipe volume where it lives.
One more thing I'd add to your docker run: the --init flag, right after docker run -d. Without an init process in the container, a failed/hung model load can leave the container wedged in D-state and you end up force-killing it. I keep it permanently in my compose (init: true) after learning this the hard way.
Good news is your lspci output looks healthy — Region 0 at 32G in the guest and AtomicOpsCtl ReqEn+ means your BAR and atomics are fine, which was the failure mode on my end originally. So this is likely just backend/kernel config, not the passthrough itself.
After the restart you can confirm it's actually on GPU by watching VRAM fill during model load:
watch -n1 'cat /sys/class/drm/card*/device/mem_info_vram_used'If that stays flat while a model loads, you're on CPU fallback.
R
-
@ravenet well, applied those updates and still the same results unfortunately...
I am definitely seeing load on the GPU, so it is able to start the process, lemonade server just sits waiting logging
2026-07-08 20:06:06.477 [Debug] (WrappedServer) Still waiting for llama-server...
while the GPUs are sitting there with the model in mem (or at least the correct amount of mem for the model size being consumed).looks like the rocm backend is using version b1292 which does note the correct gfx1201 model in the release notes.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login