RDNA 4 GPU Passthrough

ravenet

I am running a radeon ai Pro 9700 on xcp-ng 8.3 passed through to an opensuse Slowroll, with rocm and lemonade docker. Gemma 4 31B-it-MTP gets 38tps

did you pass through as pcie passthrough with both the video and audio, or just the 'gpu passthrough' option in xoa? I did the 2 devices pcie passthrough
Make sure you have Resizable BAR enabled in bios or it just will not work

R

PessimistTech

@ravenet glad to hear you were able to get it working!
Yes I passed through both the audio device and video device via PCI passthrough. I have also checked and ensured Resizeable BAR is enabled in bios.

Did you do anything to set it up directly on the host? Or did you just use the PCI device passthrough options in XO?

PessimistTech

Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.

EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

ravenet

@PessimistTech said:

Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.

EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

Do you have examples of workloads that are struggling still?
and no, didn't do anything special on the xcp host that I'm aware of. Was already using it to passthrough a radeon pro 7500 to a windows vm for revit clarity automation server. It's been a year so I'll triple check if anything special was done back then.

PessimistTech

@ravenet sure, the workloads I have been testing with include ollama, vLLM, and lemonade server (all via docker for easier setup, but I do have all the rocm/AMD drivers and what not setup on the VM directly). Ollama works just fine now, but lemonade server still hangs and vLLM fails with the same errors on startup.

Cool, thanks for info/checking!

ravenet

@PessimistTech make sure lemonade is set to use rocm as backend. It was slow and unusable from vulkan

docker compose exec lemonade lemonade config set llamacpp.backend=rocm

does it finish loading model then just respond horribly slowly, or does it not complete the model load?
did you setup as straight docker, or as a docker compose? I did mine as a compose, but shouldn't matter outside of how the config is laid out

is your guest seeing the whole mem on gpu and the PCIE link speed?

sudo lspci -vv -d 1002: | grep -E "Region [0-9]|LnkSta:"
        Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
        Region 2: Memory at 1800000000 (64-bit, prefetchable) [size=256M]
        Region 4: I/O ports at c100 [size=256]
        Region 5: Memory at f0800000 (32-bit, non-prefetchable) [size=512K]
                LnkSta: Speed 32GT/s, Width x16
        Region 0: Memory at f08c0000 (32-bit, non-prefetchable) [size=16K]
                LnkSta: Speed 32GT/s, Width x16]

You'll also want to make sure AtomicOP is working, below is example command and result

sudo lspci -vv -d 1002: | grep -iE "AtomicOp"

                         AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
                         AtomicOpsCtl: ReqEn-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                         AtomicOpsCtl: ReqEn-

PessimistTech

@ravenet I have not tried with the rocm backend yet, but I will give that a whirl. When using the GPUs it never completes the model load.

For testing I was just using the docker run command from the lemonade server docs.

docker run -d \
	--name lemonade-server \
	-p 13305:13305 \
	-v lemonade-cache:/root/.cache/huggingface \
	-v lemonade-llama:/opt/lemonade/llama \
	-v lemonade-recipe:/root/.cache/lemonade \
	--device=/dev/kfd \
	--device=/dev/dri \
	ghcr.io/lemonade-sdk/lemonade-server:latest

Command output:

lspci -vv -s 00:08.0 | grep -E "Region [0-9]|LnkSta:"
	Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
	Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at c300 [size=256]
	Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
		LnkSta:	Speed 32GT/s, Width x16

lspci -vv -s 00:08.0 | grep -iE AtomicOP
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
			 AtomicOpsCtl: ReqEn+

ravenet

@PessimistTech

switch to rocm-nightly, as that's required to bring support for our ai pro 9700

docker exec lemonade-server lemonade config set llamacpp.backend=rocm
docker exec lemonade-server lemonade config set rocm_channel=nightly
docker restart lemonade-server

The second line is important. This GPU is gfx1201 and the rocm-stable channel doesn't have HIP kernels for it yet. If you switch to rocm backend without setting the channel to nightly, it silently falls back to CPU with no error — you'll just see terrible speeds and an idle GPU. Ask me how I know.

Restart is needed for the backend change to take effect, and the config survives restarts since the docs' run command already mounts the lemonade-recipe volume where it lives.

One more thing I'd add to your docker run: the --init flag, right after docker run -d. Without an init process in the container, a failed/hung model load can leave the container wedged in D-state and you end up force-killing it. I keep it permanently in my compose (init: true) after learning this the hard way.

Good news is your lspci output looks healthy — Region 0 at 32G in the guest and AtomicOpsCtl ReqEn+ means your BAR and atomics are fine, which was the failure mode on my end originally. So this is likely just backend/kernel config, not the passthrough itself.

After the restart you can confirm it's actually on GPU by watching VRAM fill during model load:

watch -n1 'cat /sys/class/drm/card*/device/mem_info_vram_used'

If that stays flat while a model loads, you're on CPU fallback.

R

PessimistTech

@ravenet well, applied those updates and still the same results unfortunately...

I am definitely seeing load on the GPU, so it is able to start the process, lemonade server just sits waiting logging
2026-07-08 20:06:06.477 [Debug] (WrappedServer) Still waiting for llama-server...
while the GPUs are sitting there with the model in mem (or at least the correct amount of mem for the model size being consumed).

looks like the rocm backend is using version b1292 which does note the correct gfx1201 model in the release notes.

ravenet

@PessimistTech
Try pass through just 1 gpu. See where that gets you

Edit:
actually, possibly faster way to test the single-GPU theory without touching passthrough — just hide one card from ROCm:

docker run -d \
    --name lemonade-server \
    --init \
    -e HIP_VISIBLE_DEVICES=0 \
    -p 13305:13305 \
    -v lemonade-cache:/root/.cache/huggingface \
    -v lemonade-llama:/opt/lemonade/llama \
    -v lemonade-recipe:/root/.cache/lemonade \
    --device=/dev/kfd \
    --device=/dev/dri \
    ghcr.io/lemonade-sdk/lemonade-server:latest

If it loads and serves with one GPU visible, the problem is apparently cross-GPU peer-to-peer under Xen. With two passed-through devices, llama.cpp splits the model and the first cross-device sync tries P2P DMA through the virtual root complex — ROCm reports peer access as available but the transaction never completes, so the fence waits forever. Model in VRAM, llama-server alive but frozen. That would explain why it dies after the memory is allocated rather than during load. (Related: even on my single card I get bursts of sdma0 fence fallback messages under Xen from lost interrupts — harmless for me, but on a P2P path a lost fence is fatal.)

Two things to grab while it's hung, so we know for sure where it's stuck:


docker exec lemonade-server ps -eo pid,stat,wchan:32,cmd | grep llama

If llama-server shows state D with a wchan in amdgpu fence/ring wait, it's wedged in the kernel driver, not userspace. Also check guest dmesg during the hang for ring timeout / fence messages:

sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence"

If single-GPU works and you want both cards back, a couple of things to try before giving up on dual:

-e HSA_ENABLE_SDMA=0 — routes copies through shader blits instead of the SDMA engines; it's the standard workaround for ROCm fence hangs inside VMs
keep the split on one device via lemonade: docker exec lemonade-server lemonade config set llamacpp.args="--split-mode none" then restart — both GPUs visible but no cross-device traffic

Worst case, two VMs with one GPU each works around the P2P problem entirely and you can load-balance in front.

Fingers crossed

R