@PessimistTech
Try pass through just 1 gpu. See where that gets you
Edit:
actually, possibly faster way to test the single-GPU theory without touching passthrough — just hide one card from ROCm:
docker run -d \
--name lemonade-server \
--init \
-e HIP_VISIBLE_DEVICES=0 \
-p 13305:13305 \
-v lemonade-cache:/root/.cache/huggingface \
-v lemonade-llama:/opt/lemonade/llama \
-v lemonade-recipe:/root/.cache/lemonade \
--device=/dev/kfd \
--device=/dev/dri \
ghcr.io/lemonade-sdk/lemonade-server:latest
If it loads and serves with one GPU visible, the problem is apparently cross-GPU peer-to-peer under Xen. With two passed-through devices, llama.cpp splits the model and the first cross-device sync tries P2P DMA through the virtual root complex — ROCm reports peer access as available but the transaction never completes, so the fence waits forever. Model in VRAM, llama-server alive but frozen. That would explain why it dies after the memory is allocated rather than during load. (Related: even on my single card I get bursts of sdma0 fence fallback messages under Xen from lost interrupts — harmless for me, but on a P2P path a lost fence is fatal.)
Two things to grab while it's hung, so we know for sure where it's stuck:
docker exec lemonade-server ps -eo pid,stat,wchan:32,cmd | grep llama
If llama-server shows state D with a wchan in amdgpu fence/ring wait, it's wedged in the kernel driver, not userspace. Also check guest dmesg during the hang for ring timeout / fence messages:
sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence"
If single-GPU works and you want both cards back, a couple of things to try before giving up on dual:
-e HSA_ENABLE_SDMA=0 — routes copies through shader blits instead of the SDMA engines; it's the standard workaround for ROCm fence hangs inside VMs
keep the split on one device via lemonade: docker exec lemonade-server lemonade config set llamacpp.args="--split-mode none" then restart — both GPUs visible but no cross-device traffic
Worst case, two VMs with one GPU each works around the P2P problem entirely and you can load-balance in front.
Fingers crossed
R