@PessimistTech Try pass through just 1 gpu. See where that gets you Edit: actually, possibly faster way to test the single-GPU theory without touching passthrough — just hide one card from ROCm: docker run -d \ --name lemonade-server \ --init \ -e HIP_VISIBLE_DEVICES=0 \ -p 13305:13305 \ -v lemonade-cache:/root/.cache/huggingface \ -v lemonade-llama:/opt/lemonade/llama \ -v lemonade-recipe:/root/.cache/lemonade \ --device=/dev/kfd \ --device=/dev/dri \ ghcr.io/lemonade-sdk/lemonade-server:latest If it loads and serves with one GPU visible, the problem is apparently cross-GPU peer-to-peer under Xen. With two passed-through devices, llama.cpp splits the model and the first cross-device sync tries P2P DMA through the virtual root complex — ROCm reports peer access as available but the transaction never completes, so the fence waits forever. Model in VRAM, llama-server alive but frozen. That would explain why it dies after the memory is allocated rather than during load. (Related: even on my single card I get bursts of sdma0 fence fallback messages under Xen from lost interrupts — harmless for me, but on a P2P path a lost fence is fatal.) Two things to grab while it's hung, so we know for sure where it's stuck: docker exec lemonade-server ps -eo pid,stat,wchan:32,cmd | grep llama If llama-server shows state D with a wchan in amdgpu fence/ring wait, it's wedged in the kernel driver, not userspace. Also check guest dmesg during the hang for ring timeout / fence messages: sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence" If single-GPU works and you want both cards back, a couple of things to try before giving up on dual: -e HSA_ENABLE_SDMA=0 — routes copies through shader blits instead of the SDMA engines; it's the standard workaround for ROCm fence hangs inside VMs keep the split on one device via lemonade: docker exec lemonade-server lemonade config set llamacpp.args="--split-mode none" then restart — both GPUs visible but no cross-device traffic Worst case, two VMs with one GPU each works around the P2P problem entirely and you can load-balance in front. Fingers crossed R

@acebmxer This has been reported before. XO needs to report an actual error for this problem.

@MajorP93 I'l do that, didn't know about it.

@poddingue said: Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out". The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

@pkgw Thanks for the feedback. The good news is that there is already an effort under way rework and enhance the RPU process. Let me ping @Team-XAPI-Network and @Team-XO-Backend in case they want to request additional information from you.