@PessimistTech said:
@ravenet
Thanks for the ideas!
Here's the outcome of the tests
ps -eo pid,stat,wchan:32,cmd | grep llama:
62 Rl - /root/.cache/lemonade/bin/llamacpp/rocm-nightly/llama-server -m /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf --ctx-size 262144 --port 8001 --jinja --metrics --mmproj /root/.cache/huggingface/hub/models--unsloth--Qwen3.6-35B-A3B-GGUF/snapshots/a483e9e6cbd595906af30beda3187c2663a1118c/mmproj-F16.gguf --reasoning-format auto --no-webui --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 --chat-template-kwargs {"preserve_thinking":true}
sudo dmesg -w | grep -iE "amdgpu|sdma|ring|fence":
dmesgout.txt
Same overall results... I even re-ran the same tests after removing the 2nd GPU from the PCI passthrough settings and still the same...
I also seem to run into an issue where the GPU tends to get stuck after some of these tests. Even rebooting gets stuck waiting for llama server processes to stop and I end up having to force power off. Not sure if that is related as I get the same output on the first test, but it may be worth noting.
found it — or at least found a very large problem. Your kernel command line has
amdgpu.msi=0:
Command line: BOOT_IMAGE=/vmlinuz-6.8.0-134-generic root=... ro amdgpu.msi=0 console=tty1 console=ttyS0
That disables MSI interrupts for the GPU driver entirely, and your dmesg shows exactly what that costs: fence fallback timer messages every half-second on both cards from the moment the driver loads, before any workload runs. The driver is surviving on a polling fallback instead of interrupts. My card throws those messages in occasional bursts under load (lost interrupts); yours throws them constantly (no interrupts). Under load that degrades into what's in your log: sdma0 ring timeout → ring reset fails → full MODE1 GPU reset → "VRAM is lost" → your loaded model evaporates while llama-server keeps waiting. It also explains your stuck reboots — the traces at the end show unkillable fence waits in the reset path.
Remove amdgpu.msi=0 from grub, update-grub, reboot the VM, then verify interrupts are actually flowing:
grep -i amdgpu /proc/interrupts
You should see MSI/MSI-X vectors for the amdgpu devices with counts that climb while a model loads. Then watch dmesg during a load — occasional fence fallback bursts are survivable (I get them too under Xen), but the constant 0.5s drumbeat should be gone.
Out of curiosity — did you add msi=0 while chasing the "no interrupts registered" issue from your first post? If you re-enable MSI and interrupts still don't flow, that's the real bug and exactly what the Vates folks will want to see, especially since you still have the debug Xen + iommu=debug boot — xl dmesg from dom0 while the VM is loading a model would show whether MSIs are being injected.
One more thing for after interrupts are fixed: your llama-server command shows --ctx-size 262144. A 256K context KV cache on top of that model is a huge allocation for a 32GB card and can silently spill into GTT and crawl. I'd validate at something modest first (8192), confirm it serves, then step the context up while watching VRAM. That's the approach I'm using on mine.
R
