Hello @anthoineb, We captured another occurrence on the same VM, os-ott-data-3-2 (172.30.52.193), on the same hypervisor and QCOW2 VDI as the incident reported on 2026-07-25. This occurrence is useful because xen_blkfront.max_ring_page_order=3 was definitely active: xenstore showed ring-page-order=3 with eight ring references; the guest queue depth was 256; 255 write requests remained in flight; all 256 blk-mq scheduler tags were busy; wbt_lat_usec=0; I/O PSI full was approximately 98%; completed xvdb counters did not advance between samples. The backend state was nevertheless the same as in the previous capture: tapdisk was sleeping in scheduler_wait_for_events(); tap-ctl reported reqs_outstanding=0; tap request counters were 0/0; req_prod=0 and rsp_prod=0; no tap, image, VBD, map or xenbus errors were reported. One detail may be relevant: GDB still printed n_reqs=32 and n_reqs_free=32, although the frontend had negotiated an order-3 ring and the guest exposed 256 tags. Is this expected because n_reqs represents a separate fixed tapdisk request pool, or could it indicate that tapdisk is not observing the complete multi-page ring? The controller captured GDB and the guest/hypervisor state before reboot. After the ten-minute diagnostic window, the same signature was confirmed three more times and the VM was rebooted. It rejoined the cluster successfully. I have attached: the complete GDB/tap-ctl capture; the guest blk-mq, diskstats, PSI, D-state stack and kernel-journal snapshot; the corresponding tapdisk/xenstore, daemon.log and SMlog excerpt; a README and SHA-256 checksums. The important new result appears to be that increasing max_ring_page_order from 0 to 3 did not prevent the stall; the same mismatch recurred with 255/256 frontend requests/tags occupied while the backend ring and tapdisk remained empty. 20260730-005220-gdb-172.30.52.193.txt 20260730-pre-reboot-guest-172.30.52.193.txt 20260730-pre-reboot-hypervisor-172.30.50.191.txt README.txt SHA256SUMS.txt

I have been in the Network Enterprise business for 28 years. I live in Virginia which is heavily populated with Data Centers and adding more all the time. However, we have quite a few customers especially municipalities which are saying NO to having anything hosted with cloud computing due to security and endless rising cost. We have been deploying xcp-ng in quite a few installations, on customer premise. I think VATES will find there is a whole other market in these areas they can reach. That being said, Thanks to VATES and all who have worked on solutions for this! It truly benefits everyone!

I got this running and when I set up a catalog and baseline the report tells me that it has nothing in the catalog. Is that because it doesn't understand XCP-ng or because my hardware is old enough that it has dropped out of the catalog? I have two PE R630 and one PE R730. I tried both the full default Dell catalog as well as the one that limits to BIOS and firmware updates.

@TeddyAstie From what I've tried...It won't work at all in Linux. It kind of works in Windows, and while I do see some load on the GPU when running something, there's as much if not more load on the CPU. It's not what I expected at least. I've never really gamed on PC, and definitely never Linux. My only uses for GPU's have been in media composition which was always on dedicated systems and Nvidia GPUs. More recently I've been doing some AI tasks, again with passed-through NVidia GPUs. Side tangent... I ran into another workload last night where the app wanted OpenGL...But OpenGL doesn't work with the "normal" VGA adapter in XCP-ng, and I don't have a GPU on that particular pool to try a pass-through. That lead me to VirGL, but it doesn't look like that's gone anywhere.

Just FYI, we had some issues with some Debian VMs for instance, with old kernel where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive. It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore. Newer kernel don't have this issue and are able to wake up. I'm quoting Debian here but it might very well happen on other distro

@ravenet Sure, gave that a whirl and no change, though I did notice some nvtop weirdness as it would show some load, but most of the wait time there was actually no load on the GPUs instead of the constant load matching the model being loaded. For some historical context ARI support was initially disabled in the bios when I started this thread. That was on the list of things I enabled when I started seeing some success with ollama (something in the changes since has broken ollama now too, but there was at least some forward progress after enabling). dmesg output overall looked the same, but I did see this output on the console (and in dmesg) that seemed interesting. Not 100% sure at this point if this was in the previous dmesg outputs or not, but may be worth sharing. EDIT: looks like this may actually be new... I looked back through the past dm dmesg outputs and I did not see this output. [ 108.547683] amdgpu 0000:00:09.0: MES(0) failed to respond to msg=REMOVE_QUEUE [ 108.547729] amdgpu 0000:00:09.0: failed to remove hardware queue from MES, doorbell=0x1202 [ 108.547746] amdgpu 0000:00:09.0: MES might be in unrecoverable state, issue a GPU reset [ 108.547774] amdgpu 0000:00:09.0: Failed to evict queue 2 [ 108.547789] amdgpu 0000:00:09.0: Failed to evict process queues [ 108.547803] amdgpu: Failed to quiesce KFD [ 108.547870] amdgpu 0000:00:09.0: GPU reset begin!. Source: 3 [ 109.656850] amdgpu 0000:00:09.0: Failed to remove queue 0 [ 109.657324] amdgpu 0000:00:09.0: Dumping IP State [ 109.756265] amdgpu 0000:00:09.0: Dumping IP State Completed [ 112.047695] amdgpu 0000:00:09.0: MODE1 reset [ 112.047797] amdgpu 0000:00:09.0: GPU mode1 reset [ 112.054505] amdgpu 0000:00:09.0: GPU smu mode1 reset [ 113.075393] amdgpu 0000:00:09.0: GPU reset succeeded, trying to resume [ 113.090354] amdgpu 0000:00:09.0: [drm] PCIE GART of 512M enabled (table at 0x00000087D6B00000). [ 113.092905] amdgpu 0000:00:09.0: [drm] AMDGPU device coredump file has been created [ 113.092913] amdgpu 0000:00:09.0: [drm] Check your /sys/class/drm/card1/device/devcoredump/data [ 113.092917] amdgpu 0000:00:09.0: VRAM is lost due to GPU reset! [ 113.092921] amdgpu 0000:00:09.0: PSP is resuming... [ 114.997591] amdgpu 0000:00:09.0: GECC is disabled, set amdgpu_ras_enable=1 to enable GECC in next boot cycle if needed [ 115.091125] amdgpu 0000:00:09.0: RAP: optional rap ta ucode is not available [ 115.091130] amdgpu 0000:00:09.0: SECUREDISPLAY: optional securedisplay ta ucode is not available [ 115.091134] amdgpu 0000:00:09.0: SMU is resuming... [ 115.091375] amdgpu 0000:00:09.0: smu driver if version = 0x0000002e, smu fw if version = 0x00000033, smu fw program = 0, smu fw version = 0x00684c00 (104.76.0) [ 115.432371] amdgpu 0000:00:09.0: SMU is resumed successfully! [ 115.445556] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0x4000000 [ 115.445731] amdgpu 0000:00:09.0: program CP_MES_CNTL : 0xc000000 [ 115.750240] amdgpu 0000:00:09.0: [drm] DMUB hardware initialized: version=0x0A000800 As an update for other things I have tried, in order to eliminate hardware issues, or bios settings I tried installing proxmox and spinning up a VM there with both GPUs passed through and it worked just fine... With that feedback I did a fresh install of XCP-NG 8.3 and spun up a fresh VM using the same steps as I used on proxmox and still no dice. This leads me to believe the issue is somewhere in the XCP-NG passthrough stack with my specific hardware...

@jedimarcus solution work, thanks, We had server working 100% since last two weeks but error didn't disappear, after clean logs was ok thank you.

@tuxen Doing some research, it doesn't look like the Xeon's I have are affected. But I'm willing to try the next time I need to reboot. Will report back after that.

Hello, Thanks for all !

@psafont Thanks for the quick response and clarification. I appreciate you opening a work item for this. Looking forward to seeing this improvement in a future release.

Hi, XCP-ng got an event system that will propagate things like this instantly, at least that's the way it works normally Do you have the same behaviour in Xen Orchestra? Have you reported the issue to CloudStack? If you have an XCP-ng support subscription, you can also open a ticket so we can take a look on XCP-ng status to catch any obvious issue.

@psafont Thank you for the quick response. I also found a similar issue: the other-config:auto-scan=true parameter is not being applied during xe sr-create either. As with the name-description parameter, the workaround is to add it separately afterwards using xe sr-param-add.

@julienXOvates Excellent, thanks for looking at this Julien! Rob

Thanks for that information. I will make this message short because @stormi is busy but I want to say thanks to Vates and XCP-ng for all their work done to support Windows on the Xen platform. This includes TPM2 and secure boot support and Microsoft-signed pv drivers. Well done!

The "PIF is not attached" usually means the network interface selected as the migration network isn't currently active on the target host. It can happen after upgrades if the host hasn't been fully rebooted. Worth checking whether a reboot of both hosts changes anything, and running xe pif-list to see whether that specific PIF shows currently-attached: true on the target. If the PIF looks attached in xe but migration still fails, might be worth a ping to Team-XAPI-Network.

perhaps "in the context of a proceeding RPU, do not start halted VMs" ? or "boot only halted VMs that have HA enabled" ? but I can imagine corner cases where this is not wanted. some chicken & egg problem.

I can confirm that when using Citrix/Xenserver guest utilities version 8.4 (https://github.com/xenserver/xe-guest-utilities/releases/tag/v8.4.0) memory ballooning / DMC is working fine. After live migration the RAM of the linux guest is expanded to dynamic_max again. So this issue was in fact caused by Rust based xen-guest-agent. For now I'll keep using Citrix/Xenserver guest utilities on my Linux guests until the feature is implemented in Vates rust-based guest utilities. Best regards

@TeddyAstie Attached is the output you requested xen-cpuid -p.txt

@TeddyAstie That's more like it. I'm not finding the com2tcp though. I at least have something to search for. Once configured, is this persistent or do I need to create some sort of start up script that runs/loads a config on machine boot? Thanks!!