for future reference here is the xl dmesg at this point xldmesg.txt

Thanks for checking. That's useful, even if it points away from where I was looking. Clocks within a second of each other means drift probably isn't your problem, and I'd guess the MST/UTC difference is just how dom0 displays it, though I'm not sure. What I keep coming back to is that your full backups fail while the delta jobs on the same hosts never do. That's the same split in https://github.com/vatesfr/xen-orchestra/issues/9181, where full backups hit BodyTimeoutError on VMs with big disks or a lot of free space and the deltas are fine. If your failing VMs look like that, your dates and the MST/UTC detail would do more good on that issue than buried in here. MajorP93 created this issue in vatesfr/xen-orchestra open Full backup fails on large VMs / VMs with lots of free disk space #9181

Indeed, it should be detected/reported correctly to avoid losing time finding the problem. Let me ping @julienXOVates

SM logs "Date","Time","Line","tsNs","id","clusterLoggingSource","detected_level","hostname","job","priority","service_name","syslog_identifier","unit" 2026-07-09T22:01:14.842Z,1783634474842,"[2060240][MainThread] ***** Failed to snapshot!: EXCEPTION <class 'linstorvolumemanager.LinstorVolumeManagerError'>, Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01'",1783634474842687000,1783634474842687000_2194f1ea,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service 2026-07-09T22:01:15.206Z,1783634475206,"[2060240][MainThread] Raising exception [81, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']]",1783634475206274000,1783634475206274000_2137a4ec,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service 2026-07-09T22:01:15.215Z,1783634475215,"[2060240][MainThread] ***** generic exception: vdi_snapshot: EXCEPTION <class 'xs_errors.SROSError'>, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']",1783634475215733000,1783634475215733000_8881a7dc,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service 2026-07-09T22:01:15.228Z,1783634475228,"[2060240][MainThread] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'xs_errors.SROSError'>, Failed to clone VDI [opterr=Unable to activate device path of `xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40` on node `ovbh-pprod-xen01`: SUCC:Successfully set property key(s): StorPoolName, SUCC:New resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' registered., SUCC:Volume with number '0' on resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on node 'ovbh-pprod-xen01' successfully registered, ERRO:(ovbh-pprod-xen05) Failed to adjust DRBD resource xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40, SUCC:(ovbh-pprod-xen04) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen04', SUCC:(ovbh-pprod-xen02) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Added peer(s) 'ovbh-pprod-xen01' to resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen02', SUCC:(ovbh-pprod-xen01) Resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' [DRBD] adjusted., SUCC:Created resource 'xcp-volume-73b0bb04-2a8f-433d-b0bb-51a887fc4a40' on 'ovbh-pprod-xen01']",1783634475228432000,1783634475228432000_9ae0439a,xcp-ng,unknown,ovbh-pprod-xen01,systemd-journal,6,systemd-journal,SM,forkexecd.service Linstor controller and satellite logs Logs-A-data-2026-07-09 15_11_07.csv.txt

@florent Sorry, missed your previous post. I will return to master and test it (over the weekend). Thank you!

Hardware update: Got a couple M9148 (4 port Matrox DP) and Wavlink WL-UG7500DH (4 port DP/HDMI) in to test. Configured a Win11 Pro VM with the Matrox driver. It broke the VM so severely that I had to re-pave. So, I pulled the card, swapped for a Wavlink and created another Win11 Pro VM - to which I assigned the Wavlink card and loaded their drivers. It works flawlessly. Both HDMI displays I had connected were instantly recognized in Device Manager and available. I then shut it down, created another VM with Mint 22.3 and assigned resources exactly per the Windows machine. Installed the Wavlink Ubuntu driver and DisplayLink. Rebooted. Still only see the XOA virtual (Console) screen regardless if I have the VM's VGA option turned on or not. The evdi driver is running as is the DisplayLink service. Looking at Xorg.0.log, I see EDID returns for the two HDMI monitors connected to the Wavlink card along with that from the virtual screen - but no screens other than the virtual one show in Display. Interesting bit: The GT730 card behaves EXACTLY the same way. Works great under Win11, driver initializes correctly in Mint but no display other than virtual. Going to post something on the DisplayLink support forum but find it odd that two different cards behave the same way in two different OSes.

@pkgw Planning an RPU as a whole is a nice idea in theory, but it's inherently designed to co-exist with normal pool management operations (you should still be able to migrate VMs between hosts B and C during it, or start new ones), we are not "pre-booking" memory and other resources for groups of VMs, otherwise it can quickly make the rest of the pool hard to manage while some VM migrations could still be far away. Regarding the narrower case - I'll note this down and see if we can improve things, thank you!

@itservices Hello, I'm taking this issue from Pierre. Did you perform a full restart of the XCP-ng host since the issue occured ? Or just restarted the toolstask ?

The 2026-05-28 build still being good narrows the window a lot more than "sometime since November". What I keep coming back to is the shape of the failure: 14 out of 15, then 5 out of 6, so it is always exactly one that falls over and never the whole run. I don't know whether that points at a per-VM timeout or at something the last task in a run does differently, and someone on the XO team will read that better than me. @pierrebrunet still needs /var/log/xensource.log from the pool master covering one failed run's window, so even a slice from a job where only one VM failed should be enough.