Thanks for checking. That's useful, even if it points away from where I was looking. Clocks within a second of each other means drift probably isn't your problem, and I'd guess the MST/UTC difference is just how dom0 displays it, though I'm not sure. What I keep coming back to is that your full backups fail while the delta jobs on the same hosts never do. That's the same split in https://github.com/vatesfr/xen-orchestra/issues/9181, where full backups hit BodyTimeoutError on VMs with big disks or a lot of free space and the deltas are fine. If your failing VMs look like that, your dates and the MST/UTC detail would do more good on that issue than buried in here. MajorP93 created this issue in vatesfr/xen-orchestra open Full backup fails on large VMs / VMs with lots of free disk space #9181

@ravenet well... thanks for the idea. I went ahead and tried spinning up an opensuse slowroll VM. I installed docker and the amdgpu firmware/kernel drivers, and tried running lemonade-server and ollama and same behavior. So that definitely seems to point to something weird going on on my host rather than the guest software stack...

Argh...Correction... 82 shows up as 82:01 and 82:02... lspci -vv -s 0000:82:01.0 82:01.0 PCI bridge: Intel Corporation Device e2f0 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=83, subordinate=83, sec-latency=0 Memory behind bridge: f7c00000-f7dfffff Prefetchable memory behind bridge: 000001f800000000-0000020002ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+ LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport And 82:02... lspci -vv -s 0000:82:02.0 82:02.0 PCI bridge: Intel Corporation Device e2f1 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=84, subordinate=84, sec-latency=0 Memory behind bridge: f7e00000-f7efffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport

The 2026-05-28 build still being good narrows the window a lot more than "sometime since November". What I keep coming back to is the shape of the failure: 14 out of 15, then 5 out of 6, so it is always exactly one that falls over and never the whole run. I don't know whether that points at a per-VM timeout or at something the last task in a run does differently, and someone on the XO team will read that better than me. @pierrebrunet still needs /var/log/xensource.log from the pool master covering one failed run's window, so even a slice from a job where only one VM failed should be enough.

@florent Sorry, missed your previous post. I will return to master and test it (over the weekend). Thank you!

@olivierlambert said: I'm not sure it's a great idea, because even 64 hosts is huge in terms of VMs and fallout if you have a problem on your pool DB. Even if you drastically improve the current mechanism, the impact of a problem pool wide is far bigger with 64 hosts than 24 for example. It's more than purely tech, it's also a tech design/choice. Exactly, the failure domain just grows exponentially when going beyond a certain collective pool size. I was doing some more research and I found a few possible choices that Tesco may have moved too, but don't want to detract from this forum here.

@itservices Hello, I'm taking this issue from Pierre. Did you perform a full restart of the XCP-ng host since the issue occured ? Or just restarted the toolstask ?

@laszlobortel Hehe yeah, those are pretty old kernels. I'd say there's a good chance kernel upgrades will go a long way to alleviating the CPU hangs. I can't say it's exclusively a problem with lvmohba storage, that's just what we use because of our previous VMware infrastructure was block storage over fiber channel. We knew a physical infra overhaul wasn't in the cards for us for this migration so we stayed with our existing storage system. This bug bit us half way through the migration until we figured out upgrading the kernel generally fixed it.

Indeed, it should be detected/reported correctly to avoid losing time finding the problem. Let me ping @julienXOVates

@Danp said: Smart Reboot option found on the host's Advanced tab does what you are asking Very nice!

Hardware update: Got a couple M9148 (4 port Matrox DP) and Wavlink WL-UG7500DH (4 port DP/HDMI) in to test. Configured a Win11 Pro VM with the Matrox driver. It broke the VM so severely that I had to re-pave. So, I pulled the card, swapped for a Wavlink and created another Win11 Pro VM - to which I assigned the Wavlink card and loaded their drivers. It works flawlessly. Both HDMI displays I had connected were instantly recognized in Device Manager and available. I then shut it down, created another VM with Mint 22.3 and assigned resources exactly per the Windows machine. Installed the Wavlink Ubuntu driver and DisplayLink. Rebooted. Still only see the XOA virtual (Console) screen regardless if I have the VM's VGA option turned on or not. The evdi driver is running as is the DisplayLink service. Looking at Xorg.0.log, I see EDID returns for the two HDMI monitors connected to the Wavlink card along with that from the virtual screen - but no screens other than the virtual one show in Display. Interesting bit: The GT730 card behaves EXACTLY the same way. Works great under Win11, driver initializes correctly in Mint but no display other than virtual. Going to post something on the DisplayLink support forum but find it odd that two different cards behave the same way in two different OSes.

@the_jest Not showen in this picutre but this is where the message would be displayed. Next to the name of the host... [image: 1782931263879-screenshot-2026-07-01-144023.png]

For your storage question, it's fully explained in the doc: https://docs.xcp-ng.org/storage/#-how-to-modify-an-existing-sr-connection And yes, it's planned to get the complete error visible in XO, sadly, it's not "obvious" since the error message isn't returned by XAPI when you try to start but by another method we need to call after it fails ("assert can be started here" from the top of my head). Let me ping @julienXOvates

@pierrebrunet ticket created https://help.vates.tech/#ticket/zoom/59723

@pkgw Planning an RPU as a whole is a nice idea in theory, but it's inherently designed to co-exist with normal pool management operations (you should still be able to migrate VMs between hosts B and C during it, or start new ones), we are not "pre-booking" memory and other resources for groups of VMs, otherwise it can quickly make the rest of the pool hard to manage while some VM migrations could still be far away. Regarding the narrower case - I'll note this down and see if we can improve things, thank you!

OK so that resource was a pain and did not want to give up, so I restarted the host. Migration is running good now.

PRs upstream are in review

Hello @mgr42 I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng. I am currently (slowly) working on it. There are other references about this topic in the Forum and on GitHub FYI: https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories https://xcp-ng.org/forum/topic/8909/issue-to-load-gpu-passthrough-invalid-pci-rom-header-signature-expecting-0xaa55-got-0x4556 https://github.com/xcp-ng/xcp/issues/806 https://github.com/xapi-project/xen-api/commit/930ffb34e42d674b8234c3d8399fdba97efa6b38 On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table. I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build). I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails). So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR. Regards, Yann Kernel & Hypervisor team timemaster5 created this issue in xcp-ng/xcp open Unable to set romfile for xen-pci-passthrough devices #806 0 freddy77 committed to xapi-project/xen-api CA-418993: Enable PCI ROM BAR before attempting to give permissions to the VM Currently the VM is not able to access the ROM as the host ROM BAR is disabled and not set. Signed-off-by: Frediano Ziglio <frediano.ziglio@citrix.com>