Thirty to forty seconds of clock drift after a power outage, and then the backup goes through once you fix the time. That could be coincidence, but it might not be. I don't know enough about how the XAPI client handles time skew to say whether it would surface as a BodyTimeoutError rather than an auth or TLS failure, so I could easily be wrong here. If it fails again, could you look at what the host clock is doing before you correct it? @pierrebrunet from the XO team is already on this one, and a second data point either way would tell him whether it's worth pulling on. Thanks for putting the logs on the pastebin, much easier to read that way.

Argh...Correction... 82 shows up as 82:01 and 82:02... lspci -vv -s 0000:82:01.0 82:01.0 PCI bridge: Intel Corporation Device e2f0 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=83, subordinate=83, sec-latency=0 Memory behind bridge: f7c00000-f7dfffff Prefetchable memory behind bridge: 000001f800000000-0000020002ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+ LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport And 82:02... lspci -vv -s 0000:82:02.0 82:02.0 PCI bridge: Intel Corporation Device e2f1 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=84, subordinate=84, sec-latency=0 Memory behind bridge: f7e00000-f7efffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport

for future reference here is the xl dmesg at this point xldmesg.txt

The 2026-05-28 build still being good narrows the window a lot more than "sometime since November". What I keep coming back to is the shape of the failure: 14 out of 15, then 5 out of 6, so it is always exactly one that falls over and never the whole run. I don't know whether that points at a per-VM timeout or at something the last task in a run does differently, and someone on the XO team will read that better than me. @pierrebrunet still needs /var/log/xensource.log from the pool master covering one failed run's window, so even a slice from a job where only one VM failed should be enough.

@itservices Hello, I'm taking this issue from Pierre. Did you perform a full restart of the XCP-ng host since the issue occured ? Or just restarted the toolstask ?

@pedro_udifar Hi, We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream This should fix the stream issue and fix the warning.

@olivierlambert said: I'm not sure it's a great idea, because even 64 hosts is huge in terms of VMs and fallout if you have a problem on your pool DB. Even if you drastically improve the current mechanism, the impact of a problem pool wide is far bigger with 64 hosts than 24 for example. It's more than purely tech, it's also a tech design/choice. Exactly, the failure domain just grows exponentially when going beyond a certain collective pool size. I was doing some more research and I found a few possible choices that Tesco may have moved too, but don't want to detract from this forum here.

@laszlobortel Hehe yeah, those are pretty old kernels. I'd say there's a good chance kernel upgrades will go a long way to alleviating the CPU hangs. I can't say it's exclusively a problem with lvmohba storage, that's just what we use because of our previous VMware infrastructure was block storage over fiber channel. We knew a physical infra overhaul wasn't in the cards for us for this migration so we stayed with our existing storage system. This bug bit us half way through the migration until we figured out upgrading the kernel generally fixed it.

Indeed, it should be detected/reported correctly to avoid losing time finding the problem. Let me ping @julienXOVates

@Danp said: Smart Reboot option found on the host's Advanced tab does what you are asking Very nice!

@the_jest Not showen in this picutre but this is where the message would be displayed. Next to the name of the host... [image: 1782931263879-screenshot-2026-07-01-144023.png]

For your storage question, it's fully explained in the doc: https://docs.xcp-ng.org/storage/#-how-to-modify-an-existing-sr-connection And yes, it's planned to get the complete error visible in XO, sadly, it's not "obvious" since the error message isn't returned by XAPI when you try to start but by another method we need to call after it fails ("assert can be started here" from the top of my head). Let me ping @julienXOvates

@pierrebrunet ticket created https://help.vates.tech/#ticket/zoom/59723

Meant to update this again last night - as a road map for those who might try similar at some point: This community's documentation led me to some relevant information on mouse/keyboard pass-through. Namely, if you want this to work you have to pass the entire PCI controller through to the guest(s), Won't work with motherboard devices. But I'm out of PCIe slots. What to do... Simple. Chuck the two serial adapters and install two dual USB 2.0 adapters in their places. Connect the keyboard/mouse to one of the ports and a 4/8/16 port USB to serial adapter to the other. Then pass the entire controller through. In this particular server (R810) I have 5 available PCIe slots. Two will be used for the video cards, one for a 4x NVMe to PCIe adapter (which is eventually where the VMs will reside) and the remaining two for the USB interfaces. This effectively gives me two discrete physical systems, a third available VM instance (say, for network management, monitoring or whatever) and the XOA instance. For application/radio control systems hypervisors (nested) I'll probably have 2ea Win11 and whatever flavor of Linux that supports the GT730 GPU on the system - each assigned the pass-through devices - but only two running at any given time. So no resource contention issues. As far as the displays: I have the main interface of the server connected via SIP to a 2160AS KVM. I'll take 2 more SIPs and use these as the keyboard/mouse inputs for the USB cards. Video...I use an HDMI active splitter on other systems in the network so I'll employ that here. One of the ports on the add-in GPUs goes to the splitter, one of the splitter outputs goes to the SIP via HDMI to VGA adapter and the other feeds the main ceiling mount monitor. The Primary display will be replicated to the KVM screen. Resolution isn't an issue as I have a video scaler inline with the KVM for just this scenario. The concept could be expanded. I'm looking at picking up a T630 and building 4 effective "systems" out of it, following the same principle. The 630 has 8 available PCIe slots. Will update when the next batch of hardware arrives - or I get the Linux side of things sorted. That's next.

@pkgw Planning an RPU as a whole is a nice idea in theory, but it's inherently designed to co-exist with normal pool management operations (you should still be able to migrate VMs between hosts B and C during it, or start new ones), we are not "pre-booking" memory and other resources for groups of VMs, otherwise it can quickly make the rest of the pool hard to manage while some VM migrations could still be far away. Regarding the narrower case - I'll note this down and see if we can improve things, thank you!

@All-Ki Thanks for your work! ProLiant DL360p Gen8 UID Light | 0x01 | ok Sys. Health LED | no reading | ns 01-Inlet Ambient | 33 degrees C | ok 02-CPU 1 | 55 degrees C | ok 03-CPU 2 | 50 degrees C | ok 04-P1 DIMM 1-6 | 45 degrees C | ok 05-P1 DIMM 7-12 | 44 degrees C | ok 06-P2 DIMM 1-6 | 38 degrees C | ok 07-P2 DIMM 7-12 | 43 degrees C | ok 08-P1 Mem Zone | 42 degrees C | ok 09-P1 Mem Zone | 45 degrees C | ok 10-P2 Mem Zone | 39 degrees C | ok 11-P2 Mem Zone | 40 degrees C | ok 12-HD Max | 35 degrees C | ok 13-Chipset 1 | 52 degrees C | ok 14-Chipset1 Zone | 45 degrees C | ok 15-P/S 1 Inlet | 34 degrees C | ok 16-P/S 1 Zone | 40 degrees C | ok 17-P/S 2 Inlet | 42 degrees C | ok 18-P/S 2 Zone | 43 degrees C | ok 19-PCI #1 | disabled | ns 20-PCI #2 | disabled | ns 21-VR P1 | 60 degrees C | ok 22-VR P2 | 56 degrees C | ok 23-VR P1 Mem | 39 degrees C | ok 24-VR P1 Mem | 36 degrees C | ok 25-VR P2 Mem | 32 degrees C | ok 26-VR P2 Mem | 36 degrees C | ok 27-VR P1Mem Zone | 38 degrees C | ok 28-VR P1Mem Zone | 36 degrees C | ok 29-VR P2Mem Zone | 31 degrees C | ok 30-VR P2Mem Zone | 33 degrees C | ok 31-HD Controller | 71 degrees C | ok 32-HD Cntlr Zone | 52 degrees C | ok 33-PCI 1 Zone | 43 degrees C | ok 34-PCI 1 Zone | 46 degrees C | ok 35-LOM Card | 70 degrees C | ok 36-PCI 2 Zone | 50 degrees C | ok 37-System Board | 52 degrees C | ok 38-System Board | 45 degrees C | ok 39-Sys Exhaust | 43 degrees C | ok 40-Sys Exhaust | 46 degrees C | ok 41-Sys Exhaust | 46 degrees C | ok 42-SuperCAP Max | 33 degrees C | ok Fan Block 1 | 94.86 percent | ok Fan Block 2 | 94.86 percent | ok Fan Block 3 | 94.86 percent | ok Fan Block 4 | 94.86 percent | ok Fan Block 5 | 94.86 percent | ok Fan Block 6 | 94.86 percent | ok Fan Block 7 | 94.86 percent | ok Fan Block 8 | 94.86 percent | ok Power Supply 1 | 205 Watts | ok Power Supply 2 | 210 Watts | ok Power Meter | 430 Watts | ok Power Supplies | 0x01 | ok Fans | 0x02 | ok Memory | 0x40 | ok C1 P1I Bay 1 | 0x01 | ok C1 P1I Bay 2 | 0x01 | ok C1 P1I Bay 3 | 0x01 | ok C1 P1I Bay 4 | 0x01 | ok C1 P2I Bay 5 | 0x01 | ok C1 P2I Bay 6 | 0x01 | ok C1 P2I Bay 7 | 0x01 | ok C1 P2I Bay 8 | 0x01 | ok ProLiant DL360 Gen10 UID | 0x01 | ok SysHealth_Stat | 0x01 | ok 01-Inlet Ambient | 19 degrees C | ok 02-CPU 1 | 52 degrees C | ok 03-CPU 2 | 63 degrees C | ok 04-P1 DIMM 1-6 | disabled | ns 05-PMM 1-6 | disabled | ns 06-P1 DIMM 7-12 | 44 degrees C | ok 07-PMM 7-12 | disabled | ns 08-P2 DIMM 1-6 | disabled | ns 09-PMM 1-6 | disabled | ns 10-P2 DIMM 7-12 | 48 degrees C | ok 11-PMM 7-12 | disabled | ns 12-HD Max | 35 degrees C | ok 13-Exp Bay Drive | disabled | ns 14-Stor Batt 1 | 18 degrees C | ok 15-Front Ambient | 24 degrees C | ok 16-VR P1 | 48 degrees C | ok 17-VR P2 | 54 degrees C | ok 18-VR P1 Mem 1 | 31 degrees C | ok 19-VR P1 Mem 2 | 29 degrees C | ok 20-VR P2 Mem 1 | 35 degrees C | ok 21-VR P2 Mem 2 | 36 degrees C | ok 22-Chipset | 42 degrees C | ok 23-BMC | 79 degrees C | ok 24-BMC Zone | 49 degrees C | ok 26-HD Cntlr Zone | 40 degrees C | ok 29-I/O Zone | 37 degrees C | ok 30-PCI 1 | disabled | ns 31-PCI 1 Zone | 45 degrees C | ok 32-PCI 2 | disabled | ns 33-PCI 2 Zone | 44 degrees C | ok 34-PCI 3 | disabled | ns 35-PCI 3 Zone | disabled | ns 37-Rear HD Max | disabled | ns 38-Battery Zone | 43 degrees C | ok 39-P/S 1 Inlet | 42 degrees C | ok 40-P/S 2 Inlet | 48 degrees C | ok 41-P/S 1 | 46 degrees C | ok 42-P/S 2 | 55 degrees C | ok 43-E-Fuse | 45 degrees C | ok 44-P/S 2 Zone | 53 degrees C | ok 49-CPU 1 PkgTmp | 84 degrees C | ok 50-CPU 2 PkgTmp | 94 degrees C | ok 61-AHCI HD Max | disabled | ns 69-PCI 1 M2 | disabled | ns 70-PCI 1 M2 Zn | disabled | ns 71-PCI 2 M2 | disabled | ns 72-PCI 2 M2 Zn | disabled | ns 73-PCI 3 M2 | disabled | ns 74-PCI 3 M2 Zn | disabled | ns Fan 1 | 0x01 | ok Fan 1 DutyCycle | 26.26 percent | ok Fan 1 Presence | 0x02 | ok Fan 2 | 0x01 | ok Fan 2 DutyCycle | 23.52 percent | ok Fan 2 Presence | 0x02 | ok Fan 3 | 0x01 | ok Fan 3 DutyCycle | 23.52 percent | ok Fan 3 Presence | 0x02 | ok Fan 4 | 0x01 | ok Fan 4 DutyCycle | 23.52 percent | ok Fan 4 Presence | 0x02 | ok Fan 5 | 0x01 | ok Fan 5 DutyCycle | 23.52 percent | ok Fan 5 Presence | 0x02 | ok Fan 6 | 0x01 | ok Fan 6 DutyCycle | 24.30 percent | ok Fan 6 Presence | 0x02 | ok Fan 7 | 0x01 | ok Fan 7 DutyCycle | 24.30 percent | ok Fan 7 Presence | 0x02 | ok Power Supply 1 | 0x01 | ok PS 1 Input | 200 Watts | ok Power Supply 2 | 0x01 | ok PS 2 Input | 190 Watts | ok Power Meter | 400 Watts | ok Fans | 0x01 | ok Power Supplies | 0x01 | ok Memory Status | 0x40 | ok Megacell Status | 0x04 | ok Intrusion | Not Readable | ns CPU Utilization | 252 unspecified | ok PS 1 Output | 190 Watts | ok PS_Volt_Out_01 | 12 Volts | ok PS_Volt_In_01 | 205 Volts | ok PS_Curr_Out_01 | 15.80 Amps | ok PS_Curr_In_01 | 1 Amps | ok PS 2 Output | 180 Watts | ok PS_Volt_Out_02 | 12 Volts | ok PS_Volt_In_02 | 204 Volts | ok PS_Curr_Out_02 | 15.10 Amps | ok PS_Curr_In_02 | 0.90 Amps | ok 27.1-LOM-Communi | 68 degrees C | ok 28.1-LOM Card-I/ | 84 degrees C | ok 25.1-HD Controll | 39 degrees C | ok 25.2-HD Controll | 45 degrees C | ok 25.3-HD Controll | 41 degrees C | ok LOM_Link_P1 | 0x02 | ok LOM_Link_P2 | Not Readable | ns LOM_Link_P3 | Not Readable | ns LOM_Link_P4 | Not Readable | ns ALOM_Link_P1 | 0x02 | ok ALOM_Link_P2 | 0x02 | ok Dr_Stat_1I1_B001 | 0x01 | ok Dr_Stat_2I1_B005 | 0x01 | ok CPU_Stat_C1 | 0x80 | ok CPU_Stat_C2 | 0x80 | ok

PRs upstream are in review

Hello @mgr42 I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng. I am currently (slowly) working on it. There are other references about this topic in the Forum and on GitHub FYI: https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories https://xcp-ng.org/forum/topic/8909/issue-to-load-gpu-passthrough-invalid-pci-rom-header-signature-expecting-0xaa55-got-0x4556 https://github.com/xcp-ng/xcp/issues/806 https://github.com/xapi-project/xen-api/commit/930ffb34e42d674b8234c3d8399fdba97efa6b38 On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table. I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build). I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails). So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR. Regards, Yann Kernel & Hypervisor team timemaster5 created this issue in xcp-ng/xcp open Unable to set romfile for xen-pci-passthrough devices #806 0 freddy77 committed to xapi-project/xen-api CA-418993: Enable PCI ROM BAR before attempting to give permissions to the VM Currently the VM is not able to access the ROM as the host ROM BAR is disabled and not set. Signed-off-by: Frediano Ziglio <frediano.ziglio@citrix.com>