I'm picking this one up alongside the other Body Timeout thread (9002), since they look (to me, at least) like the same underlying issue. Pierre from the XO team is on it and has tested metadata backups on the XO side without hitting it, so the current thinking is it may be coming from the XCP-ng side rather than XO. What would move it along fastest is a /var/log/xensource.log from the pool master covering one failed run's window, so if anyone here can reproduce and grab that slice, it'd really help line the timeout up against what xapi was doing. I know it's especially frustrating when the thing that breaks is your backups, thanks for bearing with us while we chase it. A mention to @Team-Storage might help route the storage-layer angle.

Argh...Correction... 82 shows up as 82:01 and 82:02... lspci -vv -s 0000:82:01.0 82:01.0 PCI bridge: Intel Corporation Device e2f0 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=83, subordinate=83, sec-latency=0 Memory behind bridge: f7c00000-f7dfffff Prefetchable memory behind bridge: 000001f800000000-0000020002ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+ LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport And 82:02... lspci -vv -s 0000:82:02.0 82:02.0 PCI bridge: Intel Corporation Device e2f1 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=84, subordinate=84, sec-latency=0 Memory behind bridge: f7e00000-f7efffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport

Just to give an update. I still am having the issue with current commit 090ce. Regards, Marc

@poddingue said: Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out". The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

@pedro_udifar Hi, We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream This should fix the stream issue and fix the warning.

@olivierlambert said: I'm not sure it's a great idea, because even 64 hosts is huge in terms of VMs and fallout if you have a problem on your pool DB. Even if you drastically improve the current mechanism, the impact of a problem pool wide is far bigger with 64 hosts than 24 for example. It's more than purely tech, it's also a tech design/choice. Exactly, the failure domain just grows exponentially when going beyond a certain collective pool size. I was doing some more research and I found a few possible choices that Tesco may have moved too, but don't want to detract from this forum here.

@laszlobortel Hehe yeah, those are pretty old kernels. I'd say there's a good chance kernel upgrades will go a long way to alleviating the CPU hangs. I can't say it's exclusively a problem with lvmohba storage, that's just what we use because of our previous VMware infrastructure was block storage over fiber channel. We knew a physical infra overhaul wasn't in the cards for us for this migration so we stayed with our existing storage system. This bug bit us half way through the migration until we figured out upgrading the kernel generally fixed it.

@All-Ki Thanks for your work! ProLiant DL360p Gen8 UID Light | 0x01 | ok Sys. Health LED | no reading | ns 01-Inlet Ambient | 33 degrees C | ok 02-CPU 1 | 55 degrees C | ok 03-CPU 2 | 50 degrees C | ok 04-P1 DIMM 1-6 | 45 degrees C | ok 05-P1 DIMM 7-12 | 44 degrees C | ok 06-P2 DIMM 1-6 | 38 degrees C | ok 07-P2 DIMM 7-12 | 43 degrees C | ok 08-P1 Mem Zone | 42 degrees C | ok 09-P1 Mem Zone | 45 degrees C | ok 10-P2 Mem Zone | 39 degrees C | ok 11-P2 Mem Zone | 40 degrees C | ok 12-HD Max | 35 degrees C | ok 13-Chipset 1 | 52 degrees C | ok 14-Chipset1 Zone | 45 degrees C | ok 15-P/S 1 Inlet | 34 degrees C | ok 16-P/S 1 Zone | 40 degrees C | ok 17-P/S 2 Inlet | 42 degrees C | ok 18-P/S 2 Zone | 43 degrees C | ok 19-PCI #1 | disabled | ns 20-PCI #2 | disabled | ns 21-VR P1 | 60 degrees C | ok 22-VR P2 | 56 degrees C | ok 23-VR P1 Mem | 39 degrees C | ok 24-VR P1 Mem | 36 degrees C | ok 25-VR P2 Mem | 32 degrees C | ok 26-VR P2 Mem | 36 degrees C | ok 27-VR P1Mem Zone | 38 degrees C | ok 28-VR P1Mem Zone | 36 degrees C | ok 29-VR P2Mem Zone | 31 degrees C | ok 30-VR P2Mem Zone | 33 degrees C | ok 31-HD Controller | 71 degrees C | ok 32-HD Cntlr Zone | 52 degrees C | ok 33-PCI 1 Zone | 43 degrees C | ok 34-PCI 1 Zone | 46 degrees C | ok 35-LOM Card | 70 degrees C | ok 36-PCI 2 Zone | 50 degrees C | ok 37-System Board | 52 degrees C | ok 38-System Board | 45 degrees C | ok 39-Sys Exhaust | 43 degrees C | ok 40-Sys Exhaust | 46 degrees C | ok 41-Sys Exhaust | 46 degrees C | ok 42-SuperCAP Max | 33 degrees C | ok Fan Block 1 | 94.86 percent | ok Fan Block 2 | 94.86 percent | ok Fan Block 3 | 94.86 percent | ok Fan Block 4 | 94.86 percent | ok Fan Block 5 | 94.86 percent | ok Fan Block 6 | 94.86 percent | ok Fan Block 7 | 94.86 percent | ok Fan Block 8 | 94.86 percent | ok Power Supply 1 | 205 Watts | ok Power Supply 2 | 210 Watts | ok Power Meter | 430 Watts | ok Power Supplies | 0x01 | ok Fans | 0x02 | ok Memory | 0x40 | ok C1 P1I Bay 1 | 0x01 | ok C1 P1I Bay 2 | 0x01 | ok C1 P1I Bay 3 | 0x01 | ok C1 P1I Bay 4 | 0x01 | ok C1 P2I Bay 5 | 0x01 | ok C1 P2I Bay 6 | 0x01 | ok C1 P2I Bay 7 | 0x01 | ok C1 P2I Bay 8 | 0x01 | ok ProLiant DL360 Gen10 UID | 0x01 | ok SysHealth_Stat | 0x01 | ok 01-Inlet Ambient | 19 degrees C | ok 02-CPU 1 | 52 degrees C | ok 03-CPU 2 | 63 degrees C | ok 04-P1 DIMM 1-6 | disabled | ns 05-PMM 1-6 | disabled | ns 06-P1 DIMM 7-12 | 44 degrees C | ok 07-PMM 7-12 | disabled | ns 08-P2 DIMM 1-6 | disabled | ns 09-PMM 1-6 | disabled | ns 10-P2 DIMM 7-12 | 48 degrees C | ok 11-PMM 7-12 | disabled | ns 12-HD Max | 35 degrees C | ok 13-Exp Bay Drive | disabled | ns 14-Stor Batt 1 | 18 degrees C | ok 15-Front Ambient | 24 degrees C | ok 16-VR P1 | 48 degrees C | ok 17-VR P2 | 54 degrees C | ok 18-VR P1 Mem 1 | 31 degrees C | ok 19-VR P1 Mem 2 | 29 degrees C | ok 20-VR P2 Mem 1 | 35 degrees C | ok 21-VR P2 Mem 2 | 36 degrees C | ok 22-Chipset | 42 degrees C | ok 23-BMC | 79 degrees C | ok 24-BMC Zone | 49 degrees C | ok 26-HD Cntlr Zone | 40 degrees C | ok 29-I/O Zone | 37 degrees C | ok 30-PCI 1 | disabled | ns 31-PCI 1 Zone | 45 degrees C | ok 32-PCI 2 | disabled | ns 33-PCI 2 Zone | 44 degrees C | ok 34-PCI 3 | disabled | ns 35-PCI 3 Zone | disabled | ns 37-Rear HD Max | disabled | ns 38-Battery Zone | 43 degrees C | ok 39-P/S 1 Inlet | 42 degrees C | ok 40-P/S 2 Inlet | 48 degrees C | ok 41-P/S 1 | 46 degrees C | ok 42-P/S 2 | 55 degrees C | ok 43-E-Fuse | 45 degrees C | ok 44-P/S 2 Zone | 53 degrees C | ok 49-CPU 1 PkgTmp | 84 degrees C | ok 50-CPU 2 PkgTmp | 94 degrees C | ok 61-AHCI HD Max | disabled | ns 69-PCI 1 M2 | disabled | ns 70-PCI 1 M2 Zn | disabled | ns 71-PCI 2 M2 | disabled | ns 72-PCI 2 M2 Zn | disabled | ns 73-PCI 3 M2 | disabled | ns 74-PCI 3 M2 Zn | disabled | ns Fan 1 | 0x01 | ok Fan 1 DutyCycle | 26.26 percent | ok Fan 1 Presence | 0x02 | ok Fan 2 | 0x01 | ok Fan 2 DutyCycle | 23.52 percent | ok Fan 2 Presence | 0x02 | ok Fan 3 | 0x01 | ok Fan 3 DutyCycle | 23.52 percent | ok Fan 3 Presence | 0x02 | ok Fan 4 | 0x01 | ok Fan 4 DutyCycle | 23.52 percent | ok Fan 4 Presence | 0x02 | ok Fan 5 | 0x01 | ok Fan 5 DutyCycle | 23.52 percent | ok Fan 5 Presence | 0x02 | ok Fan 6 | 0x01 | ok Fan 6 DutyCycle | 24.30 percent | ok Fan 6 Presence | 0x02 | ok Fan 7 | 0x01 | ok Fan 7 DutyCycle | 24.30 percent | ok Fan 7 Presence | 0x02 | ok Power Supply 1 | 0x01 | ok PS 1 Input | 200 Watts | ok Power Supply 2 | 0x01 | ok PS 2 Input | 190 Watts | ok Power Meter | 400 Watts | ok Fans | 0x01 | ok Power Supplies | 0x01 | ok Memory Status | 0x40 | ok Megacell Status | 0x04 | ok Intrusion | Not Readable | ns CPU Utilization | 252 unspecified | ok PS 1 Output | 190 Watts | ok PS_Volt_Out_01 | 12 Volts | ok PS_Volt_In_01 | 205 Volts | ok PS_Curr_Out_01 | 15.80 Amps | ok PS_Curr_In_01 | 1 Amps | ok PS 2 Output | 180 Watts | ok PS_Volt_Out_02 | 12 Volts | ok PS_Volt_In_02 | 204 Volts | ok PS_Curr_Out_02 | 15.10 Amps | ok PS_Curr_In_02 | 0.90 Amps | ok 27.1-LOM-Communi | 68 degrees C | ok 28.1-LOM Card-I/ | 84 degrees C | ok 25.1-HD Controll | 39 degrees C | ok 25.2-HD Controll | 45 degrees C | ok 25.3-HD Controll | 41 degrees C | ok LOM_Link_P1 | 0x02 | ok LOM_Link_P2 | Not Readable | ns LOM_Link_P3 | Not Readable | ns LOM_Link_P4 | Not Readable | ns ALOM_Link_P1 | 0x02 | ok ALOM_Link_P2 | 0x02 | ok Dr_Stat_1I1_B001 | 0x01 | ok Dr_Stat_2I1_B005 | 0x01 | ok CPU_Stat_C1 | 0x80 | ok CPU_Stat_C2 | 0x80 | ok

Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue. EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

@Danp said: Smart Reboot option found on the host's Advanced tab does what you are asking Very nice!

@the_jest Not showen in this picutre but this is where the message would be displayed. Next to the name of the host... [image: 1782931263879-screenshot-2026-07-01-144023.png]

For your storage question, it's fully explained in the doc: https://docs.xcp-ng.org/storage/#-how-to-modify-an-existing-sr-connection And yes, it's planned to get the complete error visible in XO, sadly, it's not "obvious" since the error message isn't returned by XAPI when you try to start but by another method we need to call after it fails ("assert can be started here" from the top of my head). Let me ping @julienXOvates

@pierrebrunet ticket created https://help.vates.tech/#ticket/zoom/59723

Meant to update this again last night - as a road map for those who might try similar at some point: This community's documentation led me to some relevant information on mouse/keyboard pass-through. Namely, if you want this to work you have to pass the entire PCI controller through to the guest(s), Won't work with motherboard devices. But I'm out of PCIe slots. What to do... Simple. Chuck the two serial adapters and install two dual USB 2.0 adapters in their places. Connect the keyboard/mouse to one of the ports and a 4/8/16 port USB to serial adapter to the other. Then pass the entire controller through. In this particular server (R810) I have 5 available PCIe slots. Two will be used for the video cards, one for a 4x NVMe to PCIe adapter (which is eventually where the VMs will reside) and the remaining two for the USB interfaces. This effectively gives me two discrete physical systems, a third available VM instance (say, for network management, monitoring or whatever) and the XOA instance. For application/radio control systems hypervisors (nested) I'll probably have 2ea Win11 and whatever flavor of Linux that supports the GT730 GPU on the system - each assigned the pass-through devices - but only two running at any given time. So no resource contention issues. As far as the displays: I have the main interface of the server connected via SIP to a 2160AS KVM. I'll take 2 more SIPs and use these as the keyboard/mouse inputs for the USB cards. Video...I use an HDMI active splitter on other systems in the network so I'll employ that here. One of the ports on the add-in GPUs goes to the splitter, one of the splitter outputs goes to the SIP via HDMI to VGA adapter and the other feeds the main ceiling mount monitor. The Primary display will be replicated to the KVM screen. Resolution isn't an issue as I have a video scaler inline with the KVM for just this scenario. The concept could be expanded. I'm looking at picking up a T630 and building 4 effective "systems" out of it, following the same principle. The 630 has 8 available PCIe slots. Will update when the next batch of hardware arrives - or I get the Linux side of things sorted. That's next.

@poddingue Just ran one more backup to confirm working, and it fell back to full backup again.... What logs or other information should I look into to find the root cause or do you still think it fall under the known issue? [image: 1783507806440-screenshot_20260708_064922.png]

@julienXOvates Noted, thank you!

PRs upstream are in review

Hello @mgr42 I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng. I am currently (slowly) working on it. There are other references about this topic in the Forum and on GitHub FYI: https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories https://xcp-ng.org/forum/topic/8909/issue-to-load-gpu-passthrough-invalid-pci-rom-header-signature-expecting-0xaa55-got-0x4556 https://github.com/xcp-ng/xcp/issues/806 https://github.com/xapi-project/xen-api/commit/930ffb34e42d674b8234c3d8399fdba97efa6b38 On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table. I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build). I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails). So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR. Regards, Yann Kernel & Hypervisor team timemaster5 created this issue in xcp-ng/xcp open Unable to set romfile for xen-pci-passthrough devices #806 0 freddy77 committed to xapi-project/xen-api CA-418993: Enable PCI ROM BAR before attempting to give permissions to the VM Currently the VM is not able to access the ROM as the host ROM BAR is disabled and not set. Signed-off-by: Frediano Ziglio <frediano.ziglio@citrix.com>