@olivierlambert No metadata backup errors in the 4 scheduled backups since Monday, while on the fix_undici_paused_parser_crash branch. Previously, I would not have more that 1 day without an error. Between manual and scheduled backup jobs, I've not been able to recreate the error on this test branch.

This is good to see again...

@olivierlambert Good news. This morning, I built on the fix_undici_paused_parser_crash branch and manually ran the metadata backup job 10 times and did NOT get an error. Previously, running the job manually could result in the body timeout error, so we have progress.

I will stay on this build during this week, let the scheduled metadata backups run, and report back by this Friday.

@olivierlambert I feel like the cause of the timeout is what I posted here: https://xcp-ng.org/forum/post/107454

The incidents of the errors in syslog correspond by matching pool, date and time to the incidents of errors in XO backup. I've traced through several of these incidents in the syslog and matched them to the errors in the XO metadata backup and matched these to missing metadata backup files in the directory on the storage server (ex. 20260807T041000Z) which is created even if the backup fails.

I would look in older logs to see if maybe this has been happening for a long time, possibly before an XO update now shows the error, but my syslog data only goes back 30 days. The XO metadata backup errors started over 30 days.

@florent suggesting this looks like a network issue, I took a look at my XO's server's syslog and found the error when this occurs.

2026-07-31T00:10:00.428083-04:00 mis-26-svr xo-server[1298]: 2026-07-31T04:10:00.427Z xo:xo-server ERROR uncaught exception {
2026-07-31T00:10:00.428186-04:00 mis-26-svr xo-server[1298]:   error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
2026-07-31T00:10:00.428262-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428309-04:00 mis-26-svr xo-server[1298]:     assert(!this.paused)
2026-07-31T00:10:00.428341-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428391-04:00 mis-26-svr xo-server[1298]:       at Parser.finish (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:302:5)
2026-07-31T00:10:00.428414-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:741:32)
2026-07-31T00:10:00.428434-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.emit (node:events:521:24)
2026-07-31T00:10:00.428478-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607200851/@xen-orchestra/log/configure.js:52:17)
2026-07-31T00:10:00.428508-04:00 mis-26-svr xo-server[1298]:       at endReadableNT (node:internal/streams/readable:1729:12)
2026-07-31T00:10:00.428525-04:00 mis-26-svr xo-server[1298]:       at processTicksAndRejections (node:internal/process/task_queues:90:21) {
2026-07-31T00:10:00.428549-04:00 mis-26-svr xo-server[1298]:     generatedMessage: true,
2026-07-31T00:10:00.428567-04:00 mis-26-svr xo-server[1298]:     code: 'ERR_ASSERTION',
2026-07-31T00:10:00.428582-04:00 mis-26-svr xo-server[1298]:     actual: false,
2026-07-31T00:10:00.428599-04:00 mis-26-svr xo-server[1298]:     expected: true,
2026-07-31T00:10:00.428614-04:00 mis-26-svr xo-server[1298]:     operator: '==',
2026-07-31T00:10:00.428628-04:00 mis-26-svr xo-server[1298]:     diff: 'simple'
2026-07-31T00:10:00.428642-04:00 mis-26-svr xo-server[1298]:   }
2026-07-31T00:10:00.428659-04:00 mis-26-svr xo-server[1298]: }

So it looks like xo-server is dying, could it be the module undici that the problem here? Keep in mind that 13 other instances of xo-server performed the backup without a problem at the exact same time. Some days this never happens, some days it could be 4 instances of xo-server that die the same way in executing the backup.

I think it's also important to note that the instance of xo-server dying happens immediately upon the backup being started at 10 minutes after midnight. I would think that if the problem were the network itself, that there would be some timeout period before causing the xo-server instance to crash.

To answer a question asked of @jb , this is a local XCP-ng management network and there are no other backups occurring at 00:10.

@poddingue Maybe this log will help @pierrebrunet . I've been having the Body Timeout Error for a couple of weeks on Metadata/Config backups. I'm using XO from sources. The problem started after upgrading to commit 63f8d. I was previously at commit e6443. The errors will occur for one or more hosts, and which host(s) has the error seems to be random. The error will occur for hosts which have VMs and for hosts that have no VMs. Attached is this morning's xensource.log from 00:10 when the backup started. The backup ends at 00:15 but I've included log data through 00:20. xensource-truncated.log.txt

I have reverted to a snapshot of XO running at commit e6443 and executed multiple Metadata/Config backups without any problem.

@acebmxer I was just creating a post on this myself. I believe this is because /v5 doesn't exist without building it from sources, in the same way that previously you had to build /v6 separately when v5 was the default.

The /v5 tries to load /v5/rest/v0/gui-routes but gets a 500 error

Based on the API documentation, /gui-routes seems to be an XOA thing, so XO from sources won't have it without a separate build?

Someone please correct me if I'm wrong.

I believe I have the definitive cause for this 'random host reboot' issue.

After 6 months of problem-free operation, I have experienced the host reboot issue again on this server. The host was running only Linux VMs, so the theory of Windows VMs on the host contributing to the reboot issue has proven false. As with each time before, there are no indications in any relevant log files that the host is going to reboot. I think at this point I can definitively say that the reboot is caused by a faulty SuperMicro motherboard.

I've learned my lesson: use HPE servers! This SuperMicro system will be melted down for scrap.

@splastunov Yes, all VMs are for in-house use and all were built by me personally.

I have previously followed the same steps that you followed in your case. I updated the BIOS on the host server and moved VMs one by one.

Moving VMs one by one is how I eventually found that I only had the problem when a Windows Server VM was on the host. When I had this problem occur with a fresh Windows Server 2022 VM which had no applications installed, I started to suspect that it was related to Windows. I was then able to confirm that this only occurred with Windows VMs.

Thanks for the info. I think these are great steps toward finding the problem.

Since I last posted on this topic, I've found that the random reboots only occur when there are Windows Server VMs on the host (Tested with 2019 and 2022). The issue will not occur when running Linux VMs.

My issue seems very similar to the problem described (and solved) in https://xcp-ng.org/forum/topic/6683/windows-server-2019-sporadic-reboot/7

The difference is that in my case, the host restarted and in the other post, the poster reports that the VMs are restarting. Since the poster also tested RAM and found no problems but was able to solve the issue by replacing a suspected DIMM, that information may be useful in the host reboot scenario that I experience.

FYI, I have not replaced the RAM yet and may not actually do it since the server in question is aging and will likely be replaced (with HP hardware) soon.

Secure Boot was not enabled in the VM. I enabled Secure Boot in the VM and was able to install KB5012170 without any problem.

I tested further to see if there were any issues related to enabling then disabling Secure Boot in the VM. I did not experience any problems booting the VM after disabling Secure Boot. There were no problems booting the VM after moving it to a pool where the default UEFI Certificates had not been installed.

For anyone wanting to resolve the KB5012170 update error, here are the steps I took:

• On the pool/host for the VM, install the UEFI Certificates with secureboot-certs install
• Shut down the problem VM
• Enable Secure Boot on the VM. I do this via Xen Orchestra but it can also be done with xe vm-param-set uuid=[uuid of VM] platform:secureboot=true
• Boot the VM
• Apply the KB5012170 update
• Shut down the VM
• Disable Secure Boot on the VM via XO or xe vm-param-set uuid=[uuid of VM] platform:secureboot=false
• Boot the VM

https://xcp-ng.org/docs/guides.html#guest-uefi-secure-boot is a very thorough guide on Secure Boot in XCP-ng.

Thanks for the help @stormi

I've not been able to find information on the signature used by the UEFI bootloader and if that is on the DBX update in KB5012170. Since my original post, Microsoft has updated the Known Issues documentation for KB5012170 and it seems that this problem is now 'known' and has a proposed resolution of, "We are presently investigating and will provide an update in an upcoming release."

So at this point, it appears Microsoft is investigating this as an issue Microsoft needs to resolve, not an issue with the bootloader itself.

Reference: https://support.microsoft.com/en-us/topic/kb5012170-security-update-for-secure-boot-dbx-august-9-2022-72ff5eed-25b4-47c7-be28-c42bd211bb15
(see the third issue listed in Known Issues, which didn't exist until after my initial post)

I'm attempting to determine if a problem I'm having with installing Windows 10 update KB5012170 is related to the UEFI bootloader used by the VM.

XCP-ng version is 8.2.1 and is nearly fully patched (have not installed 4.13.4-9.24.1.xcpng8.2 yet).

I started noticing 4 Windows 10 VMs (on separate hosts) rebooting every night for the past 5 days and found that KB5012170 was not installing because of error 0x800f0922. I've found that this is probably because of the UEFI bootloader now being in the Secure Boot DBX (Forbidden Signature Database). I'm basing this on this this article: https://www.bleepingcomputer.com/news/security/windows-kb5012170-secure-boot-dbx-update-may-fail-with-0x800f0922-error/

Apparently if the UEFI bootloader is signed by keys from one of three vendors (New Horizon Datasys Inc, CryptoPro Secure Disk, Eurosoft (UK) Ltd) then KB5012170 will fail to install with error 0x800f0922 because these signatures have been blacklisted.

I know the VMs are using the TianoCore UEFI implementation. I've looked at the TianoCore bugtracker but didn't find anything related - although if issue is in fact caused by Microsoft blacklisting the signature, I don't know that would even be a bug, per se.

@olivierlambert Just tell people to stick with HP hardware This problem server is a SuperMicro system board and it's the second of the same model of which I've had a hardware problem. The other board stopped working completely so it was a different failure mode. Once I obsolete this hardware, I will have no more SuperMicro boards in production.

@olivierlambert I have been able to confirm this is a hardware reboot. Since I've been working this issue for a year and the restarts were so rare, at some point I convinced myself that the hardware was not restarting even thought my monitoring and logging was telling me otherwise.

Thanks for your help in guiding me to reconsider what I thought I already knew. Thankfully the restarts have become more frequent and I have had 3 reboots in 10 days. That frequency has allowed me to catch what was really happening.

I was wrong about the hypervisor, it is restarting. I confused myself and didn't make the connection.

In /var/log/xen/hypervisor.log... I see an entry Logfile Opened with the timestamp of when the log rotates then I see another Logfile Opened at the timestamp that the hypervisor restarts, followed by the Xen log data during boot.

So I guess I need to be thinking about why the hypervisor is restarting. Now I'm questioning if the hardware is restarting. I have not seen a hardware restart in the IPMI data and the recovery time seemed too short for a hardware restart HOWEVER the lack of evidence is not evidence itself so I think my next move will be to monitor the hardware in a way that I can confirm or deny a hardware restart.

Thanks for your help @olivierlambert . It may be a couple of months before this happens again but I'll report back what I find once it happens.

I was able to run Memtest for 86 hours, completed 9 passes and had no memory errors.

Maybe I'm approaching this incorrectly. I've been assuming there is a problem in dom0 since the kernel is starting without any indication as to why. What could be telling the kernel to restart? The hardware never restarts and since dom0 restarts in a very short period of time (seconds, from best I can tell), the hypervisor seems to keep running.

I have a very limited knowledge of this stack so I know I could be completely wrong.

@olivierlambert Sadly, nothing showing but the restart. I'll run Memtest. I think I did this at some point but I don't have a record of doing it and it's not a bad idea anyway.

I have an XCP-ng installation (8.2.1, all patches but most recent) which will restart at random intervals. Usually this interval is a couple months but has been as short as a week. This started just over one year ago. This server has been running since 2018 (with XCP-ng upgrades). The server is in a single host pool.

This isn't a normal crash. A kernel panic does not occur. There is no indication of a shutdown. The kernel just stops then is booting a couple of seconds later.
Kdump is working but there is no logging from kdump when this happens. I can force a kernel panic and I get logging by kdump when I force it so I know kdump is working.

I would expect this to be a hardware issue however the hardware does not restart. The hardware remains running. The kernel will restart. I know this by monitoring hardware, kernel uptime and reviewing log data.

There is no consistency in time of day or day of week. This usually occurs when the one VM on the server is idle.

I'm unable to find any indication in any log that something's gone wrong. I only can find the kernel restarting.

I've tried many hardware configurations, updated firmware on the system board, and RAID controller over the past year and continue to have the same results. I have re-installed XCP-ng and also have experienced the same issue through various patches applied though the past year.

If there is a way that this could be caused by hardware without leaving any trace and not rebooting the hardware, I don't know what that could be but I'd be happy to hear any ideas.

Does anyone have any thoughts on what I could monitor or what I might look into? The one thing I've not done is move the one VM on the host to another host. I don't suspect the VM itself is the cause because there is usually no load on the VM when the restart occurs. There are licensing entanglements which result in about 24 hours of downtime and require a re-install of software though the provider's support if I move the VM - so I've not done this for testing.