Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

bogikornel

@pierrebrunet This is the same error, but it occurs more often when saving metadata. I'm attaching the logfile. xo-backup-error-log.txt

CodeMercenary

This happens to me mostly with non-config backups but occasionally a config backup will fail. The VM that fails the backup with the body timeout changes around every day, then once a week or so, every VM will get backed up. It only happens on this XO instance and only with the full backups. It doesn't happen on another XO instance that performs delta backups of the same VMs a couple hours later.

christopher-petzel

@olivierlambert I feel like the cause of the timeout is what I posted here: https://xcp-ng.org/forum/post/107454

The incidents of the errors in syslog correspond by matching pool, date and time to the incidents of errors in XO backup. I've traced through several of these incidents in the syslog and matched them to the errors in the XO metadata backup and matched these to missing metadata backup files in the directory on the storage server (ex. 20260807T041000Z) which is created even if the backup fails.

I would look in older logs to see if maybe this has been happening for a long time, possibly before an XO update now shows the error, but my syslog data only goes back 30 days. The XO metadata backup errors started over 30 days.

olivierlambert

I have a theory that I'd like you to test on a specific branch. It might be an Undici bug, but I'm far from being sure.

You need Node >=22.19 to make it work. Switch to the branch named fix_undici_paused_parser_crash, build it and try again.

christopher-petzel

@olivierlambert Good news. This morning, I built on the fix_undici_paused_parser_crash branch and manually ran the metadata backup job 10 times and did NOT get an error. Previously, running the job manually could result in the body timeout error, so we have progress.

I will stay on this build during this week, let the scheduled metadata backups run, and report back by this Friday.

MajorP93

@christopher-petzel Thank you for testing this! I am also interested in this fix since I encounter this issue from time to time. Unfortunately my testing environment / lab is currently unavailable which is why I wasn't able to compile & deploy this branch myself.

GregBinSD

@christopher-petzel
Great news!
I will test and report after it is available in a global update.
Thank you Christopher.

christopher-petzel

@olivierlambert No metadata backup errors in the 4 scheduled backups since Monday, while on the fix_undici_paused_parser_crash branch. Previously, I would not have more that 1 day without an error. Between manual and scheduled backup jobs, I've not been able to recreate the error on this test branch.

This is good to see again...

olivierlambert

Excellent so it was an undici bug after all… Now I'm making sure @Team-XO-Backend won't miss it

JB

@christopher-petzel Unfortunately, for me, it still has an error.

commit c5fae

MajorP93

@JB Yes makes sense. You are not on the branch that already has the fix. Your commit is latest master branch.
Either switch to the branch that got mentioned or wait for the fix to land in master.

JB

@MajorP93 Ok. Thanks!

pierrebrunet

@JB @majorp93 We made a smaller PR to be merged just after the release (because it is linked to almost all XAPI calls), the new branch is this one if you want to test: fix_undici_timeout
Same fix, just a little smaller

christopher-petzel

@pierrebrunet I built on the fix_undici_timeout branch and manually ran the metadata backup job 10 times and did NOT get an error. I will stay on this build through the weekend, let the scheduled metadata backups run, and report back on Monday.

Tim-PT

It looks like this is identified and solved, I'll test on a new version of XO next week at some point, but I just wanted to confirm that I've been seeing what appears to be the same problem.

This hasn't affected any of my full, delta or mirror backups, on any of the NFS targets - but it does affect two different Pool metadata & xo config backups. Both on different NFS targets on different schedules.

It's an infrequent failure, sometimes not failing for a couple of days, but sometimes failing up to 3 or 4 times per day - this is on an hourly schedule.

My most recent failure on the hourly job was yesterday morning, the one before that was almost 24 hours earlier.
This is currently on Xen Orchestra, commit 24913 but it was also noticeable on a version from the end of July.

From what I've read, I'm guessing the fix will be in the main branch soon enough and an update will solve it, but if there's any useful information I can provide, please let me know.

poddingue

Thanks for the feedback so far, folks!

christopher-petzel

@poddingue On the fix_undici_timeout branch this weekend, all scheduled metadata backups completed sucesfully.

poddingue

That's great news, thanks, @christopher-petzel !

JB

commit 9cdaf

GregBinSD

Here is my story:
I responded to the notice regarding the latest update of the main branch, Master, commit 9cdaf, by updating the XO-CE vm, rebooting the XO-CE vm, bringing all 3 XCP-ng hosts on line, and then running the XO-CE GUI and selecting Pool Rolling Update.
The rolling update cycled through all 3 hosts in the pool.
I then attempted to run a pool metadata and xo_config backup.
It failed by timing out at 5 minutes.
I decided to shutdown the pool master, so I transferred all VMs to the other 2 hosts and shutdown the master for 15 seconds. I powered on the master, and transferred the VMs back to the master.
I shutdown the other 2 hosts, and then powered them back on. All hosts are indicating "green" status.
At this point I attempted to run the pool metadata and xo_config backup, and it ran successfully.
I waited 2 minutes and ran it again, with success again!
Lastly, I powered down the other 2 hosts, and with the VMs all running on the master, I ran the pool metadata and xo_config backup again, with apparent success.
If someone can tell me the location of the logs that recorded this operation, I will look at them closely to confirm that the pool metadata and xo_config backups actually took place.
Thank you.