Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"
-
5 minutes is clearly an HTTP timeout, the question is way we are reaching this timeout in the first place.
-
@bogikornel are you talking about another issue? We are on an issue specifically on "pool metadata backups". It seems you are talking about VM backups. If yes, can you send the backup log please?
-
@pierrebrunet This is the same error, but it occurs more often when saving metadata. I'm attaching the logfile. xo-backup-error-log.txt
-
This happens to me mostly with non-config backups but occasionally a config backup will fail. The VM that fails the backup with the body timeout changes around every day, then once a week or so, every VM will get backed up. It only happens on this XO instance and only with the full backups. It doesn't happen on another XO instance that performs delta backups of the same VMs a couple hours later.
-
@olivierlambert I feel like the cause of the timeout is what I posted here: https://xcp-ng.org/forum/post/107454
The incidents of the errors in syslog correspond by matching pool, date and time to the incidents of errors in XO backup. I've traced through several of these incidents in the syslog and matched them to the errors in the XO metadata backup and matched these to missing metadata backup files in the directory on the storage server (ex. 20260807T041000Z) which is created even if the backup fails.
I would look in older logs to see if maybe this has been happening for a long time, possibly before an XO update now shows the error, but my syslog data only goes back 30 days. The XO metadata backup errors started over 30 days.
-
I have a theory that I'd like you to test on a specific branch. It might be an Undici bug, but I'm far from being sure.
You need Node >=22.19 to make it work. Switch to the branch named
fix_undici_paused_parser_crash, build it and try again. -
@olivierlambert Good news. This morning, I built on the
fix_undici_paused_parser_crashbranch and manually ran the metadata backup job 10 times and did NOT get an error. Previously, running the job manually could result in the body timeout error, so we have progress.I will stay on this build during this week, let the scheduled metadata backups run, and report back by this Friday.
-
@christopher-petzel Thank you for testing this! I am also interested in this fix since I encounter this issue from time to time. Unfortunately my testing environment / lab is currently unavailable which is why I wasn't able to compile & deploy this branch myself.
-
@christopher-petzel
Great news!
I will test and report after it is available in a global update.
Thank you Christopher. -
@olivierlambert No metadata backup errors in the 4 scheduled backups since Monday, while on the
fix_undici_paused_parser_crashbranch. Previously, I would not have more that 1 day without an error. Between manual and scheduled backup jobs, I've not been able to recreate the error on this test branch.This is good to see again...
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login