Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

[christopher-petzel]

@olivierlambert No metadata backup errors in the 4 scheduled backups since Monday, while on the fix_undici_paused_parser_crash branch. Previously, I would not have more that 1 day without an error. Between manual and scheduled backup jobs, I've not been able to recreate the error on this test branch.

This is good to see again...

[olivierlambert]

Excellent so it was an undici bug after all… Now I'm making sure @Team-XO-Backend won't miss it

[JB]

@christopher-petzel Unfortunately, for me, it still has an error.

commit c5fae

[MajorP93]

@JB Yes makes sense. You are not on the branch that already has the fix. Your commit is latest master branch.
Either switch to the branch that got mentioned or wait for the fix to land in master.

[JB]

@MajorP93 Ok. Thanks!

[pierrebrunet]

@JB @majorp93 We made a smaller PR to be merged just after the release (because it is linked to almost all XAPI calls), the new branch is this one if you want to test: fix_undici_timeout
Same fix, just a little smaller

[christopher-petzel]

@pierrebrunet I built on the fix_undici_timeout branch and manually ran the metadata backup job 10 times and did NOT get an error. I will stay on this build through the weekend, let the scheduled metadata backups run, and report back on Monday.

[Tim-PT]

It looks like this is identified and solved, I'll test on a new version of XO next week at some point, but I just wanted to confirm that I've been seeing what appears to be the same problem.

This hasn't affected any of my full, delta or mirror backups, on any of the NFS targets - but it does affect two different Pool metadata & xo config backups. Both on different NFS targets on different schedules.

It's an infrequent failure, sometimes not failing for a couple of days, but sometimes failing up to 3 or 4 times per day - this is on an hourly schedule.

My most recent failure on the hourly job was yesterday morning, the one before that was almost 24 hours earlier.
This is currently on Xen Orchestra, commit 24913 but it was also noticeable on a version from the end of July.

From what I've read, I'm guessing the fix will be in the main branch soon enough and an update will solve it, but if there's any useful information I can provide, please let me know.

[poddingue]

Thanks for the feedback so far, folks!

[christopher-petzel]

@poddingue On the fix_undici_timeout branch this weekend, all scheduled metadata backups completed sucesfully.