Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

christopher-petzel

@olivierlambert Good news. This morning, I built on the fix_undici_paused_parser_crash branch and manually ran the metadata backup job 10 times and did NOT get an error. Previously, running the job manually could result in the body timeout error, so we have progress.

I will stay on this build during this week, let the scheduled metadata backups run, and report back by this Friday.

MajorP93

@christopher-petzel Thank you for testing this! I am also interested in this fix since I encounter this issue from time to time. Unfortunately my testing environment / lab is currently unavailable which is why I wasn't able to compile & deploy this branch myself.

GregBinSD

@christopher-petzel
Great news!
I will test and report after it is available in a global update.
Thank you Christopher.

christopher-petzel

@olivierlambert No metadata backup errors in the 4 scheduled backups since Monday, while on the fix_undici_paused_parser_crash branch. Previously, I would not have more that 1 day without an error. Between manual and scheduled backup jobs, I've not been able to recreate the error on this test branch.

This is good to see again...

olivierlambert

Excellent so it was an undici bug after all… Now I'm making sure @Team-XO-Backend won't miss it

JB

@christopher-petzel Unfortunately, for me, it still has an error.

commit c5fae

MajorP93

@JB Yes makes sense. You are not on the branch that already has the fix. Your commit is latest master branch.
Either switch to the branch that got mentioned or wait for the fix to land in master.

JB

@MajorP93 Ok. Thanks!

pierrebrunet

@JB @majorp93 We made a smaller PR to be merged just after the release (because it is linked to almost all XAPI calls), the new branch is this one if you want to test: fix_undici_timeout
Same fix, just a little smaller

christopher-petzel

@pierrebrunet I built on the fix_undici_timeout branch and manually ran the metadata backup job 10 times and did NOT get an error. I will stay on this build through the weekend, let the scheduled metadata backups run, and report back on Monday.