<![CDATA[Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"]]>For the last week or so, the backup on one paticular VM gives this error:

VM1 (xcp-ng-1) 
•	Snapshot 
Start: 2024-05-12 04:00
End: 2024-05-12 04:00
•	NFS2 
o	transfer
Start: 2024-05-12 04:00
End: 2024-05-12 05:21
Duration: an hour
Error: Body Timeout Error
•	Start: 2024-05-12 04:00
•	End: 2024-05-12 05:21
•	Duration: an hour
•	Error: Body Timeout Error
NFS2 
o	transfer
Start: 2024-05-12 04:00
End: 2024-05-12 05:21
Duration: an hour
Error: Body Timeout Error
•	Start: 2024-05-12 04:00
•	End: 2024-05-12 05:21
•	Duration: an hour
•	Error: Body Timeout Error
Start: 2024-05-12 04:00
End: 2024-05-12 05:23
Duration: an hour
Error: all targets have failed, step: writer.run()
Type: full

The host is running xcp beta 3 (updated through 5-12-24), XO is version "Xen Orchestra, commit 9b9c7".

This VM and numerous other VM's back up the same way (goes to two different NFS targets) for the last five months.

I've tried running it in the middle of the day (when other backups are not running) but it didn't help.

"Number of retries if VM backup fails"=0
"Timeout " = <blank>
"Compression"="zstd"

Any ideas?

]]>https://xcp-ng.org/forum/topic/9002/backup-fails-with-body-timeout-error-all-targets-have-failed-step-writer-runRSS for NodeSat, 12 Sep 2026 20:57:33 GMTSun, 12 May 2024 11:25:26 GMT60<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Wed, 09 Sep 2026 14:46:30 GMT]]>@JB @christopher-petzel Thanks for your patience! We are glad it works well!

]]>https://xcp-ng.org/forum/post/108383https://xcp-ng.org/forum/post/108383Wed, 09 Sep 2026 14:46:30 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Wed, 09 Sep 2026 14:43:24 GMT]]>4ce59e56-d379-49fc-83dd-575a82838aec-image.jpeg

That's wonderful! Thank you, XenOrchestra team!

]]>https://xcp-ng.org/forum/post/108382https://xcp-ng.org/forum/post/108382Wed, 09 Sep 2026 14:43:24 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Wed, 09 Sep 2026 12:51:48 GMT]]>Thanks a lot for the feedback, @christopher-petzel ! 👍

]]>https://xcp-ng.org/forum/post/108378https://xcp-ng.org/forum/post/108378Wed, 09 Sep 2026 12:51:48 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Wed, 09 Sep 2026 12:25:23 GMT]]>The fix has been merged into the master branch.

88f787a0-138b-40c5-9dde-dacfff97c1cf-image.jpeg

I've updated from master and tested the metadata backup by manually triggering it 10 times. I had no errors. This issue seems to be resolved.

Thanks to the XO team for this fix!

]]>https://xcp-ng.org/forum/post/108373https://xcp-ng.org/forum/post/108373Wed, 09 Sep 2026 12:25:23 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Tue, 08 Sep 2026 13:31:24 GMT]]>Looking at Git, it doesn't seem like it, so I guess that answers my question.

]]>https://xcp-ng.org/forum/post/108355https://xcp-ng.org/forum/post/108355Tue, 08 Sep 2026 13:31:24 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Tue, 08 Sep 2026 13:10:53 GMT]]>@christopher-petzel Hello. Has the fix been merged yet? I'm on commit 03865 and I'm still getting this problem.

]]>https://xcp-ng.org/forum/post/108354https://xcp-ng.org/forum/post/108354Tue, 08 Sep 2026 13:10:53 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Wed, 26 Aug 2026 12:59:31 GMT]]>@christopher-petzel Ok! Thanks!

]]>https://xcp-ng.org/forum/post/108105https://xcp-ng.org/forum/post/108105Wed, 26 Aug 2026 12:59:31 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Wed, 26 Aug 2026 00:05:01 GMT]]>@JB @gregbinsd The fix has not been merged with the master branch yet, so you won't have the fix yet. Follow this pull request https://github.com/vatesfr/xen-orchestra/pull/10282

When you see an entry at the end of the activity log which says something like ' merged commit [commit ID] into master' then you'll know the fix has been added to the master branch.

It's important to know that they could close this pull request without committing it to the master branch if they want to approach the resolution in a different way (this has already happened once with this issue).

]]>https://xcp-ng.org/forum/post/108089https://xcp-ng.org/forum/post/108089Wed, 26 Aug 2026 00:05:01 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Tue, 25 Aug 2026 18:05:13 GMT]]>Here is the log for the GUI Backup function:

1dab5aec-6f9a-4338-bf7d-dab065e17e43-image.jpeg

]]>https://xcp-ng.org/forum/post/108086https://xcp-ng.org/forum/post/108086Tue, 25 Aug 2026 18:05:13 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Tue, 25 Aug 2026 18:01:27 GMT]]>Here is my story:
I responded to the notice regarding the latest update of the main branch, Master, commit 9cdaf, by updating the XO-CE vm, rebooting the XO-CE vm, bringing all 3 XCP-ng hosts on line, and then running the XO-CE GUI and selecting Pool Rolling Update.
The rolling update cycled through all 3 hosts in the pool.
I then attempted to run a pool metadata and xo_config backup.
It failed by timing out at 5 minutes.
I decided to shutdown the pool master, so I transferred all VMs to the other 2 hosts and shutdown the master for 15 seconds. I powered on the master, and transferred the VMs back to the master.
I shutdown the other 2 hosts, and then powered them back on. All hosts are indicating "green" status.
At this point I attempted to run the pool metadata and xo_config backup, and it ran successfully.
I waited 2 minutes and ran it again, with success again!
Lastly, I powered down the other 2 hosts, and with the VMs all running on the master, I ran the pool metadata and xo_config backup again, with apparent success.
If someone can tell me the location of the logs that recorded this operation, I will look at them closely to confirm that the pool metadata and xo_config backups actually took place.
Thank you.

]]>https://xcp-ng.org/forum/post/108085https://xcp-ng.org/forum/post/108085Tue, 25 Aug 2026 18:01:27 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Tue, 25 Aug 2026 17:54:54 GMT]]>0f9a7699-7d49-4dce-a588-e7e999730275-image.jpeg

commit 9cdaf

]]>https://xcp-ng.org/forum/post/108084https://xcp-ng.org/forum/post/108084Tue, 25 Aug 2026 17:54:54 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Mon, 24 Aug 2026 13:04:21 GMT]]>That's great news, thanks, @christopher-petzel ! 👍

]]>https://xcp-ng.org/forum/post/108058https://xcp-ng.org/forum/post/108058Mon, 24 Aug 2026 13:04:21 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Mon, 24 Aug 2026 11:48:27 GMT]]>@poddingue On the fix_undici_timeout branch this weekend, all scheduled metadata backups completed sucesfully.

]]>https://xcp-ng.org/forum/post/108055https://xcp-ng.org/forum/post/108055Mon, 24 Aug 2026 11:48:27 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Mon, 24 Aug 2026 08:13:56 GMT]]>Thanks for the feedback so far, folks! 👍

]]>https://xcp-ng.org/forum/post/108039https://xcp-ng.org/forum/post/108039Mon, 24 Aug 2026 08:13:56 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Sat, 22 Aug 2026 10:36:39 GMT]]>It looks like this is identified and solved, I'll test on a new version of XO next week at some point, but I just wanted to confirm that I've been seeing what appears to be the same problem.

This hasn't affected any of my full, delta or mirror backups, on any of the NFS targets - but it does affect two different Pool metadata & xo config backups. Both on different NFS targets on different schedules.

It's an infrequent failure, sometimes not failing for a couple of days, but sometimes failing up to 3 or 4 times per day - this is on an hourly schedule.
912ddd51-e6c1-4a62-8529-79d6b2fe8ad0-image.jpeg

My most recent failure on the hourly job was yesterday morning, the one before that was almost 24 hours earlier.
This is currently on Xen Orchestra, commit 24913 but it was also noticeable on a version from the end of July.

From what I've read, I'm guessing the fix will be in the main branch soon enough and an update will solve it, but if there's any useful information I can provide, please let me know.

]]>https://xcp-ng.org/forum/post/108021https://xcp-ng.org/forum/post/108021Sat, 22 Aug 2026 10:36:39 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Thu, 20 Aug 2026 12:18:10 GMT]]>@pierrebrunet I built on the fix_undici_timeout branch and manually ran the metadata backup job 10 times and did NOT get an error. I will stay on this build through the weekend, let the scheduled metadata backups run, and report back on Monday.

]]>https://xcp-ng.org/forum/post/107972https://xcp-ng.org/forum/post/107972Thu, 20 Aug 2026 12:18:10 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Thu, 20 Aug 2026 10:07:31 GMT]]>@JB @majorp93 We made a smaller PR to be merged just after the release (because it is linked to almost all XAPI calls), the new branch is this one if you want to test: fix_undici_timeout
Same fix, just a little smaller 🙂

]]>https://xcp-ng.org/forum/post/107969https://xcp-ng.org/forum/post/107969Thu, 20 Aug 2026 10:07:31 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Fri, 14 Aug 2026 21:19:57 GMT]]>@MajorP93 Ok. Thanks!

]]>https://xcp-ng.org/forum/post/107829https://xcp-ng.org/forum/post/107829Fri, 14 Aug 2026 21:19:57 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Fri, 14 Aug 2026 15:11:54 GMT]]>@JB Yes makes sense. You are not on the branch that already has the fix. Your commit is latest master branch.
Either switch to the branch that got mentioned or wait for the fix to land in master.

]]>https://xcp-ng.org/forum/post/107815https://xcp-ng.org/forum/post/107815Fri, 14 Aug 2026 15:11:54 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Fri, 14 Aug 2026 14:37:39 GMT]]>@christopher-petzel Unfortunately, for me, it still has an error.

7425c362-284f-41d1-8fc5-ce89cc4edf4f-image.jpeg

commit c5fae

]]>https://xcp-ng.org/forum/post/107814https://xcp-ng.org/forum/post/107814Fri, 14 Aug 2026 14:37:39 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Fri, 14 Aug 2026 13:09:21 GMT]]>Excellent 🙂 so it was an undici bug after all… Now I'm making sure @Team-XO-Backend won't miss it 👍

]]>https://xcp-ng.org/forum/post/107809https://xcp-ng.org/forum/post/107809Fri, 14 Aug 2026 13:09:21 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Fri, 14 Aug 2026 11:56:38 GMT]]>@olivierlambert No metadata backup errors in the 4 scheduled backups since Monday, while on the fix_undici_paused_parser_crash branch. Previously, I would not have more that 1 day without an error. Between manual and scheduled backup jobs, I've not been able to recreate the error on this test branch.

This is good to see again...
6f8b9214-c5d9-4aef-abfa-b982dcccbe0b-image.jpeg

]]>https://xcp-ng.org/forum/post/107807https://xcp-ng.org/forum/post/107807Fri, 14 Aug 2026 11:56:38 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Mon, 10 Aug 2026 16:37:35 GMT]]>@christopher-petzel
Great news!
I will test and report after it is available in a global update.
Thank you Christopher.

]]>https://xcp-ng.org/forum/post/107688https://xcp-ng.org/forum/post/107688Mon, 10 Aug 2026 16:37:35 GMT<![CDATA[Reply to Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()" on Mon, 10 Aug 2026 15:19:43 GMT]]>@christopher-petzel Thank you for testing this! I am also interested in this fix since I encounter this issue from time to time. Unfortunately my testing environment / lab is currently unavailable which is why I wasn't able to compile & deploy this branch myself.

]]>https://xcp-ng.org/forum/post/107685https://xcp-ng.org/forum/post/107685Mon, 10 Aug 2026 15:19:43 GMT