Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

bogikornel

@poddingue said:

Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out".

The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

poddingue

The 2026-05-28 build still being good narrows the window a lot more than "sometime since November".
What I keep coming back to is the shape of the failure: 14 out of 15, then 5 out of 6, so it is always exactly one that falls over and never the whole run. I don't know whether that points at a per-VM timeout or at something the last task in a run does differently, and someone on the XO team will read that better than me.
@pierrebrunet still needs /var/log/xensource.log from the pool master covering one failed run's window, so even a slice from a job where only one VM failed should be enough.

JB

Any solution?

christopher-petzel

@poddingue Maybe this log will help @pierrebrunet . I've been having the Body Timeout Error for a couple of weeks on Metadata/Config backups. I'm using XO from sources. The problem started after upgrading to commit 63f8d. I was previously at commit e6443. The errors will occur for one or more hosts, and which host(s) has the error seems to be random. The error will occur for hosts which have VMs and for hosts that have no VMs. Attached is this morning's xensource.log from 00:10 when the backup started. The backup ends at 00:15 but I've included log data through 00:20. xensource-truncated.log.txt

I have reverted to a snapshot of XO running at commit e6443 and executed multiple Metadata/Config backups without any problem.

JB

@christopher-petzel Thanks!

MajorP93

Hello,

the issue is back on my end.
Pool and metadata backup failed on all remotes due to "body timeout error".

Attached please find full backup log:
https://paste.vates.tech/?f781d8cb757e56b1#8MN4cyjHhkWKzhhvvggckf8mqjT9gxDY6t4vkSVQcWUY

I can confirm what @christopher-petzel said. Before commit 63f8d got pushed this issue did not occur.

I am currently running commit 1d82c (feat: release 6.6.2).

Best regards

olivierlambert

Hi! Can you try to replicate on the latest commit?

JB

@olivierlambert commit 59306

florent

@JB this looks like an issue while reading the export part. Did you configure a default backup network in the pool ?
is there any firewall/sdn controller that can filter out some reading ?

note that we have removed several point that where hiding some errors this month.

JB

@florent Between one error and another, the backup sometimes succeeds.

JB

@florent