Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

[bogikornel]

@pierrebrunet I'll try to collect the logs.
However, I can't even backup VMs. What's interesting is that the error always occurs with only 1 VM or 1 Metadata backup. This morning's backup report:
Job ID: 3c17f0c1-85f5-4b98-b43a-0d117c812a9a
Run ID: 1783461900005
Mode: full
Start time: Wednesday, July 8th 2026, 12:05:00 am
End time: Wednesday, July 8th 2026, 1:22:44 am
Duration: an hour
Successes: 14 / 15
Transfer size: 276.37 GiB

Or with another host and another XO:
Job ID: 53c582e5-ba01-4a82-8a45-948a991add99
Job name: metadata
Run ID: 1783483200004
Start time: Wednesday, July 8th 2026, 6:00:00 am
End time: Wednesday, July 8th 2026, 6:05:00 am
Duration: 5 minutes
Successes: 5 / 6
Error: backup task failed with undefined error

[poddingue]

Thanks @bogikornel, that version comparison really helps.
Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out".

On the log Pierre asked for: the useful one is /var/log/xensource.log from the pool master, covering one failed run's window.
You already have the Job and Run IDs and the 00:05 to 01:22 timestamps from this morning's run, so that exact span is perfect.
If you can pull it from the master and attach it here (or just the slice around the timeout), that's what lets them line the failure up against what xapi was doing.
It's looking like it may be coming from the XCP-ng side rather than XO, so a mention to @Team-Storage might help too. I could be wrong about where it actually lands, but the log is the thing that'll tell us.

[bogikornel]

@poddingue said:

Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out".

The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

[poddingue]

The 2026-05-28 build still being good narrows the window a lot more than "sometime since November".
What I keep coming back to is the shape of the failure: 14 out of 15, then 5 out of 6, so it is always exactly one that falls over and never the whole run. I don't know whether that points at a per-VM timeout or at something the last task in a run does differently, and someone on the XO team will read that better than me.
@pierrebrunet still needs /var/log/xensource.log from the pool master covering one failed run's window, so even a slice from a job where only one VM failed should be enough.

[JB]

Any solution?

[christopher-petzel]

@poddingue Maybe this log will help @pierrebrunet . I've been having the Body Timeout Error for a couple of weeks on Metadata/Config backups. I'm using XO from sources. The problem started after upgrading to commit 63f8d. I was previously at commit e6443. The errors will occur for one or more hosts, and which host(s) has the error seems to be random. The error will occur for hosts which have VMs and for hosts that have no VMs. Attached is this morning's xensource.log from 00:10 when the backup started. The backup ends at 00:15 but I've included log data through 00:20. xensource-truncated.log.txt

I have reverted to a snapshot of XO running at commit e6443 and executed multiple Metadata/Config backups without any problem.

[JB]

@christopher-petzel Thanks!

[MajorP93]

Hello,

the issue is back on my end.
Pool and metadata backup failed on all remotes due to "body timeout error".

Attached please find full backup log:
https://paste.vates.tech/?f781d8cb757e56b1#8MN4cyjHhkWKzhhvvggckf8mqjT9gxDY6t4vkSVQcWUY

I can confirm what @christopher-petzel said. Before commit 63f8d got pushed this issue did not occur.

I am currently running commit 1d82c (feat: release 6.6.2).

Best regards

[olivierlambert]

Hi! Can you try to replicate on the latest commit?

[JB]

@olivierlambert commit 59306

[florent]

@JB this looks like an issue while reading the export part. Did you configure a default backup network in the pool ?
is there any firewall/sdn controller that can filter out some reading ?

note that we have removed several point that where hiding some errors this month.

[JB]

@florent Between one error and another, the backup sometimes succeeds.

[JB]

@florent

[florent]

@JB this really looks like a network issue

are the host/xoa remotes ?

[JB]

@florent hi! Hosts/XOA on the same 192.168.0.x network.

[JB]

[abudef]

Hi,
for several weeks, since 23 June 2026 (see below), I have been experiencing the same issue. XO (from source; daily updated) is running as a VM on the host (updated on the day or day + 1 the patches were released) being backed up.

result	
name	"BodyTimeoutError"
code	"UND_ERR_BODY_TIMEOUT"
message	"Body Timeout Error"
stack	"BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/dispatcher/client-h1.js:688:28)\n    at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:605:17)\n    at processTimers (node:internal/timers:541:7)"

Configuration	2026-07-31 20:22	2026-07-31 20:22	a few seconds		
Configuration	2026-07-31 20:16	2026-07-31 20:21	5 minutes		
Configuration	2026-07-31 20:09	2026-07-31 20:14	5 minutes		
Configuration	2026-07-31 19:50	2026-07-31 19:55	5 minutes		
Configuration	2026-07-30 20:53	2026-07-30 20:53	a few seconds		
Configuration	2026-07-30 19:50	2026-07-30 19:50	a few seconds		
Configuration	2026-07-29 19:50	2026-07-29 19:50	a few seconds		
Configuration	2026-07-28 20:05	2026-07-28 20:05	a few seconds		
Configuration	2026-07-28 19:50	2026-07-28 19:55	5 minutes		
Configuration	2026-07-27 19:50	2026-07-27 19:50	a few seconds		
Configuration	2026-07-26 20:49	2026-07-26 20:49	a few seconds		
Configuration	2026-07-26 19:50	2026-07-26 19:55	5 minutes		
Configuration	2026-07-25 19:50	2026-07-25 19:50	a few seconds		
Configuration	2026-07-24 19:50	2026-07-24 19:50	a few seconds		
Configuration	2026-07-23 20:48	2026-07-23 20:48	a few seconds		
Configuration	2026-07-23 19:50	2026-07-23 19:55	5 minutes		
Configuration	2026-07-22 19:50	2026-07-22 19:50	a few seconds		
Configuration	2026-07-21 19:50	2026-07-21 19:50	a few seconds		
Configuration	2026-07-20 19:50	2026-07-20 19:50	a few seconds		
Configuration	2026-07-19 19:50	2026-07-19 19:50	a few seconds		
Configuration	2026-07-19 07:47	2026-07-19 07:47	a few seconds		
Configuration	2026-07-18 19:50	2026-07-18 19:50	a few seconds		
Configuration	2026-07-17 19:50	2026-07-17 19:50	a few seconds		
Configuration	2026-07-16 19:50	2026-07-16 19:50	a few seconds		
Configuration	2026-07-15 19:50	2026-07-15 19:50	a few seconds		
Configuration	2026-07-14 19:50	2026-07-14 19:55	6 minutes		
Configuration	2026-07-13 23:03	2026-07-13 23:03	a few seconds		
Configuration	2026-07-13 19:50	2026-07-13 19:55	5 minutes		
Configuration	2026-07-12 19:50	2026-07-12 19:50	a few seconds		
Configuration	2026-07-11 19:50	2026-07-11 19:50	a few seconds		
Configuration	2026-07-10 19:50	2026-07-10 19:50	a few seconds		
Configuration	2026-07-09 19:50	2026-07-09 19:55	5 minutes		
Configuration	2026-07-08 19:50	2026-07-08 19:50	a few seconds		
Configuration	2026-07-07 19:50	2026-07-07 19:55	5 minutes		
Configuration	2026-07-06 19:50	2026-07-06 19:50	a few seconds		
Configuration	2026-07-06 13:34	2026-07-06 13:34	a few seconds		
Configuration	2026-07-05 19:50	2026-07-05 19:55	6 minutes		
Configuration	2026-07-04 19:50	2026-07-04 19:50	a few seconds		
Configuration	2026-07-03 19:50	2026-07-03 19:50	a few seconds		
Configuration	2026-07-02 19:50	2026-07-02 19:50	a few seconds		
Configuration	2026-07-01 19:50	2026-07-01 19:50	a few seconds		
Configuration	2026-07-01 06:28	2026-07-01 06:28	a few seconds		
Configuration	2026-07-01 00:52	2026-07-01 00:57	5 minutes		
Configuration	2026-06-30 19:50	2026-06-30 19:55	5 minutes		
Configuration	2026-06-29 20:23	2026-06-29 20:23	a few seconds		
Configuration	2026-06-29 19:50	2026-06-29 19:55	5 minutes		
Configuration	2026-06-28 19:50	2026-06-28 19:50	a few seconds		
Configuration	2026-06-27 19:50	2026-06-27 19:50	a few seconds		
Configuration	2026-06-26 19:50	2026-06-26 19:50	a few seconds		
Configuration	2026-06-25 19:50	2026-06-25 19:50	a few seconds		
Configuration	2026-06-24 19:50	2026-06-24 19:50	a few seconds		
Configuration	2026-06-24 07:03	2026-06-24 07:03	a few seconds		
Configuration	2026-06-23 19:50	2026-06-23 19:55	5 minutes			

[acebmxer]

Looking threw the backup logs at work and it seems i might be having a similar issue with only one of my pools and it does not happen all the time....

Dates of errors... Each time if I restarted the tool stack for host 1 on the effected pool the follow run of the job would be successful until one day it fails again. Let me know if you would like for me to open a support tunel.
7/30/26
7/25/26
7/24/26

 },
  "id": "1784976796895",
  "jobId": "42b76839-453e-4486-83e7-d736d8625ce5",
  "jobName": "Tilton Pool Backup",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "828b5e60-4302-455c-94f1-479a8001f955",
  "start": 1784976796895,
  "status": "failure",
  "tasks": [
    {
      "id": "0ms093lzi-tuc7iks1oqp",
      "start": 1784976800526,
      "status": "success",
      "tasks": [
        {
          "id": "0ms093lzj-fo91s5yb7xv",
          "start": 1784976800527,
          "status": "success",
          "end": 1784976800603,
          "message": "Starting XO metadata backup for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote"
          }
        }
      ],
      "end": 1784976800603,
      "message": "Starting XO metadata backup. (42b76839-453e-4486-83e7-d736d8625ce5)",
      "data": {
        "type": "xo"
      }
    },
    {
      "id": "0ms093lzn-gw1xvnk515",
      "start": 1784976800531,
      "status": "failure",
      "tasks": [
        {
          "id": "0ms093m09-e5tj4brl4hh",
          "start": 1784976800553,
          "status": "failure",
          "end": 1784977100916,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/dispatcher/client-h1.js:702:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at process.processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (64466023-8ef0-f5de-54ed-57e30eca644b) for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote",
            "progress": 0
          }
        }
      ],
      "end": 1784977100917,

[christopher-petzel]

@florent suggesting this looks like a network issue, I took a look at my XO's server's syslog and found the error when this occurs.

2026-07-31T00:10:00.428083-04:00 mis-26-svr xo-server[1298]: 2026-07-31T04:10:00.427Z xo:xo-server ERROR uncaught exception {
2026-07-31T00:10:00.428186-04:00 mis-26-svr xo-server[1298]:   error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
2026-07-31T00:10:00.428262-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428309-04:00 mis-26-svr xo-server[1298]:     assert(!this.paused)
2026-07-31T00:10:00.428341-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428391-04:00 mis-26-svr xo-server[1298]:       at Parser.finish (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:302:5)
2026-07-31T00:10:00.428414-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:741:32)
2026-07-31T00:10:00.428434-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.emit (node:events:521:24)
2026-07-31T00:10:00.428478-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607200851/@xen-orchestra/log/configure.js:52:17)
2026-07-31T00:10:00.428508-04:00 mis-26-svr xo-server[1298]:       at endReadableNT (node:internal/streams/readable:1729:12)
2026-07-31T00:10:00.428525-04:00 mis-26-svr xo-server[1298]:       at processTicksAndRejections (node:internal/process/task_queues:90:21) {
2026-07-31T00:10:00.428549-04:00 mis-26-svr xo-server[1298]:     generatedMessage: true,
2026-07-31T00:10:00.428567-04:00 mis-26-svr xo-server[1298]:     code: 'ERR_ASSERTION',
2026-07-31T00:10:00.428582-04:00 mis-26-svr xo-server[1298]:     actual: false,
2026-07-31T00:10:00.428599-04:00 mis-26-svr xo-server[1298]:     expected: true,
2026-07-31T00:10:00.428614-04:00 mis-26-svr xo-server[1298]:     operator: '==',
2026-07-31T00:10:00.428628-04:00 mis-26-svr xo-server[1298]:     diff: 'simple'
2026-07-31T00:10:00.428642-04:00 mis-26-svr xo-server[1298]:   }
2026-07-31T00:10:00.428659-04:00 mis-26-svr xo-server[1298]: }

So it looks like xo-server is dying, could it be the module undici that the problem here? Keep in mind that 13 other instances of xo-server performed the backup without a problem at the exact same time. Some days this never happens, some days it could be 4 instances of xo-server that die the same way in executing the backup.

I think it's also important to note that the instance of xo-server dying happens immediately upon the backup being started at 10 minutes after midnight. I would think that if the problem were the network itself, that there would be some timeout period before causing the xo-server instance to crash.

To answer a question asked of @jb , this is a local XCP-ng management network and there are no other backups occurring at 00:10.

[poddingue]

Trying to pull the last few posts together, because there is a pattern I keep staring at.
Every failed run in @JB's screenshots lasts exactly 5 minutes while the successful ones finish in seconds, and @abudef's history going back to 23 June shows the same 5-minute wall.
That reads to me like a fixed timeout firing rather than a merely slow transfer, though I could be wrong about which one.
@christopher-petzel's syslog looks like a different animal, since assert(!this.paused) in undici's Parser.finish is xo-server crashing outright at job start while 13 other instances backed up fine the same minute. The only knob I can find written down is httpInactivityTimeout at https://docs.xen-orchestra.com/xo5/backup_troubleshooting#error-http-connection-has-timed-out, but that one sits on the XAPI GET path, and I do not know whether it touches pool metadata jobs at all. @florent, does the 5-minute signature change anything on your side?