Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

[JB]

@florent hi! Hosts/XOA on the same 192.168.0.x network.

[JB]

[abudef]

Hi,
for several weeks, since 23 June 2026 (see below), I have been experiencing the same issue. XO (from source; daily updated) is running as a VM on the host (updated on the day or day + 1 the patches were released) being backed up.

result	
name	"BodyTimeoutError"
code	"UND_ERR_BODY_TIMEOUT"
message	"Body Timeout Error"
stack	"BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/dispatcher/client-h1.js:688:28)\n    at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:605:17)\n    at processTimers (node:internal/timers:541:7)"

Configuration	2026-07-31 20:22	2026-07-31 20:22	a few seconds		
Configuration	2026-07-31 20:16	2026-07-31 20:21	5 minutes		
Configuration	2026-07-31 20:09	2026-07-31 20:14	5 minutes		
Configuration	2026-07-31 19:50	2026-07-31 19:55	5 minutes		
Configuration	2026-07-30 20:53	2026-07-30 20:53	a few seconds		
Configuration	2026-07-30 19:50	2026-07-30 19:50	a few seconds		
Configuration	2026-07-29 19:50	2026-07-29 19:50	a few seconds		
Configuration	2026-07-28 20:05	2026-07-28 20:05	a few seconds		
Configuration	2026-07-28 19:50	2026-07-28 19:55	5 minutes		
Configuration	2026-07-27 19:50	2026-07-27 19:50	a few seconds		
Configuration	2026-07-26 20:49	2026-07-26 20:49	a few seconds		
Configuration	2026-07-26 19:50	2026-07-26 19:55	5 minutes		
Configuration	2026-07-25 19:50	2026-07-25 19:50	a few seconds		
Configuration	2026-07-24 19:50	2026-07-24 19:50	a few seconds		
Configuration	2026-07-23 20:48	2026-07-23 20:48	a few seconds		
Configuration	2026-07-23 19:50	2026-07-23 19:55	5 minutes		
Configuration	2026-07-22 19:50	2026-07-22 19:50	a few seconds		
Configuration	2026-07-21 19:50	2026-07-21 19:50	a few seconds		
Configuration	2026-07-20 19:50	2026-07-20 19:50	a few seconds		
Configuration	2026-07-19 19:50	2026-07-19 19:50	a few seconds		
Configuration	2026-07-19 07:47	2026-07-19 07:47	a few seconds		
Configuration	2026-07-18 19:50	2026-07-18 19:50	a few seconds		
Configuration	2026-07-17 19:50	2026-07-17 19:50	a few seconds		
Configuration	2026-07-16 19:50	2026-07-16 19:50	a few seconds		
Configuration	2026-07-15 19:50	2026-07-15 19:50	a few seconds		
Configuration	2026-07-14 19:50	2026-07-14 19:55	6 minutes		
Configuration	2026-07-13 23:03	2026-07-13 23:03	a few seconds		
Configuration	2026-07-13 19:50	2026-07-13 19:55	5 minutes		
Configuration	2026-07-12 19:50	2026-07-12 19:50	a few seconds		
Configuration	2026-07-11 19:50	2026-07-11 19:50	a few seconds		
Configuration	2026-07-10 19:50	2026-07-10 19:50	a few seconds		
Configuration	2026-07-09 19:50	2026-07-09 19:55	5 minutes		
Configuration	2026-07-08 19:50	2026-07-08 19:50	a few seconds		
Configuration	2026-07-07 19:50	2026-07-07 19:55	5 minutes		
Configuration	2026-07-06 19:50	2026-07-06 19:50	a few seconds		
Configuration	2026-07-06 13:34	2026-07-06 13:34	a few seconds		
Configuration	2026-07-05 19:50	2026-07-05 19:55	6 minutes		
Configuration	2026-07-04 19:50	2026-07-04 19:50	a few seconds		
Configuration	2026-07-03 19:50	2026-07-03 19:50	a few seconds		
Configuration	2026-07-02 19:50	2026-07-02 19:50	a few seconds		
Configuration	2026-07-01 19:50	2026-07-01 19:50	a few seconds		
Configuration	2026-07-01 06:28	2026-07-01 06:28	a few seconds		
Configuration	2026-07-01 00:52	2026-07-01 00:57	5 minutes		
Configuration	2026-06-30 19:50	2026-06-30 19:55	5 minutes		
Configuration	2026-06-29 20:23	2026-06-29 20:23	a few seconds		
Configuration	2026-06-29 19:50	2026-06-29 19:55	5 minutes		
Configuration	2026-06-28 19:50	2026-06-28 19:50	a few seconds		
Configuration	2026-06-27 19:50	2026-06-27 19:50	a few seconds		
Configuration	2026-06-26 19:50	2026-06-26 19:50	a few seconds		
Configuration	2026-06-25 19:50	2026-06-25 19:50	a few seconds		
Configuration	2026-06-24 19:50	2026-06-24 19:50	a few seconds		
Configuration	2026-06-24 07:03	2026-06-24 07:03	a few seconds		
Configuration	2026-06-23 19:50	2026-06-23 19:55	5 minutes			

[acebmxer]

Looking threw the backup logs at work and it seems i might be having a similar issue with only one of my pools and it does not happen all the time....

Dates of errors... Each time if I restarted the tool stack for host 1 on the effected pool the follow run of the job would be successful until one day it fails again. Let me know if you would like for me to open a support tunel.
7/30/26
7/25/26
7/24/26

 },
  "id": "1784976796895",
  "jobId": "42b76839-453e-4486-83e7-d736d8625ce5",
  "jobName": "Tilton Pool Backup",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "828b5e60-4302-455c-94f1-479a8001f955",
  "start": 1784976796895,
  "status": "failure",
  "tasks": [
    {
      "id": "0ms093lzi-tuc7iks1oqp",
      "start": 1784976800526,
      "status": "success",
      "tasks": [
        {
          "id": "0ms093lzj-fo91s5yb7xv",
          "start": 1784976800527,
          "status": "success",
          "end": 1784976800603,
          "message": "Starting XO metadata backup for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote"
          }
        }
      ],
      "end": 1784976800603,
      "message": "Starting XO metadata backup. (42b76839-453e-4486-83e7-d736d8625ce5)",
      "data": {
        "type": "xo"
      }
    },
    {
      "id": "0ms093lzn-gw1xvnk515",
      "start": 1784976800531,
      "status": "failure",
      "tasks": [
        {
          "id": "0ms093m09-e5tj4brl4hh",
          "start": 1784976800553,
          "status": "failure",
          "end": 1784977100916,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/dispatcher/client-h1.js:702:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at process.processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (64466023-8ef0-f5de-54ed-57e30eca644b) for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote",
            "progress": 0
          }
        }
      ],
      "end": 1784977100917,

[christopher-petzel]

@florent suggesting this looks like a network issue, I took a look at my XO's server's syslog and found the error when this occurs.

2026-07-31T00:10:00.428083-04:00 mis-26-svr xo-server[1298]: 2026-07-31T04:10:00.427Z xo:xo-server ERROR uncaught exception {
2026-07-31T00:10:00.428186-04:00 mis-26-svr xo-server[1298]:   error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
2026-07-31T00:10:00.428262-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428309-04:00 mis-26-svr xo-server[1298]:     assert(!this.paused)
2026-07-31T00:10:00.428341-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428391-04:00 mis-26-svr xo-server[1298]:       at Parser.finish (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:302:5)
2026-07-31T00:10:00.428414-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:741:32)
2026-07-31T00:10:00.428434-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.emit (node:events:521:24)
2026-07-31T00:10:00.428478-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607200851/@xen-orchestra/log/configure.js:52:17)
2026-07-31T00:10:00.428508-04:00 mis-26-svr xo-server[1298]:       at endReadableNT (node:internal/streams/readable:1729:12)
2026-07-31T00:10:00.428525-04:00 mis-26-svr xo-server[1298]:       at processTicksAndRejections (node:internal/process/task_queues:90:21) {
2026-07-31T00:10:00.428549-04:00 mis-26-svr xo-server[1298]:     generatedMessage: true,
2026-07-31T00:10:00.428567-04:00 mis-26-svr xo-server[1298]:     code: 'ERR_ASSERTION',
2026-07-31T00:10:00.428582-04:00 mis-26-svr xo-server[1298]:     actual: false,
2026-07-31T00:10:00.428599-04:00 mis-26-svr xo-server[1298]:     expected: true,
2026-07-31T00:10:00.428614-04:00 mis-26-svr xo-server[1298]:     operator: '==',
2026-07-31T00:10:00.428628-04:00 mis-26-svr xo-server[1298]:     diff: 'simple'
2026-07-31T00:10:00.428642-04:00 mis-26-svr xo-server[1298]:   }
2026-07-31T00:10:00.428659-04:00 mis-26-svr xo-server[1298]: }

So it looks like xo-server is dying, could it be the module undici that the problem here? Keep in mind that 13 other instances of xo-server performed the backup without a problem at the exact same time. Some days this never happens, some days it could be 4 instances of xo-server that die the same way in executing the backup.

I think it's also important to note that the instance of xo-server dying happens immediately upon the backup being started at 10 minutes after midnight. I would think that if the problem were the network itself, that there would be some timeout period before causing the xo-server instance to crash.

To answer a question asked of @jb , this is a local XCP-ng management network and there are no other backups occurring at 00:10.

[poddingue]

Trying to pull the last few posts together, because there is a pattern I keep staring at.
Every failed run in @JB's screenshots lasts exactly 5 minutes while the successful ones finish in seconds, and @abudef's history going back to 23 June shows the same 5-minute wall.
That reads to me like a fixed timeout firing rather than a merely slow transfer, though I could be wrong about which one.
@christopher-petzel's syslog looks like a different animal, since assert(!this.paused) in undici's Parser.finish is xo-server crashing outright at job start while 13 other instances backed up fine the same minute. The only knob I can find written down is httpInactivityTimeout at https://docs.xen-orchestra.com/xo5/backup_troubleshooting#error-http-connection-has-timed-out, but that one sits on the XAPI GET path, and I do not know whether it touches pool metadata jobs at all. @florent, does the 5-minute signature change anything on your side?

[bogikornel]

@poddingue httpInactivityTimeout changed the only thing for me, I received the email half an hour later that the metadata could not be saved.

[marcoi]

looks like im getting the xo config backup body timeout error as well.
Xen Orchestra, commit a94eb
Master, commit 9c6d1

  "result": {
    "message": "backup task failed with undefined error",
    "name": "Error",
    "stack": "Error: backup task failed with undefined error\n    at forwardResult (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/_handleBackupLog.mjs:37:25)\n    at handleBackupLog (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/_handleBackupLog.mjs:68:12)\n    at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/metadata-backups.mjs:133:13)\n    at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202608030700/@xen-orchestra/mixins/Tasks.mjs:205:23)\n    at Task.onProgress [as _onProgress] (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/combineEvents.js:61:5)\n    at Task.#emit (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:157:10)\n    at Task.#end (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:168:15)\n    at Task.run (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:190:16)\n    at Jobs.runJob (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/jobs/index.mjs:297:7)\n    at Jobs.runJobSequence (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/jobs/index.mjs:339:7)\n    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:204:22)\n    at Task.run (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:188:20)\n    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/api.mjs:475:18)"
  }

[JB]

@christopher-petzel Olá Chris!
Yes, this is a local XCP-ng management network and there are no other backups simultaneous.

[JB]

@poddingue Olá! The failure has a fixed duration of 5 minutes.