Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

[acebmxer]

Looking threw the backup logs at work and it seems i might be having a similar issue with only one of my pools and it does not happen all the time....

Dates of errors... Each time if I restarted the tool stack for host 1 on the effected pool the follow run of the job would be successful until one day it fails again. Let me know if you would like for me to open a support tunel.
7/30/26
7/25/26
7/24/26

 },
  "id": "1784976796895",
  "jobId": "42b76839-453e-4486-83e7-d736d8625ce5",
  "jobName": "Tilton Pool Backup",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "828b5e60-4302-455c-94f1-479a8001f955",
  "start": 1784976796895,
  "status": "failure",
  "tasks": [
    {
      "id": "0ms093lzi-tuc7iks1oqp",
      "start": 1784976800526,
      "status": "success",
      "tasks": [
        {
          "id": "0ms093lzj-fo91s5yb7xv",
          "start": 1784976800527,
          "status": "success",
          "end": 1784976800603,
          "message": "Starting XO metadata backup for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote"
          }
        }
      ],
      "end": 1784976800603,
      "message": "Starting XO metadata backup. (42b76839-453e-4486-83e7-d736d8625ce5)",
      "data": {
        "type": "xo"
      }
    },
    {
      "id": "0ms093lzn-gw1xvnk515",
      "start": 1784976800531,
      "status": "failure",
      "tasks": [
        {
          "id": "0ms093m09-e5tj4brl4hh",
          "start": 1784976800553,
          "status": "failure",
          "end": 1784977100916,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/dispatcher/client-h1.js:702:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at process.processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (64466023-8ef0-f5de-54ed-57e30eca644b) for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote",
            "progress": 0
          }
        }
      ],
      "end": 1784977100917,

[christopher-petzel]

@florent suggesting this looks like a network issue, I took a look at my XO's server's syslog and found the error when this occurs.

2026-07-31T00:10:00.428083-04:00 mis-26-svr xo-server[1298]: 2026-07-31T04:10:00.427Z xo:xo-server ERROR uncaught exception {
2026-07-31T00:10:00.428186-04:00 mis-26-svr xo-server[1298]:   error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
2026-07-31T00:10:00.428262-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428309-04:00 mis-26-svr xo-server[1298]:     assert(!this.paused)
2026-07-31T00:10:00.428341-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428391-04:00 mis-26-svr xo-server[1298]:       at Parser.finish (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:302:5)
2026-07-31T00:10:00.428414-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:741:32)
2026-07-31T00:10:00.428434-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.emit (node:events:521:24)
2026-07-31T00:10:00.428478-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607200851/@xen-orchestra/log/configure.js:52:17)
2026-07-31T00:10:00.428508-04:00 mis-26-svr xo-server[1298]:       at endReadableNT (node:internal/streams/readable:1729:12)
2026-07-31T00:10:00.428525-04:00 mis-26-svr xo-server[1298]:       at processTicksAndRejections (node:internal/process/task_queues:90:21) {
2026-07-31T00:10:00.428549-04:00 mis-26-svr xo-server[1298]:     generatedMessage: true,
2026-07-31T00:10:00.428567-04:00 mis-26-svr xo-server[1298]:     code: 'ERR_ASSERTION',
2026-07-31T00:10:00.428582-04:00 mis-26-svr xo-server[1298]:     actual: false,
2026-07-31T00:10:00.428599-04:00 mis-26-svr xo-server[1298]:     expected: true,
2026-07-31T00:10:00.428614-04:00 mis-26-svr xo-server[1298]:     operator: '==',
2026-07-31T00:10:00.428628-04:00 mis-26-svr xo-server[1298]:     diff: 'simple'
2026-07-31T00:10:00.428642-04:00 mis-26-svr xo-server[1298]:   }
2026-07-31T00:10:00.428659-04:00 mis-26-svr xo-server[1298]: }

So it looks like xo-server is dying, could it be the module undici that the problem here? Keep in mind that 13 other instances of xo-server performed the backup without a problem at the exact same time. Some days this never happens, some days it could be 4 instances of xo-server that die the same way in executing the backup.

I think it's also important to note that the instance of xo-server dying happens immediately upon the backup being started at 10 minutes after midnight. I would think that if the problem were the network itself, that there would be some timeout period before causing the xo-server instance to crash.

To answer a question asked of @jb , this is a local XCP-ng management network and there are no other backups occurring at 00:10.