Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

acebmxer

Looking threw the backup logs at work and it seems i might be having a similar issue with only one of my pools and it does not happen all the time....

Dates of errors... Each time if I restarted the tool stack for host 1 on the effected pool the follow run of the job would be successful until one day it fails again. Let me know if you would like for me to open a support tunel.
7/30/26
7/25/26
7/24/26

 },
  "id": "1784976796895",
  "jobId": "42b76839-453e-4486-83e7-d736d8625ce5",
  "jobName": "Tilton Pool Backup",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "828b5e60-4302-455c-94f1-479a8001f955",
  "start": 1784976796895,
  "status": "failure",
  "tasks": [
    {
      "id": "0ms093lzi-tuc7iks1oqp",
      "start": 1784976800526,
      "status": "success",
      "tasks": [
        {
          "id": "0ms093lzj-fo91s5yb7xv",
          "start": 1784976800527,
          "status": "success",
          "end": 1784976800603,
          "message": "Starting XO metadata backup for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote"
          }
        }
      ],
      "end": 1784976800603,
      "message": "Starting XO metadata backup. (42b76839-453e-4486-83e7-d736d8625ce5)",
      "data": {
        "type": "xo"
      }
    },
    {
      "id": "0ms093lzn-gw1xvnk515",
      "start": 1784976800531,
      "status": "failure",
      "tasks": [
        {
          "id": "0ms093m09-e5tj4brl4hh",
          "start": 1784976800553,
          "status": "failure",
          "end": 1784977100916,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/dispatcher/client-h1.js:702:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at process.processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (64466023-8ef0-f5de-54ed-57e30eca644b) for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)",
          "data": {
            "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
            "type": "remote",
            "progress": 0
          }
        }
      ],
      "end": 1784977100917,

christopher-petzel

@florent suggesting this looks like a network issue, I took a look at my XO's server's syslog and found the error when this occurs.

2026-07-31T00:10:00.428083-04:00 mis-26-svr xo-server[1298]: 2026-07-31T04:10:00.427Z xo:xo-server ERROR uncaught exception {
2026-07-31T00:10:00.428186-04:00 mis-26-svr xo-server[1298]:   error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
2026-07-31T00:10:00.428262-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428309-04:00 mis-26-svr xo-server[1298]:     assert(!this.paused)
2026-07-31T00:10:00.428341-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428391-04:00 mis-26-svr xo-server[1298]:       at Parser.finish (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:302:5)
2026-07-31T00:10:00.428414-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:741:32)
2026-07-31T00:10:00.428434-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.emit (node:events:521:24)
2026-07-31T00:10:00.428478-04:00 mis-26-svr xo-server[1298]:       at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607200851/@xen-orchestra/log/configure.js:52:17)
2026-07-31T00:10:00.428508-04:00 mis-26-svr xo-server[1298]:       at endReadableNT (node:internal/streams/readable:1729:12)
2026-07-31T00:10:00.428525-04:00 mis-26-svr xo-server[1298]:       at processTicksAndRejections (node:internal/process/task_queues:90:21) {
2026-07-31T00:10:00.428549-04:00 mis-26-svr xo-server[1298]:     generatedMessage: true,
2026-07-31T00:10:00.428567-04:00 mis-26-svr xo-server[1298]:     code: 'ERR_ASSERTION',
2026-07-31T00:10:00.428582-04:00 mis-26-svr xo-server[1298]:     actual: false,
2026-07-31T00:10:00.428599-04:00 mis-26-svr xo-server[1298]:     expected: true,
2026-07-31T00:10:00.428614-04:00 mis-26-svr xo-server[1298]:     operator: '==',
2026-07-31T00:10:00.428628-04:00 mis-26-svr xo-server[1298]:     diff: 'simple'
2026-07-31T00:10:00.428642-04:00 mis-26-svr xo-server[1298]:   }
2026-07-31T00:10:00.428659-04:00 mis-26-svr xo-server[1298]: }

So it looks like xo-server is dying, could it be the module undici that the problem here? Keep in mind that 13 other instances of xo-server performed the backup without a problem at the exact same time. Some days this never happens, some days it could be 4 instances of xo-server that die the same way in executing the backup.

I think it's also important to note that the instance of xo-server dying happens immediately upon the backup being started at 10 minutes after midnight. I would think that if the problem were the network itself, that there would be some timeout period before causing the xo-server instance to crash.

To answer a question asked of @jb , this is a local XCP-ng management network and there are no other backups occurring at 00:10.

poddingue

Trying to pull the last few posts together, because there is a pattern I keep staring at.
Every failed run in @JB's screenshots lasts exactly 5 minutes while the successful ones finish in seconds, and @abudef's history going back to 23 June shows the same 5-minute wall.
That reads to me like a fixed timeout firing rather than a merely slow transfer, though I could be wrong about which one.
@christopher-petzel's syslog looks like a different animal, since assert(!this.paused) in undici's Parser.finish is xo-server crashing outright at job start while 13 other instances backed up fine the same minute. The only knob I can find written down is httpInactivityTimeout at https://docs.xen-orchestra.com/xo5/backup_troubleshooting#error-http-connection-has-timed-out, but that one sits on the XAPI GET path, and I do not know whether it touches pool metadata jobs at all. @florent, does the 5-minute signature change anything on your side?

bogikornel

@poddingue httpInactivityTimeout changed the only thing for me, I received the email half an hour later that the metadata could not be saved.

marcoi

looks like im getting the xo config backup body timeout error as well.
Xen Orchestra, commit a94eb
Master, commit 9c6d1

  "result": {
    "message": "backup task failed with undefined error",
    "name": "Error",
    "stack": "Error: backup task failed with undefined error\n    at forwardResult (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/_handleBackupLog.mjs:37:25)\n    at handleBackupLog (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/_handleBackupLog.mjs:68:12)\n    at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/metadata-backups.mjs:133:13)\n    at onTaskUpdate (file:///opt/xo/xo-builds/xen-orchestra-202608030700/@xen-orchestra/mixins/Tasks.mjs:205:23)\n    at Task.onProgress [as _onProgress] (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/combineEvents.js:61:5)\n    at Task.#emit (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:157:10)\n    at Task.#end (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:168:15)\n    at Task.run (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:190:16)\n    at Jobs.runJob (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/jobs/index.mjs:297:7)\n    at Jobs.runJobSequence (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/jobs/index.mjs:339:7)\n    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:204:22)\n    at Task.run (/opt/xo/xo-builds/xen-orchestra-202608030700/@vates/task/index.js:188:20)\n    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202608030700/packages/xo-server/src/xo-mixins/api.mjs:475:18)"
  }

JB

@christopher-petzel Olá Chris!
Yes, this is a local XCP-ng management network and there are no other backups simultaneous.

JB

@poddingue Olá! The failure has a fixed duration of 5 minutes.

acebmxer

Just got a failure today but with a different pool.

    "message": "backup task failed with undefined error",
    "name": "Error",
    "stack": "Error: backup task failed with undefined error\n    at forwardResult (file:///usr/local/lib/node_modules/xo-server/src/_handleBackupLog.mjs:37:25)\n    at handleBackupLog (file:///usr/local/lib/node_modules/xo-server/src/_handleBackupLog.mjs:68:12)\n    at onTaskUpdate (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/metadata-backups.mjs:117:13)\n    at onTaskUpdate (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/mixins/Tasks.mjs:205:23)\n    at onLogFct (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/combineEvents.js:61:5)\n    at metadataBackup._executor (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/metadata-backups.mjs:122:11)\n    at Jobs.runJob (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/jobs/index.mjs:297:7)\n    at file:///usr/local/lib/node_modules/xo-server/src/api/schedule.mjs:84:9\n    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:204:22)\n    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:188:20)\n    at Xo.runSequence (file:///usr/local/lib/node_modules/xo-server/src/api/schedule.mjs:73:3)\n    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:204:22)\n    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:188:20)\n    at Api.#callApiMethod (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/api.mjs:475:18)"
  }
}

JB

Every day I update XO-CE hoping this error gets fixed, but so far, no solution!

olivierlambert

Is there a way in private message to give access to the XCP-ng host and XO VM in question OR to get all logs from 2 machines somewhere? 5min is clearly a timeout error.

acebmxer

@olivierlambert

Mine has not done it since last post but i have opened a support ticket along with tunnel being opened - Ticket#7762089

JB

@olivierlambert Yes, I can grant it.

marcoi

looks like mine also just started working.

JB

Master, commit be960