Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"
-
Hi! Can you try to replicate on the latest commit?
-
@olivierlambert commit 59306
-
@JB this looks like an issue while reading the export part. Did you configure a default backup network in the pool ?
is there any firewall/sdn controller that can filter out some reading ?note that we have removed several point that where hiding some errors this month.
-
@florent Between one error and another, the backup sometimes succeeds.
-
-
P poddingue marked this topic as a question
-
@JB this really looks like a network issue
are the host/xoa remotes ?
-
@florent hi! Hosts/XOA on the same 192.168.0.x network.
-
-
Hi,
for several weeks, since 23 June 2026 (see below), I have been experiencing the same issue. XO (from source; daily updated) is running as a VM on the host (updated on the day or day + 1 the patches were released) being backed up.result name "BodyTimeoutError" code "UND_ERR_BODY_TIMEOUT" message "Body Timeout Error" stack "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/dispatcher/client-h1.js:688:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:605:17)\n at processTimers (node:internal/timers:541:7)"Configuration 2026-07-31 20:22 2026-07-31 20:22 a few seconds Configuration 2026-07-31 20:16 2026-07-31 20:21 5 minutes Configuration 2026-07-31 20:09 2026-07-31 20:14 5 minutes Configuration 2026-07-31 19:50 2026-07-31 19:55 5 minutes Configuration 2026-07-30 20:53 2026-07-30 20:53 a few seconds Configuration 2026-07-30 19:50 2026-07-30 19:50 a few seconds Configuration 2026-07-29 19:50 2026-07-29 19:50 a few seconds Configuration 2026-07-28 20:05 2026-07-28 20:05 a few seconds Configuration 2026-07-28 19:50 2026-07-28 19:55 5 minutes Configuration 2026-07-27 19:50 2026-07-27 19:50 a few seconds Configuration 2026-07-26 20:49 2026-07-26 20:49 a few seconds Configuration 2026-07-26 19:50 2026-07-26 19:55 5 minutes Configuration 2026-07-25 19:50 2026-07-25 19:50 a few seconds Configuration 2026-07-24 19:50 2026-07-24 19:50 a few seconds Configuration 2026-07-23 20:48 2026-07-23 20:48 a few seconds Configuration 2026-07-23 19:50 2026-07-23 19:55 5 minutes Configuration 2026-07-22 19:50 2026-07-22 19:50 a few seconds Configuration 2026-07-21 19:50 2026-07-21 19:50 a few seconds Configuration 2026-07-20 19:50 2026-07-20 19:50 a few seconds Configuration 2026-07-19 19:50 2026-07-19 19:50 a few seconds Configuration 2026-07-19 07:47 2026-07-19 07:47 a few seconds Configuration 2026-07-18 19:50 2026-07-18 19:50 a few seconds Configuration 2026-07-17 19:50 2026-07-17 19:50 a few seconds Configuration 2026-07-16 19:50 2026-07-16 19:50 a few seconds Configuration 2026-07-15 19:50 2026-07-15 19:50 a few seconds Configuration 2026-07-14 19:50 2026-07-14 19:55 6 minutes Configuration 2026-07-13 23:03 2026-07-13 23:03 a few seconds Configuration 2026-07-13 19:50 2026-07-13 19:55 5 minutes Configuration 2026-07-12 19:50 2026-07-12 19:50 a few seconds Configuration 2026-07-11 19:50 2026-07-11 19:50 a few seconds Configuration 2026-07-10 19:50 2026-07-10 19:50 a few seconds Configuration 2026-07-09 19:50 2026-07-09 19:55 5 minutes Configuration 2026-07-08 19:50 2026-07-08 19:50 a few seconds Configuration 2026-07-07 19:50 2026-07-07 19:55 5 minutes Configuration 2026-07-06 19:50 2026-07-06 19:50 a few seconds Configuration 2026-07-06 13:34 2026-07-06 13:34 a few seconds Configuration 2026-07-05 19:50 2026-07-05 19:55 6 minutes Configuration 2026-07-04 19:50 2026-07-04 19:50 a few seconds Configuration 2026-07-03 19:50 2026-07-03 19:50 a few seconds Configuration 2026-07-02 19:50 2026-07-02 19:50 a few seconds Configuration 2026-07-01 19:50 2026-07-01 19:50 a few seconds Configuration 2026-07-01 06:28 2026-07-01 06:28 a few seconds Configuration 2026-07-01 00:52 2026-07-01 00:57 5 minutes Configuration 2026-06-30 19:50 2026-06-30 19:55 5 minutes Configuration 2026-06-29 20:23 2026-06-29 20:23 a few seconds Configuration 2026-06-29 19:50 2026-06-29 19:55 5 minutes Configuration 2026-06-28 19:50 2026-06-28 19:50 a few seconds Configuration 2026-06-27 19:50 2026-06-27 19:50 a few seconds Configuration 2026-06-26 19:50 2026-06-26 19:50 a few seconds Configuration 2026-06-25 19:50 2026-06-25 19:50 a few seconds Configuration 2026-06-24 19:50 2026-06-24 19:50 a few seconds Configuration 2026-06-24 07:03 2026-06-24 07:03 a few seconds Configuration 2026-06-23 19:50 2026-06-23 19:55 5 minutes -
Looking threw the backup logs at work and it seems i might be having a similar issue with only one of my pools and it does not happen all the time....
Dates of errors... Each time if I restarted the tool stack for host 1 on the effected pool the follow run of the job would be successful until one day it fails again. Let me know if you would like for me to open a support tunel.
7/30/26
7/25/26
7/24/26}, "id": "1784976796895", "jobId": "42b76839-453e-4486-83e7-d736d8625ce5", "jobName": "Tilton Pool Backup", "message": "backup", "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df", "scheduleId": "828b5e60-4302-455c-94f1-479a8001f955", "start": 1784976796895, "status": "failure", "tasks": [ { "id": "0ms093lzi-tuc7iks1oqp", "start": 1784976800526, "status": "success", "tasks": [ { "id": "0ms093lzj-fo91s5yb7xv", "start": 1784976800527, "status": "success", "end": 1784976800603, "message": "Starting XO metadata backup for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)", "data": { "id": "3022703c-5029-4ac3-bda1-3a76c6634a39", "type": "remote" } } ], "end": 1784976800603, "message": "Starting XO metadata backup. (42b76839-453e-4486-83e7-d736d8625ce5)", "data": { "type": "xo" } }, { "id": "0ms093lzn-gw1xvnk515", "start": 1784976800531, "status": "failure", "tasks": [ { "id": "0ms093m09-e5tj4brl4hh", "start": 1784976800553, "status": "failure", "end": 1784977100916, "result": { "name": "BodyTimeoutError", "code": "UND_ERR_BODY_TIMEOUT", "message": "Body Timeout Error", "stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/dispatcher/client-h1.js:702:28)\n at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/@xen-orchestra/proxy/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:585:17)\n at process.processTimers (node:internal/timers:521:7)" }, "message": "Starting metadata backup for the pool (64466023-8ef0-f5de-54ed-57e30eca644b) for the remote (3022703c-5029-4ac3-bda1-3a76c6634a39). (42b76839-453e-4486-83e7-d736d8625ce5)", "data": { "id": "3022703c-5029-4ac3-bda1-3a76c6634a39", "type": "remote", "progress": 0 } } ], "end": 1784977100917, -
@florent suggesting this looks like a network issue, I took a look at my XO's server's syslog and found the error when this occurs.
2026-07-31T00:10:00.428083-04:00 mis-26-svr xo-server[1298]: 2026-07-31T04:10:00.427Z xo:xo-server ERROR uncaught exception { 2026-07-31T00:10:00.428186-04:00 mis-26-svr xo-server[1298]: error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value: 2026-07-31T00:10:00.428262-04:00 mis-26-svr xo-server[1298]: 2026-07-31T00:10:00.428309-04:00 mis-26-svr xo-server[1298]: assert(!this.paused) 2026-07-31T00:10:00.428341-04:00 mis-26-svr xo-server[1298]: 2026-07-31T00:10:00.428391-04:00 mis-26-svr xo-server[1298]: at Parser.finish (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:302:5) 2026-07-31T00:10:00.428414-04:00 mis-26-svr xo-server[1298]: at TLSSocket.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:741:32) 2026-07-31T00:10:00.428434-04:00 mis-26-svr xo-server[1298]: at TLSSocket.emit (node:events:521:24) 2026-07-31T00:10:00.428478-04:00 mis-26-svr xo-server[1298]: at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607200851/@xen-orchestra/log/configure.js:52:17) 2026-07-31T00:10:00.428508-04:00 mis-26-svr xo-server[1298]: at endReadableNT (node:internal/streams/readable:1729:12) 2026-07-31T00:10:00.428525-04:00 mis-26-svr xo-server[1298]: at processTicksAndRejections (node:internal/process/task_queues:90:21) { 2026-07-31T00:10:00.428549-04:00 mis-26-svr xo-server[1298]: generatedMessage: true, 2026-07-31T00:10:00.428567-04:00 mis-26-svr xo-server[1298]: code: 'ERR_ASSERTION', 2026-07-31T00:10:00.428582-04:00 mis-26-svr xo-server[1298]: actual: false, 2026-07-31T00:10:00.428599-04:00 mis-26-svr xo-server[1298]: expected: true, 2026-07-31T00:10:00.428614-04:00 mis-26-svr xo-server[1298]: operator: '==', 2026-07-31T00:10:00.428628-04:00 mis-26-svr xo-server[1298]: diff: 'simple' 2026-07-31T00:10:00.428642-04:00 mis-26-svr xo-server[1298]: } 2026-07-31T00:10:00.428659-04:00 mis-26-svr xo-server[1298]: }So it looks like xo-server is dying, could it be the module
undicithat the problem here? Keep in mind that 13 other instances of xo-server performed the backup without a problem at the exact same time. Some days this never happens, some days it could be 4 instances of xo-server that die the same way in executing the backup.I think it's also important to note that the instance of xo-server dying happens immediately upon the backup being started at 10 minutes after midnight. I would think that if the problem were the network itself, that there would be some timeout period before causing the xo-server instance to crash.
To answer a question asked of @jb , this is a local XCP-ng management network and there are no other backups occurring at 00:10.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login