@florent suggesting this looks like a network issue, I took a look at my XO's server's syslog and found the error when this occurs.
2026-07-31T00:10:00.428083-04:00 mis-26-svr xo-server[1298]: 2026-07-31T04:10:00.427Z xo:xo-server ERROR uncaught exception {
2026-07-31T00:10:00.428186-04:00 mis-26-svr xo-server[1298]: error: AssertionError [ERR_ASSERTION]: The expression evaluated to a falsy value:
2026-07-31T00:10:00.428262-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428309-04:00 mis-26-svr xo-server[1298]: assert(!this.paused)
2026-07-31T00:10:00.428341-04:00 mis-26-svr xo-server[1298]:
2026-07-31T00:10:00.428391-04:00 mis-26-svr xo-server[1298]: at Parser.finish (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:302:5)
2026-07-31T00:10:00.428414-04:00 mis-26-svr xo-server[1298]: at TLSSocket.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202607200851/node_modules/undici/lib/dispatcher/client-h1.js:741:32)
2026-07-31T00:10:00.428434-04:00 mis-26-svr xo-server[1298]: at TLSSocket.emit (node:events:521:24)
2026-07-31T00:10:00.428478-04:00 mis-26-svr xo-server[1298]: at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607200851/@xen-orchestra/log/configure.js:52:17)
2026-07-31T00:10:00.428508-04:00 mis-26-svr xo-server[1298]: at endReadableNT (node:internal/streams/readable:1729:12)
2026-07-31T00:10:00.428525-04:00 mis-26-svr xo-server[1298]: at processTicksAndRejections (node:internal/process/task_queues:90:21) {
2026-07-31T00:10:00.428549-04:00 mis-26-svr xo-server[1298]: generatedMessage: true,
2026-07-31T00:10:00.428567-04:00 mis-26-svr xo-server[1298]: code: 'ERR_ASSERTION',
2026-07-31T00:10:00.428582-04:00 mis-26-svr xo-server[1298]: actual: false,
2026-07-31T00:10:00.428599-04:00 mis-26-svr xo-server[1298]: expected: true,
2026-07-31T00:10:00.428614-04:00 mis-26-svr xo-server[1298]: operator: '==',
2026-07-31T00:10:00.428628-04:00 mis-26-svr xo-server[1298]: diff: 'simple'
2026-07-31T00:10:00.428642-04:00 mis-26-svr xo-server[1298]: }
2026-07-31T00:10:00.428659-04:00 mis-26-svr xo-server[1298]: }
So it looks like xo-server is dying, could it be the module undici that the problem here? Keep in mind that 13 other instances of xo-server performed the backup without a problem at the exact same time. Some days this never happens, some days it could be 4 instances of xo-server that die the same way in executing the backup.
I think it's also important to note that the instance of xo-server dying happens immediately upon the backup being started at 10 minutes after midnight. I would think that if the problem were the network itself, that there would be some timeout period before causing the xo-server instance to crash.
To answer a question asked of @jb , this is a local XCP-ng management network and there are no other backups occurring at 00:10.
