Error: invalid HTTP header in response body

[peo]

@olivierlambert Also the same in XOA, switched "latest" in the "release channel" before updating:

Initial backup succeeds, the second one fails, the third one succeeds (but is transferred as "full" again, as the first one)

[olivierlambert]

Yes but I was talking about the sources. latest isn't master.

[florent]

@peo said in Error: invalid HTTP header in response body:

@olivierlambert Also the same in XOA, switched "latest" in the "release channel" before updating:

this is probably a CBT issue
you will have more infomation in the log ( journalctl as root )
can you try running the backup without NBD/CBT enabled ?

[peo]

@florent as described by myself and others in this thread the error occurs only when "Purge snapshot data when using CBT" is enabled.
As expected, it runs fine (every time) when "Use NBT+CBT if available" is disabled, which also disables "Purge snapshot data when using CBT".

Jun 25 06:58:25 xoa xo-server[2661108]: 2025-06-25T10:58:25.998Z xo:backups:worker INFO starting backup
Jun 25 06:58:26 xoa nfsrahead[2661133]: setting /run/xo-server/mounts/2ad70aa9-8f27-4353-8dde-5623f31cd49f readahead to 128
Jun 25 06:59:03 xoa xo-server[2661108]: 2025-06-25T10:59:03.721Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
Jun 25 06:59:03 xoa xo-server[2661108]: 2025-06-25T10:59:03.798Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
Jun 25 06:59:03 xoa xo-server[2661108]: 2025-06-25T10:59:03.822Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
Jun 25 06:59:03 xoa xo-server[2661108]: 2025-06-25T10:59:03.952Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
Jun 25 06:59:04 xoa xo-server[2661108]: 2025-06-25T10:59:04.031Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
Jun 25 07:01:37 xoa xo-server[2661108]: 2025-06-25T11:01:37.465Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
Jun 25 07:01:37 xoa xo-server[2661108]:   path: '/xo-vm-backups/30db3746-fecc-4b49-e7af-8f15d13d573c/20250625T105907Z.json',
Jun 25 07:01:37 xoa xo-server[2661108]:   actual: 10166992896,
Jun 25 07:01:37 xoa xo-server[2661108]:   expected: 10169530368
Jun 25 07:01:37 xoa xo-server[2661108]: }
Jun 25 07:01:37 xoa xo-server[2661108]: 2025-06-25T11:01:37.555Z xo:backups:worker INFO backup has ended
Jun 25 07:01:37 xoa xo-server[2661108]: 2025-06-25T11:01:37.607Z xo:backups:worker INFO process will exit {
Jun 25 07:01:37 xoa xo-server[2661108]:   duration: 191607947,
Jun 25 07:01:37 xoa xo-server[2661108]:   exitCode: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:   resourceUsage: {
Jun 25 07:01:37 xoa xo-server[2661108]:     userCPUTime: 122499805,
Jun 25 07:01:37 xoa xo-server[2661108]:     systemCPUTime: 32534032,
Jun 25 07:01:37 xoa xo-server[2661108]:     maxRSS: 125060,
Jun 25 07:01:37 xoa xo-server[2661108]:     sharedMemorySize: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     unsharedDataSize: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     unsharedStackSize: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     minorPageFault: 585389,
Jun 25 07:01:37 xoa xo-server[2661108]:     majorPageFault: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     swappedOut: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     fsRead: 2056,
Jun 25 07:01:37 xoa xo-server[2661108]:     fsWrite: 19863128,
Jun 25 07:01:37 xoa xo-server[2661108]:     ipcSent: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     ipcReceived: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     signalsCount: 0,
Jun 25 07:01:37 xoa xo-server[2661108]:     voluntaryContextSwitches: 112269,
Jun 25 07:01:37 xoa xo-server[2661108]:     involuntaryContextSwitches: 90074
Jun 25 07:01:37 xoa xo-server[2661108]:   },
Jun 25 07:01:37 xoa xo-server[2661108]:   summary: { duration: '3m', cpuUsage: '81%', memoryUsage: '122.13 MiB' }
Jun 25 07:01:37 xoa xo-server[2661108]: }

Jun 25 07:01:58 xoa xo-server[2661382]: 2025-06-25T11:01:58.035Z xo:backups:worker INFO starting backup
Jun 25 07:02:23 xoa xo-server[2661382]: 2025-06-25T11:02:23.856Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT Error: can't connect to any nbd client
Jun 25 07:02:23 xoa xo-server[2661382]:     at connectNbdClientIfPossible (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/utils.mjs:23:19)
Jun 25 07:02:23 xoa xo-server[2661382]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jun 25 07:02:23 xoa xo-server[2661382]:     at async XapiVhdCbtSource.init (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/XapiVhdCbt.mjs:75:20)
Jun 25 07:02:23 xoa xo-server[2661382]:     at async #openNbdCbt (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/Xapi.mjs:129:7)
Jun 25 07:02:23 xoa xo-server[2661382]:     at async XapiDiskSource.init (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Jun 25 07:02:23 xoa xo-server[2661382]:     at async file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_incrementalVm.mjs:65:5
Jun 25 07:02:23 xoa xo-server[2661382]:     at async Promise.all (index 0)
Jun 25 07:02:23 xoa xo-server[2661382]:     at async cancelableMap (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_cancelableMap.mjs:11:12)
Jun 25 07:02:23 xoa xo-server[2661382]:     at async exportIncrementalVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_incrementalVm.mjs:28:3)
Jun 25 07:02:23 xoa xo-server[2661382]:     at async IncrementalXapiVmBackupRunner._copy (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:38:25) {
Jun 25 07:02:23 xoa xo-server[2661382]:   code: 'NO_NBD_AVAILABLE'
Jun 25 07:02:23 xoa xo-server[2661382]: }
Jun 25 07:02:27 xoa xo-server[2661382]: 2025-06-25T11:02:27.312Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 25 07:02:27 xoa xo-server[2661382]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 25 07:02:27 xoa xo-server[2661382]:     'content-length: 318\r\n' +
Jun 25 07:02:27 xoa xo-server[2661382]:     'content-type: text/html\r\n' +
Jun 25 07:02:27 xoa xo-server[2661382]:     'connection: close\r\n' +
Jun 25 07:02:27 xoa xo-server[2661382]:     'cache-control: no-cache, no-store\r\n' +
Jun 25 07:02:27 xoa xo-server[2661382]:     '\r\n' +
Jun 25 07:02:27 xoa xo-server[2661382]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:31a2142e-c677-6c86-e916-0ac19ffbe40f; CBT metadata ]</body></html>'
Jun 25 07:02:27 xoa xo-server[2661382]: }
Jun 25 07:02:39 xoa xo-server[2661382]: 2025-06-25T11:02:39.117Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT Error: can't connect to any nbd client
Jun 25 07:02:39 xoa xo-server[2661382]:     at connectNbdClientIfPossible (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/utils.mjs:23:19)
Jun 25 07:02:39 xoa xo-server[2661382]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async XapiVhdCbtSource.init (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/XapiVhdCbt.mjs:75:20)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async #openNbdCbt (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/Xapi.mjs:129:7)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async XapiDiskSource.init (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_incrementalVm.mjs:65:5
Jun 25 07:02:39 xoa xo-server[2661382]:     at async Promise.all (index 3) {
Jun 25 07:02:39 xoa xo-server[2661382]:   code: 'NO_NBD_AVAILABLE'
Jun 25 07:02:39 xoa xo-server[2661382]: }
Jun 25 07:02:39 xoa xo-server[2661382]: 2025-06-25T11:02:39.539Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT Error: can't connect to any nbd client
Jun 25 07:02:39 xoa xo-server[2661382]:     at connectNbdClientIfPossible (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/utils.mjs:23:19)
Jun 25 07:02:39 xoa xo-server[2661382]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async XapiVhdCbtSource.init (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/XapiVhdCbt.mjs:75:20)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async #openNbdCbt (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xapi/disks/Xapi.mjs:129:7)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async XapiDiskSource.init (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Jun 25 07:02:39 xoa xo-server[2661382]:     at async file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_incrementalVm.mjs:65:5
Jun 25 07:02:39 xoa xo-server[2661382]:     at async Promise.all (index 2) {
Jun 25 07:02:39 xoa xo-server[2661382]:   code: 'NO_NBD_AVAILABLE'
Jun 25 07:02:39 xoa xo-server[2661382]: }
Jun 25 07:02:42 xoa xo-server[2661382]: 2025-06-25T11:02:42.588Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 25 07:02:42 xoa xo-server[2661382]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'content-length: 318\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'content-type: text/html\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'connection: close\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'cache-control: no-cache, no-store\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     '\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:f0379a82-6fce-c6fa-a4c7-b7b6dcc5df26; CBT metadata ]</body></html>'
Jun 25 07:02:42 xoa xo-server[2661382]: }
Jun 25 07:02:42 xoa xo-server[2661382]: 2025-06-25T11:02:42.950Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 25 07:02:42 xoa xo-server[2661382]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'content-length: 318\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'content-type: text/html\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'connection: close\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     'cache-control: no-cache, no-store\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     '\r\n' +
Jun 25 07:02:42 xoa xo-server[2661382]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:85222592-ba8f-e189-8389-6cb4d8dd038b; CBT metadata ]</body></html>'
Jun 25 07:02:42 xoa xo-server[2661382]: }
Jun 25 07:02:43 xoa xo-server[2661382]: 2025-06-25T11:02:43.467Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT XapiError: HANDLE_INVALID(VDI, OpaqueRef:0671251f-d1f0-2a16-53c8-125f2b357e0d)
Jun 25 07:02:43 xoa xo-server[2661382]:     at XapiError.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Jun 25 07:02:43 xoa xo-server[2661382]:     at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
Jun 25 07:02:43 xoa xo-server[2661382]:     at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1072:24)
Jun 25 07:02:43 xoa xo-server[2661382]:     at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1106:14
Jun 25 07:02:43 xoa xo-server[2661382]:     at Array.forEach (<anonymous>)
Jun 25 07:02:43 xoa xo-server[2661382]:     at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1096:12)
Jun 25 07:02:43 xoa xo-server[2661382]:     at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1269:14)
Jun 25 07:02:43 xoa xo-server[2661382]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Jun 25 07:02:43 xoa xo-server[2661382]:   code: 'HANDLE_INVALID',
Jun 25 07:02:43 xoa xo-server[2661382]:   params: [ 'VDI', 'OpaqueRef:0671251f-d1f0-2a16-53c8-125f2b357e0d' ],
Jun 25 07:02:43 xoa xo-server[2661382]:   call: undefined,
Jun 25 07:02:43 xoa xo-server[2661382]:   url: undefined,
Jun 25 07:02:43 xoa xo-server[2661382]:   task: task {
Jun 25 07:02:43 xoa xo-server[2661382]:     uuid: '19892e76-0681-defa-7d86-8adff4c519df',
Jun 25 07:02:43 xoa xo-server[2661382]:     name_label: 'Async.VDI.list_changed_blocks',
Jun 25 07:02:43 xoa xo-server[2661382]:     name_description: '',
Jun 25 07:02:43 xoa xo-server[2661382]:     allowed_operations: [],
Jun 25 07:02:43 xoa xo-server[2661382]:     current_operations: {},
Jun 25 07:02:43 xoa xo-server[2661382]:     created: '20250625T11:02:23Z',
Jun 25 07:02:43 xoa xo-server[2661382]:     finished: '20250625T11:02:43Z',
Jun 25 07:02:43 xoa xo-server[2661382]:     status: 'failure',
Jun 25 07:02:43 xoa xo-server[2661382]:     resident_on: 'OpaqueRef:38c38c49-d15f-e42a-7aca-ae093fca92c6',
Jun 25 07:02:43 xoa xo-server[2661382]:     progress: 1,
Jun 25 07:02:43 xoa xo-server[2661382]:     type: '<none/>',
Jun 25 07:02:43 xoa xo-server[2661382]:     result: '',
Jun 25 07:02:43 xoa xo-server[2661382]:     error_info: [
Jun 25 07:02:43 xoa xo-server[2661382]:       'HANDLE_INVALID',
Jun 25 07:02:43 xoa xo-server[2661382]:       'VDI',
Jun 25 07:02:43 xoa xo-server[2661382]:       'OpaqueRef:0671251f-d1f0-2a16-53c8-125f2b357e0d'
Jun 25 07:02:43 xoa xo-server[2661382]:     ],
Jun 25 07:02:43 xoa xo-server[2661382]:     other_config: {},
Jun 25 07:02:43 xoa xo-server[2661382]:     subtask_of: 'OpaqueRef:NULL',
Jun 25 07:02:43 xoa xo-server[2661382]:     subtasks: [],
Jun 25 07:02:43 xoa xo-server[2661382]:     backtrace: '(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 11643))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 144))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 188))((process xapi)(filename ocaml/xapi/rbac.ml)(line 197))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 77)))'
Jun 25 07:02:43 xoa xo-server[2661382]:   }
Jun 25 07:02:43 xoa xo-server[2661382]: }
Jun 25 07:02:46 xoa xo-server[2661382]: 2025-06-25T11:02:46.867Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 25 07:02:46 xoa xo-server[2661382]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 25 07:02:46 xoa xo-server[2661382]:     'content-length: 346\r\n' +
Jun 25 07:02:46 xoa xo-server[2661382]:     'content-type: text/html\r\n' +
Jun 25 07:02:46 xoa xo-server[2661382]:     'connection: close\r\n' +
Jun 25 07:02:46 xoa xo-server[2661382]:     'cache-control: no-cache, no-store\r\n' +
Jun 25 07:02:46 xoa xo-server[2661382]:     '\r\n' +
Jun 25 07:02:46 xoa xo-server[2661382]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:0671251f-d1f0-2a16-53c8-125f2b357e0d&quot;)</body></html>'
Jun 25 07:02:46 xoa xo-server[2661382]: }
Jun 25 07:02:52 xoa xo-server[2661382]: 2025-06-25T11:02:52.120Z xo:backups:worker INFO backup has ended
Jun 25 07:02:52 xoa xo-server[2661382]: 2025-06-25T11:02:52.133Z xo:backups:worker INFO process will exit {
Jun 25 07:02:52 xoa xo-server[2661382]:   duration: 54097776,
Jun 25 07:02:52 xoa xo-server[2661382]:   exitCode: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:   resourceUsage: {
Jun 25 07:02:52 xoa xo-server[2661382]:     userCPUTime: 2370678,
Jun 25 07:02:52 xoa xo-server[2661382]:     systemCPUTime: 266735,
Jun 25 07:02:52 xoa xo-server[2661382]:     maxRSS: 37208,
Jun 25 07:02:52 xoa xo-server[2661382]:     sharedMemorySize: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     unsharedDataSize: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     unsharedStackSize: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     minorPageFault: 22126,
Jun 25 07:02:52 xoa xo-server[2661382]:     majorPageFault: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     swappedOut: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     fsRead: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     fsWrite: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     ipcSent: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     ipcReceived: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     signalsCount: 0,
Jun 25 07:02:52 xoa xo-server[2661382]:     voluntaryContextSwitches: 2163,
Jun 25 07:02:52 xoa xo-server[2661382]:     involuntaryContextSwitches: 658
Jun 25 07:02:52 xoa xo-server[2661382]:   },
Jun 25 07:02:52 xoa xo-server[2661382]:   summary: { duration: '54s', cpuUsage: '5%', memoryUsage: '36.34 MiB' }
Jun 25 07:02:52 xoa xo-server[2661382]: }

[florent]

@peo said in Error: invalid HTTP header in response body:

can't connect through NBD, fallback to stream export

"maybe it's the can't connect through NBD"

do you have VM with a lot of disks ? if yes, can you reduce the concurrency , or the number of nbd connection ?

[markxc]

@olivierlambert

dear olivier is the bug now fixed in master branch?

br

Mark

[peo]

@florent said in Error: invalid HTTP header in response body:

@peo said in Error: invalid HTTP header in response body:

can't connect through NBD, fallback to stream export

"maybe it's the can't connect through NBD"

do you have VM with a lot of disks ? if yes, can you reduce the concurrency , or the number of nbd connection ?

it's independent of the number of disks attached to the VM and the NBD concurrency

[markxc]

I have updated to last master branch bda87

issue remains

Error: invalid HTTP header in response body
Type: full

[peo]

After updating to yesterday's master ("a348c", just before a non-important update for this case), now also "continuous replication" jobs fail (gets stuck):

The jobs here are stuck at the transfer stage (both as delta, both to a local SSD on the destination host) since they started 14:30 and 21:00 yesterday.

Update:
I have restarted the VM to be backed up (replicated), updated XO (source) and tried again. Running time so far (after some retries which were cancelled because I updated and restarted stuff) more than 90 minutes (the disk on this VM is 27GB, should take at most 10 minutes to transfer, at only 50% efficiency over the gigabit connection to the other host).
The two other stuck jobs were also cancelled at the update/reboots, one (21:00) was started using the scheduler in XO, the other triggered through crontab (xo-cli script I wrote) on my host with the XO installation, and now when troubleshooting I start the job manually from the overview (as can be seen, makes no difference).
Next steps:
I will change the destination to another host, assuming my SSD in the host it tries to replicate to is broken (but I can list the files on it, and it shows that the timestamp of the vhd for the job is updated recently/all the time)

Another thought:
Is this maybe a new feature ? A true "continuous replication" as a service which never stops ?

Imaginary problem solved:
Replication job still locked up when I changed the destination (let it run for more than 5 hours), so I reverted to the replicated version of my xo VM (Deb12-XO) from before the update of XO to 'a348c' (now running '1a7b5' again). Replication now works, and I accidentally made a new (first) copy of the broken one (forgot to change VM in the backup job).

[flakpyro]

Mine was working on the latest commit however tonight due to a power failure my XO VM rebooted and now i too am experiencing the same errors. My hosts also rebooted as a result. Rolled back to a build from earlier this month and all is well again.