@olivierlambert Yes, I understand that you need a bit more details about the configuration to be able to find out what's causing the problems.
I'm using XO for now, latest version as of yesterday (as I try with updating on each day I get these recurring failures), so I'm just 3 commits behind (Xen Orchestra, commit c5a26).
Tonight (as expected) 6 machines got that same backup failure. I'm now running one of them manually with a script I wrote, and it looks good so far.
I asked for more debug-options to enable, to be able to catch these errors, but did not get any reply in time for tonight's failure. I don't know how much the short backup logs might help (there is actually nothing else useful in there than the mentioned error).
The next failure is scheduled for 03:15 tonight (15 3 * * 3)
This is the configuration you might find useful:
Sequence (3c05) "Special-task machines" (15 1 * * 2)
Unnamed schedule (Docker)
Unnamed schedule (HomeAssistant)
weekly tuesday (TrueNAS)
Unnamed schedule (Win10 22H2 (XCP-VM-BU1))
Unnamed schedule (pfSense)
Unnamed schedule (unifi)
The schedules included in the sequence are disabled in the backup job setup.
Backup log for job "Docker"
{
"data": {
"mode": "delta",
"reportWhen": "failure"
},
"id": "1748301300016",
"jobId": "b9142eba-4954-414a-bbc0-180b071a6490",
"jobName": "Docker",
"message": "backup",
"scheduleId": "148eb0e1-1a22-4966-9143-ce53849875de",
"start": 1748301300016,
"status": "failure",
"infos": [
{
"data": {
"vms": [
"2af0192f-3cb3-daa2-33e6-ec8f18b0cfb1"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "2af0192f-3cb3-daa2-33e6-ec8f18b0cfb1",
"name_label": "Docker"
},
"id": "1748301324573",
"message": "backup VM",
"start": 1748301324573,
"status": "failure",
"tasks": [
{
"id": "1748301324578",
"message": "clean-vm",
"start": 1748301324578,
"status": "success",
"end": 1748301326826,
"result": {
"merge": false
}
},
{
"id": "1748301327442",
"message": "snapshot",
"start": 1748301327442,
"status": "success",
"end": 1748301336945,
"result": "2bfe93f6-4c49-2756-2e27-f620946392ab"
},
{
"data": {
"id": "2b919467-704c-4e35-bac9-2d6a43118bda",
"isFull": false,
"type": "remote"
},
"id": "1748301336945:0",
"message": "export",
"start": 1748301336945,
"status": "success",
"tasks": [
{
"id": "1748301360976",
"message": "clean-vm",
"start": 1748301360976,
"status": "success",
"end": 1748301362952,
"result": {
"merge": false
}
}
],
"end": 1748301362953
}
],
"infos": [
{
"message": "will delete snapshot data"
},
{
"data": {
"vdiRef": "OpaqueRef:ccff14c1-2a28-38fb-57e2-4040e4081ea3"
},
"message": "Snapshot data has been deleted"
},
{
"data": {
"vdiRef": "OpaqueRef:72d435e0-19f9-7b84-2f57-950c66d0c5a7"
},
"message": "Snapshot data has been deleted"
}
],
"end": 1748301362953,
"result": {
"message": "invalid HTTP header in response body",
"name": "Error",
"stack": "Error: invalid HTTP header in response body\n at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:37:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:244:5)\n at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs:31:5)\n at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/Xapi.mjs:69:7)\n at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)\n at async file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/backups/_incrementalVm.mjs:65:5\n at async Promise.all (index 0)"
}
}
],
"end": 1748301362954
}
Schedule for backup job "Docker"
{
id: 'b9142eba-4954-414a-bbc0-180b071a6490',
mode: 'delta',
name: 'Docker',
remotes: { id: '2b919467-704c-4e35-bac9-2d6a43118bda' },
settings: {
'': {
cbtDestroySnapshotData: true,
fullInterval: 28,
longTermRetention: {
daily: { retention: 32, settings: {} },
monthly: { retention: 13, settings: {} },
weekly: { retention: 5, settings: {} }
},
nbdConcurrency: 3,
preferNbd: true,
timezone: 'Europe/Stockholm'
},
'148eb0e1-1a22-4966-9143-ce53849875de': { exportRetention: 1 }
},
srs: { id: { __or: [] } },
type: 'backup',
vms: { id: '2af0192f-3cb3-daa2-33e6-ec8f18b0cfb1' }
},
Backup log for job "HomeAssistant"
{
"data": {
"mode": "delta",
"reportWhen": "failure"
},
"id": "1748301362981",
"jobId": "38f0068f-c124-4876-85d3-83f1003db60c",
"jobName": "HomeAssistant",
"message": "backup",
"scheduleId": "dcb1c759-76b8-441b-9dc0-595914e60608",
"start": 1748301362981,
"status": "failure",
"infos": [
{
"data": {
"vms": [
"ed4758f3-de34-7a7e-a46b-dc007d52f5c3"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "ed4758f3-de34-7a7e-a46b-dc007d52f5c3",
"name_label": "HomeAssistant"
},
"id": "1748301365217",
"message": "backup VM",
"start": 1748301365217,
"status": "failure",
"tasks": [
{
"id": "1748301365221",
"message": "clean-vm",
"start": 1748301365221,
"status": "success",
"end": 1748301367237,
"result": {
"merge": false
}
},
{
"id": "1748301411356",
"message": "snapshot",
"start": 1748301411356,
"status": "success",
"end": 1748301414333,
"result": "a3284471-7138-9fed-b6c2-c1c4314a6569"
},
{
"data": {
"id": "2b919467-704c-4e35-bac9-2d6a43118bda",
"isFull": false,
"type": "remote"
},
"id": "1748301414334",
"message": "export",
"start": 1748301414334,
"status": "success",
"tasks": [
{
"id": "1748301433923",
"message": "clean-vm",
"start": 1748301433923,
"status": "success",
"end": 1748301436079,
"result": {
"merge": false
}
}
],
"end": 1748301436080
}
],
"infos": [
{
"message": "will delete snapshot data"
},
{
"data": {
"vdiRef": "OpaqueRef:ee706ccd-faeb-3c3e-3f17-153b9767a44e"
},
"message": "Snapshot data has been deleted"
},
{
"data": {
"vdiRef": "OpaqueRef:6be5cd98-da3e-9c9b-e193-aa80b78608d5"
},
"message": "Snapshot data has been deleted"
}
],
"end": 1748301436080,
"result": {
"message": "invalid HTTP header in response body",
"name": "Error",
"stack": "Error: invalid HTTP header in response body\n at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:37:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:244:5)\n at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs:31:5)\n at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/Xapi.mjs:69:7)\n at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)\n at async file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/backups/_incrementalVm.mjs:65:5\n at async Promise.all (index 0)"
}
}
],
"end": 1748301436081
}
Schedule for backup job "HomeAssistant"
{
id: '38f0068f-c124-4876-85d3-83f1003db60c',
mode: 'delta',
name: 'HomeAssistant',
remotes: { id: '2b919467-704c-4e35-bac9-2d6a43118bda' },
settings: {
'': {
cbtDestroySnapshotData: true,
checkpointSnapshot: false,
fullInterval: 28,
longTermRetention: {
daily: { retention: 8, settings: {} },
monthly: { retention: 13, settings: {} },
weekly: { retention: 5, settings: {} }
},
nbdConcurrency: 3,
offlineSnapshot: true,
preferNbd: true,
timezone: 'Europe/Stockholm'
},
'dcb1c759-76b8-441b-9dc0-595914e60608': { exportRetention: 1 }
},
srs: { id: { __or: [] } },
type: 'backup',
vms: { id: 'ed4758f3-de34-7a7e-a46b-dc007d52f5c3' }
},
Backup log for job "TrueNAS"
{
"data": {
"mode": "delta",
"reportWhen": "failure"
},
"id": "1748301436105",
"jobId": "d40f8444-0364-4945-acd9-01f85ef4a53d",
"jobName": "TrueNAS",
"message": "backup",
"scheduleId": "ffd808a2-563f-4f3c-84b3-445e724ce78c",
"start": 1748301436105,
"status": "failure",
"infos": [
{
"data": {
"vms": [
"086b19ec-2162-6b28-c9d2-0e2214ab85ad"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "086b19ec-2162-6b28-c9d2-0e2214ab85ad",
"name_label": "TrueNAS"
},
"id": "1748301438810",
"message": "backup VM",
"start": 1748301438810,
"status": "failure",
"tasks": [
{
"id": "1748301438815",
"message": "clean-vm",
"start": 1748301438815,
"status": "success",
"end": 1748301441801,
"result": {
"merge": false
}
},
{
"id": "1748301442293",
"message": "snapshot",
"start": 1748301442293,
"status": "success",
"end": 1748301447502,
"result": "e6988956-774f-7421-d14d-5585bc9f06a8"
},
{
"data": {
"id": "2b919467-704c-4e35-bac9-2d6a43118bda",
"isFull": false,
"type": "remote"
},
"id": "1748301447503",
"message": "export",
"start": 1748301447503,
"status": "success",
"tasks": [
{
"id": "1748301452311",
"message": "clean-vm",
"start": 1748301452311,
"status": "success",
"end": 1748301454142,
"result": {
"merge": false
}
}
],
"end": 1748301454143
}
],
"infos": [
{
"message": "will delete snapshot data"
},
{
"data": {
"vdiRef": "OpaqueRef:a6285710-3e8d-a7cc-1f87-284146c0e5be"
},
"message": "Snapshot data has been deleted"
},
{
"data": {
"vdiRef": "OpaqueRef:02477e9a-146e-34f8-0729-a04deb9ce7a2"
},
"message": "Snapshot data has been deleted"
}
],
"end": 1748301454143,
"result": {
"message": "invalid HTTP header in response body",
"name": "Error",
"stack": "Error: invalid HTTP header in response body\n at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:37:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:244:5)\n at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs:31:5)\n at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/Xapi.mjs:69:7)\n at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)\n at async file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/backups/_incrementalVm.mjs:65:5\n at async Promise.all (index 1)"
}
}
],
"end": 1748301454143
}
Schedule for backup job "TrueNAS"
{
id: 'd40f8444-0364-4945-acd9-01f85ef4a53d',
mode: 'delta',
name: 'TrueNAS',
remotes: { id: '2b919467-704c-4e35-bac9-2d6a43118bda' },
settings: {
'': {
cbtDestroySnapshotData: true,
fullInterval: 28,
longTermRetention: {
daily: { retention: 8, settings: {} },
monthly: { retention: 13, settings: {} },
weekly: { retention: 5, settings: {} }
},
nbdConcurrency: 3,
preferNbd: true,
timezone: 'Europe/Stockholm'
},
'ffd808a2-563f-4f3c-84b3-445e724ce78c': { exportRetention: 1 }
},
srs: { id: { __or: [] } },
type: 'backup',
vms: { id: '086b19ec-2162-6b28-c9d2-0e2214ab85ad' }
},
Backup log for job "Win10 22H2 (XCP-VM-BU1)"
{
"data": {
"mode": "delta",
"reportWhen": "failure"
},
"id": "1748301454157",
"jobId": "006447e3-d632-4088-871d-d31925146165",
"jobName": "Win10 22H2 (XCP-VM-BU1)",
"message": "backup",
"scheduleId": "49db3524-1fea-4964-9eb9-1ee78c7f1533",
"start": 1748301454157,
"status": "failure",
"infos": [
{
"data": {
"vms": [
"8a2b28e8-17fe-1a22-6b69-dc9e70b54440"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "8a2b28e8-17fe-1a22-6b69-dc9e70b54440",
"name_label": "Win10 22H2 (XCP-VM-BU1)"
},
"id": "1748301457105",
"message": "backup VM",
"start": 1748301457105,
"status": "failure",
"tasks": [
{
"id": "1748301457116",
"message": "clean-vm",
"start": 1748301457116,
"status": "success",
"end": 1748301459104,
"result": {
"merge": false
}
},
{
"id": "1748301459708",
"message": "snapshot",
"start": 1748301459708,
"status": "success",
"end": 1748301465288,
"result": "6eeb6ce1-3867-1bf2-ee02-61629c335998"
},
{
"data": {
"id": "2b919467-704c-4e35-bac9-2d6a43118bda",
"isFull": false,
"type": "remote"
},
"id": "1748301465289",
"message": "export",
"start": 1748301465289,
"status": "success",
"tasks": [
{
"id": "1748301469585",
"message": "clean-vm",
"start": 1748301469585,
"status": "success",
"end": 1748301471474,
"result": {
"merge": false
}
}
],
"end": 1748301471474
}
],
"infos": [
{
"message": "will delete snapshot data"
},
{
"data": {
"vdiRef": "OpaqueRef:d93298c1-6b15-75f1-abfe-23d7f09bea18"
},
"message": "Snapshot data has been deleted"
},
{
"data": {
"vdiRef": "OpaqueRef:4962f8a3-17bf-2724-1b11-52a0b718bc93"
},
"message": "Snapshot data has been deleted"
}
],
"end": 1748301471474,
"result": {
"message": "invalid HTTP header in response body",
"name": "Error",
"stack": "Error: invalid HTTP header in response body\n at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:37:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:244:5)\n at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs:31:5)\n at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/Xapi.mjs:69:7)\n at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)\n at async file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/backups/_incrementalVm.mjs:65:5\n at async Promise.all (index 1)"
}
}
],
"end": 1748301471474
}
Schedule for backup job "Win10 22H2 (XCP-VM-BU1)"
{
id: '006447e3-d632-4088-871d-d31925146165',
mode: 'delta',
name: 'Win10 22H2 (XCP-VM-BU1)',
remotes: { id: '2b919467-704c-4e35-bac9-2d6a43118bda' },
settings: {
'': {
cbtDestroySnapshotData: true,
fullInterval: 28,
longTermRetention: {
daily: { retention: 8, settings: {} },
monthly: { retention: 13, settings: {} },
weekly: { retention: 5, settings: {} }
},
nbdConcurrency: 3,
preferNbd: true,
timezone: 'Europe/Stockholm'
},
'49db3524-1fea-4964-9eb9-1ee78c7f1533': { exportRetention: 1 }
},
srs: { id: { __or: [] } },
type: 'backup',
vms: { id: '8a2b28e8-17fe-1a22-6b69-dc9e70b54440' }
},
Backup log for job "pfSense"
{
"data": {
"mode": "delta",
"reportWhen": "failure"
},
"id": "1748301471487",
"jobId": "ed71705b-61aa-4081-b765-bfa5e2bb8dd6",
"jobName": "pfSense",
"message": "backup",
"scheduleId": "2f5fc714-9c4d-4b46-8da6-db771926de68",
"start": 1748301471487,
"status": "failure",
"infos": [
{
"data": {
"vms": [
"34bb1a0f-d662-669d-6e07-39f3ef102129"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "34bb1a0f-d662-669d-6e07-39f3ef102129",
"name_label": "pfSense"
},
"id": "1748301474076",
"message": "backup VM",
"start": 1748301474076,
"status": "failure",
"tasks": [
{
"id": "1748301474081",
"message": "clean-vm",
"start": 1748301474081,
"status": "success",
"end": 1748301475569,
"result": {
"merge": false
}
},
{
"id": "1748301476075",
"message": "snapshot",
"start": 1748301476075,
"status": "success",
"end": 1748301479451,
"result": "6fb562a0-e544-cc43-92a8-b250f0d6e55f"
},
{
"data": {
"id": "2b919467-704c-4e35-bac9-2d6a43118bda",
"isFull": false,
"type": "remote"
},
"id": "1748301479451:0",
"message": "export",
"start": 1748301479451,
"status": "success",
"tasks": [
{
"id": "1748301484740",
"message": "clean-vm",
"start": 1748301484740,
"status": "success",
"end": 1748301486146,
"result": {
"merge": false
}
}
],
"end": 1748301486147
}
],
"infos": [
{
"message": "will delete snapshot data"
},
{
"data": {
"vdiRef": "OpaqueRef:9bbe8ff4-5366-e76f-c422-b11c02f2dbeb"
},
"message": "Snapshot data has been deleted"
}
],
"end": 1748301486147,
"result": {
"message": "invalid HTTP header in response body",
"name": "Error",
"stack": "Error: invalid HTTP header in response body\n at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:37:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:244:5)\n at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs:31:5)\n at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/Xapi.mjs:69:7)\n at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)\n at async file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/backups/_incrementalVm.mjs:65:5\n at async Promise.all (index 0)"
}
}
],
"end": 1748301486148
}
Schedule for backup job "pfSense"
{
id: 'ed71705b-61aa-4081-b765-bfa5e2bb8dd6',
mode: 'delta',
name: 'pfSense',
remotes: { id: '2b919467-704c-4e35-bac9-2d6a43118bda' },
settings: {
'': {
cbtDestroySnapshotData: true,
fullInterval: 28,
longTermRetention: {
daily: { retention: 8, settings: {} },
monthly: { retention: 13, settings: {} },
weekly: { retention: 5, settings: {} }
},
nbdConcurrency: 3,
preferNbd: true,
timezone: 'Europe/Stockholm'
},
'2f5fc714-9c4d-4b46-8da6-db771926de68': { exportRetention: 1 }
},
srs: { id: { __or: [] } },
type: 'backup',
vms: { id: '34bb1a0f-d662-669d-6e07-39f3ef102129' }
},
Backup log for job "unifi"
{
"data": {
"mode": "delta",
"reportWhen": "failure"
},
"id": "1748301486170",
"jobId": "167accc6-65e1-4782-a7e1-b7af72070286",
"jobName": "unifi",
"message": "backup",
"scheduleId": "25636582-2efd-45a9-b90a-f0e2e96df179",
"start": 1748301486170,
"status": "failure",
"infos": [
{
"data": {
"vms": [
"95e2e5a4-4f21-bd2c-0af5-0a1216d05d0e"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "95e2e5a4-4f21-bd2c-0af5-0a1216d05d0e",
"name_label": "unifi"
},
"id": "1748301488467",
"message": "backup VM",
"start": 1748301488467,
"status": "failure",
"tasks": [
{
"id": "1748301488472",
"message": "clean-vm",
"start": 1748301488472,
"status": "success",
"end": 1748301490266,
"result": {
"merge": false
}
},
{
"id": "1748301490816",
"message": "snapshot",
"start": 1748301490816,
"status": "success",
"end": 1748301494252,
"result": "513ce90c-c893-45ce-6693-a74e39be90b8"
},
{
"data": {
"id": "2b919467-704c-4e35-bac9-2d6a43118bda",
"isFull": false,
"type": "remote"
},
"id": "1748301494253",
"message": "export",
"start": 1748301494253,
"status": "success",
"tasks": [
{
"id": "1748301497980",
"message": "clean-vm",
"start": 1748301497980,
"status": "success",
"end": 1748301499587,
"result": {
"merge": false
}
}
],
"end": 1748301499587
}
],
"infos": [
{
"message": "will delete snapshot data"
},
{
"data": {
"vdiRef": "OpaqueRef:062d4f15-9f44-cfb9-840c-4af51ef8a4aa"
},
"message": "Snapshot data has been deleted"
},
{
"data": {
"vdiRef": "OpaqueRef:67d3668b-bd8d-db05-76a2-f9d0da5db47a"
},
"message": "Snapshot data has been deleted"
}
],
"end": 1748301499587,
"result": {
"message": "invalid HTTP header in response body",
"name": "Error",
"stack": "Error: invalid HTTP header in response body\n at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:37:19)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/vdi.mjs:244:5)\n at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)\n at async XapiVhdStreamNbdSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs:31:5)\n at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/xapi/disks/Xapi.mjs:69:7)\n at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)\n at async file:///opt/xo/xo-builds/xen-orchestra-202505261454/@xen-orchestra/backups/_incrementalVm.mjs:65:5\n at async Promise.all (index 1)"
}
}
],
"end": 1748301499587
}
Schedule for backup job "unifi"
{
id: '167accc6-65e1-4782-a7e1-b7af72070286',
mode: 'delta',
name: 'unifi',
remotes: { id: '2b919467-704c-4e35-bac9-2d6a43118bda' },
settings: {
'': {
cbtDestroySnapshotData: true,
fullInterval: 28,
longTermRetention: {
daily: { retention: 8, settings: {} },
monthly: { retention: 13, settings: {} },
weekly: { retention: 5, settings: {} }
},
nbdConcurrency: 3,
preferNbd: true,
timezone: 'Europe/Stockholm'
},
'25636582-2efd-45a9-b90a-f0e2e96df179': { exportRetention: 1 }
},
srs: { id: { __or: [] } },
type: 'backup',
vms: { id: '95e2e5a4-4f21-bd2c-0af5-0a1216d05d0e' }
},
The job ("HomeAssistant") I ran manually after the failure, without any changes to it, from my shellscript succeeded. This is the log:
{
"data": {
"mode": "delta",
"reportWhen": "failure"
},
"id": "1748332955709",
"jobId": "38f0068f-c124-4876-85d3-83f1003db60c",
"jobName": "HomeAssistant",
"message": "backup",
"scheduleId": "dcb1c759-76b8-441b-9dc0-595914e60608",
"start": 1748332955709,
"status": "success",
"infos": [
{
"data": {
"vms": [
"ed4758f3-de34-7a7e-a46b-dc007d52f5c3"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "ed4758f3-de34-7a7e-a46b-dc007d52f5c3",
"name_label": "HomeAssistant"
},
"id": "1748332999004",
"message": "backup VM",
"start": 1748332999004,
"status": "success",
"tasks": [
{
"id": "1748332999009",
"message": "clean-vm",
"start": 1748332999009,
"status": "success",
"end": 1748333001471,
"result": {
"merge": false
}
},
{
"id": "1748333024461",
"message": "snapshot",
"start": 1748333024461,
"status": "success",
"end": 1748333027231,
"result": "7c1c5df6-5a0f-28c6-eaa3-8cf96b871bf8"
},
{
"data": {
"id": "2b919467-704c-4e35-bac9-2d6a43118bda",
"isFull": true,
"type": "remote"
},
"id": "1748333027231:0",
"message": "export",
"start": 1748333027231,
"status": "success",
"tasks": [
{
"id": "1748333029816",
"message": "transfer",
"start": 1748333029816,
"status": "success",
"end": 1748333658179,
"result": {
"size": 38681968640
}
},
{
"id": "1748333661183",
"message": "clean-vm",
"start": 1748333661183,
"status": "success",
"end": 1748333669221,
"result": {
"merge": false
}
}
],
"end": 1748333669222
}
],
"infos": [
{
"message": "will delete snapshot data"
},
{
"data": {
"vdiRef": "OpaqueRef:e40df229-f55d-e658-5242-cd0746079661"
},
"message": "Snapshot data has been deleted"
},
{
"data": {
"vdiRef": "OpaqueRef:f03dc466-24a3-5269-3e9b-5634a99219f9"
},
"message": "Snapshot data has been deleted"
}
],
"end": 1748333669222
}
],
"end": 1748333669223
}