Strange backup error. false !== true
-
@florent @julien-f Running a S3 delta backup that has worked before. Other VM backups in the same job work. This one VM does not backup any more.
Running XO source, current master commit 395d8.
AssertionError. Expected values to be strictly equal: false !== true{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1656279942960", "jobId": "f9405e51-da48-4eb1-980d-99da34359b3c", "jobName": "wasabi-Backup", "message": "backup", "scheduleId": "4b919ebd-37cf-4048-ab9f-745772b4a97d", "start": 1656279942960, "status": "failure", "infos": [ { "data": { "vms": [ "2ff4f95d-7e3a-8bcf-baf1-ef1d78d4b902" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "2ff4f95d-7e3a-8bcf-baf1-ef1d78d4b902" }, "id": "1656279944045", "message": "backup VM", "start": 1656279944045, "status": "failure", "tasks": [ { "id": "1656279945329", "message": "clean-vm", "start": 1656279945329, "status": "failure", "tasks": [ { "id": "1656279951412", "message": "merge", "start": 1656279951412, "status": "failure", "end": 1656279952377, "result": { "generatedMessage": true, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "Expected values to be strictly equal:\n\nfalse !== true\n", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:\n\nfalse !== true\n\n at Function.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202206241045/packages/vhd-lib/merge.js:120:14)\n at AsyncGenerator.next (<anonymous>)\n at wrapCall (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/wrapCall.js:7:23)\n at loop (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:116:25)\n at onEvalDisposable (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:112:42)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)" } } ], "end": 1656279952377, "result": { "generatedMessage": true, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "Expected values to be strictly equal:\n\nfalse !== true\n", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:\n\nfalse !== true\n\n at Function.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202206241045/packages/vhd-lib/merge.js:120:14)\n at AsyncGenerator.next (<anonymous>)\n at wrapCall (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/wrapCall.js:7:23)\n at loop (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:116:25)\n at onEvalDisposable (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:112:42)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)" } }, { "id": "1656279959826", "message": "snapshot", "start": 1656279959826, "status": "success", "end": 1656279961823, "result": "708e79b5-1bdc-82ef-e751-390f5bcff99f" }, { "data": { "id": "db9ad0a8-bce6-4a2b-b9fd-5c4cecf059c4", "isFull": false, "type": "remote" }, "id": "1656279961823:0", "message": "export", "start": 1656279961823, "status": "failure", "tasks": [ { "id": "1656279962034", "message": "transfer", "start": 1656279962034, "status": "success", "end": 1656279970302, "result": { "size": 107003904 } }, { "id": "1656279970885", "message": "clean-vm", "start": 1656279970885, "status": "failure", "tasks": [ { "id": "1656279979398", "message": "merge", "start": 1656279979398, "status": "failure", "end": 1656279980298, "result": { "generatedMessage": true, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "Expected values to be strictly equal:\n\nfalse !== true\n", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:\n\nfalse !== true\n\n at Function.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202206241045/packages/vhd-lib/merge.js:120:14)\n at AsyncGenerator.next (<anonymous>)\n at wrapCall (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/wrapCall.js:7:23)\n at loop (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:116:25)\n at onEvalDisposable (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:112:42)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)" } } ], "end": 1656279980299, "result": { "generatedMessage": true, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "Expected values to be strictly equal:\n\nfalse !== true\n", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:\n\nfalse !== true\n\n at Function.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202206241045/packages/vhd-lib/merge.js:120:14)\n at AsyncGenerator.next (<anonymous>)\n at wrapCall (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/wrapCall.js:7:23)\n at loop (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:116:25)\n at onEvalDisposable (/opt/xo/xo-builds/xen-orchestra-202206241045/node_modules/promise-toolbox/Disposable.js:112:42)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)" } } ], "end": 1656279980354 } ], "end": 1656279980354 } ], "end": 1656279980354 } -
@Andrew Thank you, I'm investigating
-
I hope I'm not hijacking the thread, but having the same problem since over a week. A delta backup job with 4 VMs. 1 constantly failing with the same error. NFS share as remote.
commit 395d8{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1656280800006", "jobId": "974bff41-5d64-49a9-bdcf-1638ea45802f", "jobName": "Daily Delta backup", "message": "backup", "scheduleId": "df3701cc-ca3e-4a7c-894c-5899a0d3c3e7", "start": 1656280800006, "status": "failure", "infos": [ { "data": { "vms": [ "c07d9c18-065c-be11-2a6d-a6c3988b0538", "15d248ea-f95c-332f-8aa8-851a49a29494", "93f1634d-ecaa-e4b7-b9b5-2c2fc4a76695", "d71b5012-b13d-eae5-71fc-5028fa44611e" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c07d9c18-065c-be11-2a6d-a6c3988b0538" }, "id": "1656280802155", "message": "backup VM", "start": 1656280802155, "status": "failure", "tasks": [ { "id": "1656280802805", "message": "clean-vm", "start": 1656280802805, "status": "success", "end": 1656280802835, "result": { "merge": false } }, { "id": "1656280802874", "message": "snapshot", "start": 1656280802874, "status": "success", "end": 1656280809980, "result": "b6daf2e9-bbed-97fc-ed59-a6070a948cf9" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": true, "type": "remote" }, "id": "1656280809981", "message": "export", "start": 1656280809981, "status": "failure", "tasks": [ { "id": "1656280810039", "message": "transfer", "start": 1656280810039, "status": "failure", "end": 1656281025733, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:189:7)\n at async Function.open (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:93:5)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/openVhd.js:10:12)\n at async Promise.all (index 0)\n at async checkVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/_checkVhd.js:7:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:604:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:245:5)\n at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:594:5)\n at async RemoteAdapter.writeVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:589:7)\n at async /opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/DeltaBackupWriter.js:233:11" } } ], "end": 1656281025734, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:189:7)\n at async Function.open (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:93:5)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/openVhd.js:10:12)\n at async Promise.all (index 0)\n at async checkVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/_checkVhd.js:7:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:604:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:245:5)\n at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:594:5)\n at async RemoteAdapter.writeVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:589:7)\n at async /opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/DeltaBackupWriter.js:233:11" } } ], "end": 1656281026510, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:189:7)\n at async Function.open (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:93:5)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/openVhd.js:10:12)\n at async Promise.all (index 0)\n at async checkVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/_checkVhd.js:7:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:604:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:245:5)\n at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:594:5)\n at async RemoteAdapter.writeVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:589:7)\n at async /opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/DeltaBackupWriter.js:233:11" } }, { "data": { "type": "VM", "id": "15d248ea-f95c-332f-8aa8-851a49a29494" }, "id": "1656280802194", "message": "backup VM", "start": 1656280802194, "status": "success", "tasks": [ { "id": "1656280802804", "message": "clean-vm", "start": 1656280802804, "status": "success", "end": 1656280802921, "result": { "merge": false } }, { "id": "1656280803262", "message": "snapshot", "start": 1656280803262, "status": "success", "end": 1656280821804, "result": "6e02fc2a-9fbc-6e3b-f895-a4dd079105f8" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": false, "type": "remote" }, "id": "1656280821805", "message": "export", "start": 1656280821805, "status": "success", "tasks": [ { "id": "1656280822234", "message": "transfer", "start": 1656280822234, "status": "success", "end": 1656281410464, "result": { "size": 12277793792 } }, { "id": "1656281411017", "message": "clean-vm", "start": 1656281411017, "status": "success", "end": 1656281411097, "result": { "merge": false } } ], "end": 1656281411101 } ], "end": 1656281411102 }, { "data": { "type": "VM", "id": "93f1634d-ecaa-e4b7-b9b5-2c2fc4a76695" }, "id": "1656281026511", "message": "backup VM", "start": 1656281026511, "status": "success", "tasks": [ { "id": "1656281026565", "message": "clean-vm", "start": 1656281026565, "status": "success", "end": 1656281026660, "result": { "merge": false } }, { "id": "1656281026800", "message": "snapshot", "start": 1656281026800, "status": "success", "end": 1656281028768, "result": "d59b9d10-69c1-d55f-664b-f1320d12edf5" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": false, "type": "remote" }, "id": "1656281028769", "message": "export", "start": 1656281028769, "status": "success", "tasks": [ { "id": "1656281028808", "message": "transfer", "start": 1656281028808, "status": "success", "end": 1656281035797, "result": { "size": 121699840 } }, { "id": "1656281036348", "message": "clean-vm", "start": 1656281036348, "status": "success", "end": 1656281037475, "result": { "merge": false } } ], "end": 1656281037657 } ], "end": 1656281037658 }, { "data": { "type": "VM", "id": "d71b5012-b13d-eae5-71fc-5028fa44611e" }, "id": "1656281037658:0", "message": "backup VM", "start": 1656281037658, "status": "success", "tasks": [ { "id": "1656281037766", "message": "clean-vm", "start": 1656281037766, "status": "success", "end": 1656281037817, "result": { "merge": false } }, { "id": "1656281037940", "message": "snapshot", "start": 1656281037940, "status": "success", "end": 1656281039700, "result": "cb25e9c2-d870-2760-2ce5-5a0ee6c141f3" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": false, "type": "remote" }, "id": "1656281039700:0", "message": "export", "start": 1656281039700, "status": "success", "tasks": [ { "id": "1656281039771", "message": "transfer", "start": 1656281039771, "status": "success", "end": 1656281064244, "result": { "size": 669190144 } }, { "id": "1656281065362", "message": "clean-vm", "start": 1656281065362, "status": "success", "end": 1656281066689, "result": { "merge": false } } ], "end": 1656281066938 } ], "end": 1656281066939 } ], "end": 1656281411102 } -
@Andrew Could show us the info of the problematic VHD files?
> cd xo-vm-backups/<VM UUID>/vdis/<job UUID>/<VDI UUID>/ > npx vhd-cli info *.vhd -
@julien-f No, it's on S3.
-
@Andrew Interesting
-
-
-
@julien-f unfortunately it did not help. I removed all snapshots for this VM in snapshots tab of this VM in XO. This caused some coalescing in SR>Advanced tab of SR where this VM is located. I waited till it ended and tried to run backup again. I got the same error.
-
@krakers This is unrelated to the current thread and very likely not an XO problem. Please open a separate thread, maybe the community will be able to help.
-
@julien-f Restore showed that there were backups even when it said it failed.... I did not test one, sorry.
I deleted the snapshots and old backup data which forced a full backup to S3 and XO was happy. A delta backup after that was also successful.
