Strange backup error. false !== true
-
@Andrew Thank you, I'm investigating
-
I hope I'm not hijacking the thread, but having the same problem since over a week. A delta backup job with 4 VMs. 1 constantly failing with the same error. NFS share as remote.
commit 395d8{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1656280800006", "jobId": "974bff41-5d64-49a9-bdcf-1638ea45802f", "jobName": "Daily Delta backup", "message": "backup", "scheduleId": "df3701cc-ca3e-4a7c-894c-5899a0d3c3e7", "start": 1656280800006, "status": "failure", "infos": [ { "data": { "vms": [ "c07d9c18-065c-be11-2a6d-a6c3988b0538", "15d248ea-f95c-332f-8aa8-851a49a29494", "93f1634d-ecaa-e4b7-b9b5-2c2fc4a76695", "d71b5012-b13d-eae5-71fc-5028fa44611e" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c07d9c18-065c-be11-2a6d-a6c3988b0538" }, "id": "1656280802155", "message": "backup VM", "start": 1656280802155, "status": "failure", "tasks": [ { "id": "1656280802805", "message": "clean-vm", "start": 1656280802805, "status": "success", "end": 1656280802835, "result": { "merge": false } }, { "id": "1656280802874", "message": "snapshot", "start": 1656280802874, "status": "success", "end": 1656280809980, "result": "b6daf2e9-bbed-97fc-ed59-a6070a948cf9" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": true, "type": "remote" }, "id": "1656280809981", "message": "export", "start": 1656280809981, "status": "failure", "tasks": [ { "id": "1656280810039", "message": "transfer", "start": 1656280810039, "status": "failure", "end": 1656281025733, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:189:7)\n at async Function.open (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:93:5)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/openVhd.js:10:12)\n at async Promise.all (index 0)\n at async checkVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/_checkVhd.js:7:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:604:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:245:5)\n at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:594:5)\n at async RemoteAdapter.writeVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:589:7)\n at async /opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/DeltaBackupWriter.js:233:11" } } ], "end": 1656281025734, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:189:7)\n at async Function.open (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:93:5)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/openVhd.js:10:12)\n at async Promise.all (index 0)\n at async checkVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/_checkVhd.js:7:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:604:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:245:5)\n at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:594:5)\n at async RemoteAdapter.writeVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:589:7)\n at async /opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/DeltaBackupWriter.js:233:11" } } ], "end": 1656281026510, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:189:7)\n at async Function.open (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/Vhd/VhdFile.js:93:5)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/packages/vhd-lib/openVhd.js:10:12)\n at async Promise.all (index 0)\n at async checkVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/_checkVhd.js:7:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:604:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/fs/dist/abstract.js:245:5)\n at async RemoteAdapter.outputStream (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:594:5)\n at async RemoteAdapter.writeVhd (/opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/RemoteAdapter.js:589:7)\n at async /opt/xo/xo-builds/xen-orchestra-202206241101/@xen-orchestra/backups/writers/DeltaBackupWriter.js:233:11" } }, { "data": { "type": "VM", "id": "15d248ea-f95c-332f-8aa8-851a49a29494" }, "id": "1656280802194", "message": "backup VM", "start": 1656280802194, "status": "success", "tasks": [ { "id": "1656280802804", "message": "clean-vm", "start": 1656280802804, "status": "success", "end": 1656280802921, "result": { "merge": false } }, { "id": "1656280803262", "message": "snapshot", "start": 1656280803262, "status": "success", "end": 1656280821804, "result": "6e02fc2a-9fbc-6e3b-f895-a4dd079105f8" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": false, "type": "remote" }, "id": "1656280821805", "message": "export", "start": 1656280821805, "status": "success", "tasks": [ { "id": "1656280822234", "message": "transfer", "start": 1656280822234, "status": "success", "end": 1656281410464, "result": { "size": 12277793792 } }, { "id": "1656281411017", "message": "clean-vm", "start": 1656281411017, "status": "success", "end": 1656281411097, "result": { "merge": false } } ], "end": 1656281411101 } ], "end": 1656281411102 }, { "data": { "type": "VM", "id": "93f1634d-ecaa-e4b7-b9b5-2c2fc4a76695" }, "id": "1656281026511", "message": "backup VM", "start": 1656281026511, "status": "success", "tasks": [ { "id": "1656281026565", "message": "clean-vm", "start": 1656281026565, "status": "success", "end": 1656281026660, "result": { "merge": false } }, { "id": "1656281026800", "message": "snapshot", "start": 1656281026800, "status": "success", "end": 1656281028768, "result": "d59b9d10-69c1-d55f-664b-f1320d12edf5" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": false, "type": "remote" }, "id": "1656281028769", "message": "export", "start": 1656281028769, "status": "success", "tasks": [ { "id": "1656281028808", "message": "transfer", "start": 1656281028808, "status": "success", "end": 1656281035797, "result": { "size": 121699840 } }, { "id": "1656281036348", "message": "clean-vm", "start": 1656281036348, "status": "success", "end": 1656281037475, "result": { "merge": false } } ], "end": 1656281037657 } ], "end": 1656281037658 }, { "data": { "type": "VM", "id": "d71b5012-b13d-eae5-71fc-5028fa44611e" }, "id": "1656281037658:0", "message": "backup VM", "start": 1656281037658, "status": "success", "tasks": [ { "id": "1656281037766", "message": "clean-vm", "start": 1656281037766, "status": "success", "end": 1656281037817, "result": { "merge": false } }, { "id": "1656281037940", "message": "snapshot", "start": 1656281037940, "status": "success", "end": 1656281039700, "result": "cb25e9c2-d870-2760-2ce5-5a0ee6c141f3" }, { "data": { "id": "e0bd1fee-ca05-4f0e-98bf-595c41be9924", "isFull": false, "type": "remote" }, "id": "1656281039700:0", "message": "export", "start": 1656281039700, "status": "success", "tasks": [ { "id": "1656281039771", "message": "transfer", "start": 1656281039771, "status": "success", "end": 1656281064244, "result": { "size": 669190144 } }, { "id": "1656281065362", "message": "clean-vm", "start": 1656281065362, "status": "success", "end": 1656281066689, "result": { "merge": false } } ], "end": 1656281066938 } ], "end": 1656281066939 } ], "end": 1656281411102 } -
@Andrew Could show us the info of the problematic VHD files?
> cd xo-vm-backups/<VM UUID>/vdis/<job UUID>/<VDI UUID>/ > npx vhd-cli info *.vhd -
@julien-f No, it's on S3.
-
@Andrew Interesting
-
-
-
@julien-f unfortunately it did not help. I removed all snapshots for this VM in snapshots tab of this VM in XO. This caused some coalescing in SR>Advanced tab of SR where this VM is located. I waited till it ended and tried to run backup again. I got the same error.
-
@krakers This is unrelated to the current thread and very likely not an XO problem. Please open a separate thread, maybe the community will be able to help.
-
@julien-f Restore showed that there were backups even when it said it failed.... I did not test one, sorry.
I deleted the snapshots and old backup data which forced a full backup to S3 and XO was happy. A delta backup after that was also successful.
