Llarge vm fails delta backup
-
Good morning,
I have a problem with the backup of a really large vm.
The vm has 4 disks with the following sizes:- 200GB
- 2TB
- 1.5TB
- 1TB
The VM runs on an XCP-NG 8.2.1 with ext storage.
I am using a Xen Orchestra Community (Xen Orchestra, commit f8303, Master, commit ad8f8)
The configured backup is of type Delta to a nas as a destination.After many hours of execution, the backup fails both using nfs and smb.
At a certain point I notice that the export/inport tasks disappear but apparently it seems that the backup continues: the space occupied on the destination continues to grow and the job status remains in running.
After a few more hours the job fails.
The backup logs are as follows.
{ "data": { "mode": "delta", "reportWhen": "always" }, "id": "1742229954295", "jobId": "86618265-dc14-434c-aaf3-5e8376623fe4", "jobName": "delta", "message": "backup", "scheduleId": "52158eff-4c17-492e-ae95-f11d7de27c4b", "start": 1742229954295, "status": "failure", "infos": [ { "data": { "vms": [ "db412111-605b-bb13-20cb-fe45ab767afc", "96080cea-76c6-2232-b453-c801920a1d9f", "b86e0a24-9836-3438-057d-cac947298fdf", "8a896312-f4c1-a9a9-f883-db036ff9d506", "adcb3b3f-16d2-3cbb-d905-5ab6e3fd43e7", "0ebe42ae-6d0a-ba9b-a8c1-64e686e71f62", "1e84c1b6-ebe9-5a4e-3b5b-9013c286caa7" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "db412111-605b-bb13-20cb-fe45ab767afc", "name_label": "FileSrv2022" }, "id": "1742229955380", "message": "backup VM", "start": 1742229955380, "status": "failure", "tasks": [ { "id": "1742229955411", "message": "clean-vm", "start": 1742229955411, "status": "success", "end": 1742229955415, "result": { "merge": false } }, { "id": "1742229955580", "message": "snapshot", "start": 1742229955580, "status": "success", "end": 1742229969690, "result": "265d070b-9a68-bb99-3426-e2d8800ffca3" }, { "data": { "id": "f7b4e6f0-94bb-45cf-8c87-e046848e564d", "isFull": true, "type": "remote" }, "id": "1742229969691", "message": "export", "start": 1742229969691, "status": "failure", "tasks": [ { "id": "1742229971026", "message": "transfer", "start": 1742229971026, "status": "failure", "end": 1742361517276, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/Vhd/VhdFile.js:195:7)\n at async VhdFile.open (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/Vhd/VhdFile.js:95:7)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/openVhd.js:15:14)\n at async Promise.all (index 0)\n at async checkVhd (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/_runners/_writers/_checkVhd.mjs:5:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/fs/dist/abstract.js:558:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/fs/dist/abstract.js:180:5)\n at async RemoteAdapter.outputStream (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/RemoteAdapter.mjs:705:5)\n at async asyncEach.concurrency (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:236:39)" } } ], "end": 1742361517279, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/Vhd/VhdFile.js:195:7)\n at async VhdFile.open (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/Vhd/VhdFile.js:95:7)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/openVhd.js:15:14)\n at async Promise.all (index 0)\n at async checkVhd (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/_runners/_writers/_checkVhd.mjs:5:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/fs/dist/abstract.js:558:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/fs/dist/abstract.js:180:5)\n at async RemoteAdapter.outputStream (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/RemoteAdapter.mjs:705:5)\n at async asyncEach.concurrency (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:236:39)" } } ], "end": 1742361517723, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "==", "message": "footer1 !== footer2", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n at VhdFile.readHeaderAndFooter (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/Vhd/VhdFile.js:195:7)\n at async VhdFile.open (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/Vhd/VhdFile.js:95:7)\n at async openVhd (/opt/xo/xo-builds/xen-orchestra-202412170947/packages/vhd-lib/openVhd.js:15:14)\n at async Promise.all (index 0)\n at async checkVhd (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/_runners/_writers/_checkVhd.mjs:5:3)\n at async NfsHandler._outputStream (/opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/fs/dist/abstract.js:558:9)\n at async NfsHandler.outputStream (/opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/fs/dist/abstract.js:180:5)\n at async RemoteAdapter.outputStream (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/RemoteAdapter.mjs:705:5)\n at async asyncEach.concurrency (file:///opt/xo/xo-builds/xen-orchestra-202412170947/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:236:39)" } }, { "data": { "type": "VM", "id": "96080cea-76c6-2232-b453-c801920a1d9f", "name_label": "TEW12DC1" }, "id": "1742229955382", "message": "backup VM", "start": 1742229955382, "status": "success", "tasks": [ { "id": "1742229955412", "message": "clean-vm", "start": 1742229955412, "status": "success", "end": 1742229955417, "result": { "merge": false } }, { "id": "1742229955581", "message": "snapshot", "start": 1742229955581, "status": "success", "end": 1742229956614, "result": "ab4d40c7-2809-2833-1594-ccef0a746f75" }, { "data": { "id": "f7b4e6f0-94bb-45cf-8c87-e046848e564d", "isFull": true, "type": "remote" }, "id": "1742229956615", "message": "export", "start": 1742229956615, "status": "success", "tasks": [ { "id": "1742229957191", "message": "transfer", "start": 1742229957191, "status": "success", "end": 1742243763381, "result": { "size": 72084292096 } }, { "id": "1742243764544", "message": "clean-vm", "start": 1742243764544, "status": "success", "end": 1742243765209, "result": { "merge": false } } ], "end": 1742243765303 } ], "end": 1742243765304 }, { "data": { "type": "VM", "id": "b86e0a24-9836-3438-057d-cac947298fdf", "name_label": "TEW19DC2" }, "id": "1742243765309", "message": "backup VM", "start": 1742243765309, "status": "success", "tasks": [ { "id": "1742243765443", "message": "clean-vm", "start": 1742243765443, "status": "success", "end": 1742243765621, "result": { "merge": false } }, { "id": "1742243765679", "message": "snapshot", "start": 1742243765679, "status": "success", "end": 1742243766542, "result": "2fee117e-3232-5e4a-6b10-09bff6b485f6" }, { "data": { "id": "f7b4e6f0-94bb-45cf-8c87-e046848e564d", "isFull": true, "type": "remote" }, "id": "1742243766542:0", "message": "export", "start": 1742243766542, "status": "success", "tasks": [ { "id": "1742243767274", "message": "transfer", "start": 1742243767274, "status": "success", "end": 1742251099064, "result": { "size": 45571916800 } }, { "id": "1742251100383", "message": "clean-vm", "start": 1742251100383, "status": "success", "end": 1742251100985, "result": { "merge": false } } ], "end": 1742251101051 } ], "end": 1742251101051 }, { "data": { "type": "VM", "id": "8a896312-f4c1-a9a9-f883-db036ff9d506", "name_label": "centos" }, "id": "1742251101069", "message": "backup VM", "start": 1742251101069, "status": "success", "tasks": [ { "id": "1742251101196", "message": "clean-vm", "start": 1742251101196, "status": "success", "end": 1742251101402, "result": { "merge": false } }, { "id": "1742251101460", "message": "snapshot", "start": 1742251101460, "status": "success", "end": 1742251102307, "result": "698975a3-2709-8ee8-2d73-351757be4d37" }, { "data": { "id": "f7b4e6f0-94bb-45cf-8c87-e046848e564d", "isFull": true, "type": "remote" }, "id": "1742251102307:0", "message": "export", "start": 1742251102307, "status": "success", "tasks": [ { "id": "1742251103003", "message": "transfer", "start": 1742251103003, "status": "success", "end": 1742251388494, "result": { "size": 1451618816 } }, { "id": "1742251389453", "message": "clean-vm", "start": 1742251389453, "status": "success", "end": 1742251390163, "result": { "merge": false } } ], "end": 1742251390206 } ], "end": 1742251390206 }, { "data": { "type": "VM", "id": "adcb3b3f-16d2-3cbb-d905-5ab6e3fd43e7", "name_label": "centos7" }, "id": "1742251390211", "message": "backup VM", "start": 1742251390211, "status": "success", "tasks": [ { "id": "1742251390325", "message": "clean-vm", "start": 1742251390325, "status": "success", "end": 1742251390527, "result": { "merge": false } }, { "id": "1742251390585", "message": "snapshot", "start": 1742251390585, "status": "success", "end": 1742251391424, "result": "0544df94-804c-e8c7-8b05-6e84f95ebf00" }, { "data": { "id": "f7b4e6f0-94bb-45cf-8c87-e046848e564d", "isFull": true, "type": "remote" }, "id": "1742251391424:0", "message": "export", "start": 1742251391424, "status": "success", "tasks": [ { "id": "1742251392106", "message": "transfer", "start": 1742251392106, "status": "success", "end": 1742262159532, "result": { "size": 72560461824 } }, { "id": "1742262160630", "message": "clean-vm", "start": 1742262160630, "status": "success", "end": 1742262161235, "result": { "merge": false } } ], "end": 1742262161294 } ], "end": 1742262161294 }, { "data": { "type": "VM", "id": "0ebe42ae-6d0a-ba9b-a8c1-64e686e71f62", "name_label": "srvdom01" }, "id": "1742262161300", "message": "backup VM", "start": 1742262161300, "status": "success", "tasks": [ { "id": "1742262161426", "message": "clean-vm", "start": 1742262161426, "status": "success", "end": 1742262161577, "result": { "merge": false } }, { "id": "1742262161639", "message": "snapshot", "start": 1742262161639, "status": "success", "end": 1742262162905, "result": "abf05fd8-8934-cc51-61f6-e22fecb36c38" }, { "data": { "id": "f7b4e6f0-94bb-45cf-8c87-e046848e564d", "isFull": true, "type": "remote" }, "id": "1742262162906", "message": "export", "start": 1742262162906, "status": "success", "tasks": [ { "id": "1742262163607", "message": "transfer", "start": 1742262163607, "status": "success", "end": 1742273509362, "result": { "size": 70087459328 } }, { "id": "1742273510399", "message": "clean-vm", "start": 1742273510399, "status": "success", "end": 1742273511063, "result": { "merge": false } } ], "end": 1742273511150 } ], "end": 1742273511151 }, { "data": { "type": "VM", "id": "1e84c1b6-ebe9-5a4e-3b5b-9013c286caa7", "name_label": "xo-ce" }, "id": "1742273511154", "message": "backup VM", "start": 1742273511154, "status": "success", "tasks": [ { "id": "1742273511356", "message": "clean-vm", "start": 1742273511356, "status": "success", "end": 1742273511553, "result": { "merge": false } }, { "id": "1742273511612", "message": "snapshot", "start": 1742273511612, "status": "success", "end": 1742273512854, "result": "aa14e880-1171-0d2e-3885-47ab8c791c67" }, { "data": { "id": "f7b4e6f0-94bb-45cf-8c87-e046848e564d", "isFull": true, "type": "remote" }, "id": "1742273512854:0", "message": "export", "start": 1742273512854, "status": "success", "tasks": [ { "id": "1742273513493", "message": "transfer", "start": 1742273513493, "status": "success", "end": 1742274467330, "result": { "size": 6769184768 } }, { "id": "1742274468420", "message": "clean-vm", "start": 1742274468420, "status": "success", "end": 1742274469111, "result": { "merge": false } } ], "end": 1742274469153 } ], "end": 1742274469153 } ], "end": 1742361517723 }Can anyone tell me what the problem might be.
-
Are you sure the backup is under 24h?
-
@olivierlambert No. The backup takes more than 24 hours.
-
That's your problem then. Check https://docs.xcp-ng.org/management/manage-locally/api/#24h-task-timeout
-
@olivierlambert wow, thanks for this trick.
I'll try it and let you know the result -
@david.bigagli
Hi Oliver,
thanks to your suggestion I solved the problem.Thanks again.
-
O olivierlambert marked this topic as a question on
-
O olivierlambert has marked this topic as solved on
-
Great! Enjoy
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login