Long-term retention of backups
-
Hi
a backup problem with the new Long-term retention of backups
I have been running it for 6 days, but now I get an errorentries must be sorted in asc order 2025-01 2025-52
VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)
This is a XenServer/XCP-ng errorCould this be because of the new year ??
I updated XOCE to latest db9c8 but with the same result.
I am running:
Delta backup
2x remote target (one is SMB and one is NFS)
concurrency 1
number of retry 2
Timeout 2
no NBD or CBT
Offline mode
Backup retension 7
type Full
Number of weekly backups kept 2
Number of monthly backups kept 1The other jobs are running OK
-
Hi @ph7,
Can you post the full error from the backup log? Also, are you running XCP-ng 8.2 or 8.3?
Dan
-
@ph7 said in Long-term retention of backups:
VM_LACKS_FEATURE
This means there's no tools installed or working on that VM
-
@Danp said in Long-term retention of backups:
8.3@olivierlambert said in Long-term retention of backups:
Yes
fist 5 days without tools
after error i installed the tools. same error
Have to run, be back ASAP -
On 25 dec, I "installed" a debian vm, but without installing anything, only the iso attached.
I ran 7+ manual backups to see if the 7 retension was working, it did.
I then let the job run on schedule from 26 dec 06:00 CET
The result was Successful.{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1735448400010", "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867", "jobName": "LongTerm-5+2", "message": "backup", "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7", "start": 1735448400010, "status": "success", "infos": [ { "data": { "vms": [ "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6", "name_label": "Debian Bookworm 12" }, "id": "1735448401748", "message": "backup VM", "start": 1735448401748, "status": "success", "tasks": [ { "id": "1735448401758", "message": "clean-vm", "start": 1735448401758, "status": "success", "tasks": [ { "id": "1735448401954", "message": "merge", "start": 1735448401954, "status": "success", "end": 1735448402090 } ], "end": 1735448402140, "result": { "merge": true } }, { "id": "1735448401828", "message": "clean-vm", "start": 1735448401828, "status": "success", "tasks": [ { "id": "1735448401999", "message": "merge", "start": 1735448401999, "status": "success", "end": 1735448402201 } ], "end": 1735448402262, "result": { "merge": true } }, { "id": "1735448402589", "message": "snapshot", "start": 1735448402589, "status": "success", "end": 1735448403661, "result": "868bb4b2-1b4a-1ba3-9f04-00e86bb6098b" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": false, "type": "remote" }, "id": "1735448403662", "message": "export", "start": 1735448403662, "status": "success", "tasks": [ { "id": "1735448405593", "message": "transfer", "start": 1735448405593, "status": "success", "end": 1735448407141, "result": { "size": 145920 } }, { "id": "1735448407995", "message": "clean-vm", "start": 1735448407995, "status": "success", "end": 1735448408048, "result": { "merge": true } } ], "end": 1735448408063 }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": false, "type": "remote" }, "id": "1735448403663", "message": "export", "start": 1735448403663, "status": "success", "tasks": [ { "id": "1735448405591", "message": "transfer", "start": 1735448405591, "status": "success", "end": 1735448407139, "result": { "size": 145920 } }, { "id": "1735448408063:0", "message": "clean-vm", "start": 1735448408063, "status": "success", "end": 1735448408110, "result": { "merge": true } } ], "end": 1735448408119 } ], "end": 1735448408119 } ], "end": 1735448408120 }Then I added a proper Debian VM with 1.0.0-proto-0.4.0 tools.
I ran the job manually, Again Successful{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1735505513119", "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867", "jobName": "LongTerm-5+2", "message": "backup", "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7", "start": 1735505513119, "status": "success", "infos": [ { "data": { "vms": [ "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6", "5f029e45-a01c-310b-9895-a9e58bf92b96" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6", "name_label": "Debian" }, "id": "1735505515013", "message": "backup VM", "start": 1735505515013, "status": "success", "tasks": [ { "id": "1735505515063", "message": "clean-vm", "start": 1735505515063, "status": "success", "tasks": [ { "id": "1735505515145", "message": "merge", "start": 1735505515145, "status": "success", "end": 1735505515281 } ], "end": 1735505515317, "result": { "merge": true } }, { "id": "1735505515064", "message": "clean-vm", "start": 1735505515064, "status": "success", "tasks": [ { "id": "1735505515190", "message": "merge", "start": 1735505515190, "status": "success", "end": 1735505515397 } ], "end": 1735505515459, "result": { "merge": true } }, { "id": "1735505520519", "message": "snapshot", "start": 1735505520519, "status": "success", "end": 1735505521796, "result": "321f6c53-acca-8cac-1373-9537e3ce4836" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": false, "type": "remote" }, "id": "1735505521798", "message": "export", "start": 1735505521798, "status": "success", "tasks": [ { "id": "1735505523731", "message": "transfer", "start": 1735505523731, "status": "success", "end": 1735505525358, "result": { "size": 145920 } }, { "id": "1735505526203", "message": "clean-vm", "start": 1735505526203, "status": "success", "end": 1735505526288, "result": { "merge": true } } ], "end": 1735505526305 }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": false, "type": "remote" }, "id": "1735505521799", "message": "export", "start": 1735505521799, "status": "success", "tasks": [ { "id": "1735505523729", "message": "transfer", "start": 1735505523729, "status": "success", "end": 1735505525356, "result": { "size": 145920 } }, { "id": "1735505526305:0", "message": "clean-vm", "start": 1735505526305, "status": "success", "end": 1735505526338, "result": { "merge": true } } ], "end": 1735505526344 } ], "end": 1735505526345 }, { "data": { "type": "VM", "id": "5f029e45-a01c-310b-9895-a9e58bf92b96", "name_label": "PiHole 🗑️" }, "id": "1735505526351", "message": "backup VM", "start": 1735505526351, "status": "success", "tasks": [ { "id": "1735505526355", "message": "clean-vm", "start": 1735505526355, "status": "success", "end": 1735505526360, "result": { "merge": false } }, { "id": "1735505526361", "message": "clean-vm", "start": 1735505526361, "status": "success", "end": 1735505526366, "result": { "merge": false } }, { "id": "1735505535390", "message": "snapshot", "start": 1735505535390, "status": "success", "end": 1735505536723, "result": "f1af0fb7-5eb5-004d-64e2-37bef4db95c2" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735505536724", "message": "export", "start": 1735505536724, "status": "success", "tasks": [ { "id": "1735505537852", "message": "transfer", "start": 1735505537852, "status": "success", "end": 1735505655199, "result": { "size": 5902978560 } }, { "id": "1735505657049", "message": "clean-vm", "start": 1735505657049, "status": "success", "end": 1735505657082, "result": { "merge": false } } ], "end": 1735505657086 }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735505536724:0", "message": "export", "start": 1735505536724, "status": "success", "tasks": [ { "id": "1735505537850", "message": "transfer", "start": 1735505537850, "status": "success", "end": 1735505655897, "result": { "size": 5902978560 } }, { "id": "1735505657087", "message": "clean-vm", "start": 1735505657087, "status": "success", "end": 1735505657107, "result": { "merge": false } } ], "end": 1735505657109 } ], "end": 1735505657109 } ], "end": 1735505657109 }Then it was time for the scheduled job
And now it Failed{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1735534080010", "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867", "jobName": "LongTerm-5+2", "message": "backup", "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7", "start": 1735534080010, "status": "failure", "infos": [ { "data": { "vms": [ "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6", "5f029e45-a01c-310b-9895-a9e58bf92b96" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6", "name_label": "Debian" }, "id": "1735534083984", "message": "backup VM", "start": 1735534083984, "status": "failure", "tasks": [ { "id": "1735534084013", "message": "clean-vm", "start": 1735534084013, "status": "success", "tasks": [ { "id": "1735534084261", "message": "merge", "start": 1735534084261, "status": "success", "end": 1735534084505 } ], "end": 1735534084565, "result": { "merge": true } }, { "id": "1735534084018", "message": "clean-vm", "start": 1735534084018, "status": "success", "tasks": [ { "id": "1735534084190", "message": "merge", "start": 1735534084190, "status": "success", "end": 1735534084397 } ], "end": 1735534084447, "result": { "merge": true } }, { "id": "1735534085223", "message": "clean-vm", "start": 1735534085223, "status": "success", "end": 1735534085295, "result": { "merge": false } }, { "id": "1735534085298", "message": "clean-vm", "start": 1735534085298, "status": "success", "end": 1735534085343, "result": { "merge": false } }, { "id": "1735534128548", "message": "clean-vm", "start": 1735534128548, "status": "success", "end": 1735534128621, "result": { "merge": false } }, { "id": "1735534128559", "message": "clean-vm", "start": 1735534128559, "status": "success", "end": 1735534128634, "result": { "merge": false } }, { "id": "1735534129252", "message": "clean-vm", "start": 1735534129252, "status": "success", "end": 1735534129327, "result": { "merge": false } }, { "id": "1735534129332", "message": "clean-vm", "start": 1735534129332, "status": "success", "end": 1735534129387, "result": { "merge": false } }, { "id": "1735534129399", "message": "clean-vm", "start": 1735534129399, "status": "success", "end": 1735534129485, "result": { "merge": false } }, { "id": "1735534129432", "message": "clean-vm", "start": 1735534129432, "status": "success", "end": 1735534129549, "result": { "merge": false } }, { "id": "1735534130061", "message": "clean-vm", "start": 1735534130061, "status": "success", "end": 1735534130164, "result": { "merge": false } }, { "id": "1735534130166", "message": "clean-vm", "start": 1735534130166, "status": "success", "end": 1735534130206, "result": { "merge": false } } ], "warnings": [ { "data": { "attempt": 1, "error": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)" }, "message": "Retry the VM backup due to an error" }, { "data": { "attempt": 2, "error": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)" }, "message": "Retry the VM backup due to an error" } ], "end": 1735534130208, "result": { "code": "VM_LACKS_FEATURE", "params": [ "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956" ], "task": { "uuid": "b31c85f3-e58d-53bc-55ff-7bc122e7c5e5", "name_label": "Async.VM.clean_shutdown", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241230T04:48:49Z", "finished": "20241230T04:48:49Z", "status": "failure", "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "VM_LACKS_FEATURE", "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)", "name": "XapiError", "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)\n at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1047:24)\n at file:///root/xen-orchestra/packages/xen-api/index.mjs:1081:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1071:12)\n at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1244:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "type": "VM", "id": "5f029e45-a01c-310b-9895-a9e58bf92b96", "name_label": "PiHole 🗑️" }, "id": "1735534085350", "message": "backup VM", "start": 1735534085350, "status": "success", "tasks": [ { "id": "1735534085357", "message": "clean-vm", "start": 1735534085357, "status": "success", "end": 1735534085396, "result": { "merge": false } }, { "id": "1735534085367", "message": "clean-vm", "start": 1735534085367, "status": "success", "end": 1735534085431, "result": { "merge": false } }, { "id": "1735534090190", "message": "snapshot", "start": 1735534090190, "status": "success", "end": 1735534091533, "result": "16c70b87-c65c-4c5a-660d-30a67446d36f" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": false, "type": "remote" }, "id": "1735534091535", "message": "export", "start": 1735534091535, "status": "success", "tasks": [ { "id": "1735534093576", "message": "transfer", "start": 1735534093576, "status": "success", "end": 1735534126295, "result": { "size": 765799424 } }, { "id": "1735534128431", "message": "clean-vm", "start": 1735534128431, "status": "success", "end": 1735534128472, "result": { "merge": false } } ], "end": 1735534128476 }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": false, "type": "remote" }, "id": "1735534091535:0", "message": "export", "start": 1735534091535, "status": "success", "tasks": [ { "id": "1735534093570", "message": "transfer", "start": 1735534093570, "status": "success", "end": 1735534127145, "result": { "size": 765799424 } }, { "id": "1735534128476:0", "message": "clean-vm", "start": 1735534128476, "status": "success", "end": 1735534128519, "result": { "merge": false } } ], "end": 1735534128520 } ], "end": 1735534128521 } ], "end": 1735534130209 }I then removed the vm without the tools to test if that was the problem and ran the job manually
Failed again{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1735559371587", "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867", "jobName": "LongTerm-5+2", "message": "backup", "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7", "start": 1735559371587, "status": "failure", "infos": [ { "data": { "vms": [ "5f029e45-a01c-310b-9895-a9e58bf92b96" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "5f029e45-a01c-310b-9895-a9e58bf92b96", "name_label": "PiHole 🗑️" }, "id": "1735559373175", "message": "backup VM", "start": 1735559373175, "status": "failure", "tasks": [ { "id": "1735559373184", "message": "clean-vm", "start": 1735559373184, "status": "success", "end": 1735559373261, "result": { "merge": false } }, { "id": "1735559373193", "message": "clean-vm", "start": 1735559373193, "status": "success", "end": 1735559373276, "result": { "merge": false } }, { "id": "1735559377708", "message": "snapshot", "start": 1735559377708, "status": "success", "end": 1735559378775, "result": "c27f03ee-f606-6b8f-1d56-564011f7d819" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735559378776", "message": "export", "start": 1735559378776, "status": "failure", "end": 1735559378799, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735559378777", "message": "export", "start": 1735559378777, "status": "failure", "end": 1735559378793, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1735559379622", "message": "clean-vm", "start": 1735559379622, "status": "success", "end": 1735559379643, "result": { "merge": false } }, { "id": "1735559379630", "message": "clean-vm", "start": 1735559379630, "status": "success", "end": 1735559379660, "result": { "merge": false } }, { "id": "1735559379784", "message": "snapshot", "start": 1735559379784, "status": "success", "end": 1735559393280, "result": "f7929692-70fb-3a83-8c2a-b02d349abd9e" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735559393280:0", "message": "export", "start": 1735559393280, "status": "failure", "end": 1735559393289, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735559393280:1", "message": "export", "start": 1735559393280, "status": "failure", "end": 1735559393284, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1735559394101", "message": "clean-vm", "start": 1735559394101, "status": "success", "end": 1735559394121, "result": { "merge": false } }, { "id": "1735559394110", "message": "clean-vm", "start": 1735559394110, "status": "success", "end": 1735559394148, "result": { "merge": false } }, { "id": "1735559394495", "message": "clean-vm", "start": 1735559394495, "status": "success", "end": 1735559394538, "result": { "merge": false } }, { "id": "1735559394540", "message": "clean-vm", "start": 1735559394540, "status": "success", "end": 1735559394562, "result": { "merge": false } } ], "warnings": [ { "data": { "attempt": 1, "error": "all targets have failed, step: writer.prepare()" }, "message": "Retry the VM backup due to an error" }, { "data": { "attempt": 2, "error": "all targets have failed, step: writer.prepare()" }, "message": "Retry the VM backup due to an error" } ], "end": 1735559394564, "result": { "code": "VM_LACKS_FEATURE", "params": [ "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a" ], "task": { "uuid": "b11403fb-1455-886a-dc28-c962ac295f40", "name_label": "Async.VM.clean_shutdown", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241230T11:49:54Z", "finished": "20241230T11:49:54Z", "status": "failure", "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "VM_LACKS_FEATURE", "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)", "name": "XapiError", "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)\n at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1047:24)\n at file:///root/xen-orchestra/packages/xen-api/index.mjs:1081:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1071:12)\n at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1244:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1735559394565 }Happy reading on the new year
-
It failed when XO asked the VM with
OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956to do aAsync.VM.clean_shutdown. Because without tools, you cannot make a clean shutdown. I don't remember in which scenario you need that action during a backup, probably related to auto healthcheck. -
@olivierlambert
The last log is from the Debian VM only and with 1.0.0-proto-0.4.0 tools
The one without tools was removed from the jobThanks for Your engagement, It's great with people that don't sleep ;o)
-
I cannot tell which VM is
OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956but you can, try to see to which VM it refers
-
I cannot tell which VM is
OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956but you can, try to see to which VM it refersIt's the one without tools
But in the third log I only ran the vm with tools and still got errors:
"OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a"and"AssertionError"I got no clue
I installed vm1 (the "unistalled" one), updated and installed the 1.0.0-proto-0.4.0 tools
made a new simple backup job with normal snapshot and it ran successfully
I added vm2 and the job ran successfully again.Changed the snapshot mode to offline
SuccessfulII added the same schedule as in the first job
retention 7
Number of daily backups kept"blank"
Number of weekly backups kept2
Number of monthly backups kept1
Number of yearly backups kept1
And it failed.Changed
Number of daily backups kept"blank" to 7
And Viola Success.
It must have been deleted by mistake.
Seems like it doesn't like theNumber of daily backups keptto be blankI will change this in the first job and hopefully this is solved.
And it's not
I changedNumber of daily backups kept"blank" to 7
Ran the job, Failure.
Rebooted everything included the host
Ran the job. Failure again.{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1735582708642", "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867", "jobName": "LongTerm-5+2", "message": "backup", "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7", "start": 1735582708642, "status": "failure", "infos": [ { "data": { "vms": [ "5f029e45-a01c-310b-9895-a9e58bf92b96", "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "5f029e45-a01c-310b-9895-a9e58bf92b96", "name_label": "PiHole 🗑️" }, "id": "1735582710485", "message": "backup VM", "start": 1735582710485, "status": "failure", "tasks": [ { "id": "1735582710508", "message": "clean-vm", "start": 1735582710508, "status": "success", "end": 1735582710559, "result": { "merge": false } }, { "id": "1735582710511", "message": "clean-vm", "start": 1735582710511, "status": "success", "end": 1735582710584, "result": { "merge": false } }, { "id": "1735582714860", "message": "snapshot", "start": 1735582714860, "status": "success", "end": 1735582716135, "result": "37392150-1519-8b53-dd16-bcbfb64a1972" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735582716136", "message": "export", "start": 1735582716136, "status": "failure", "end": 1735582716159, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735582716137", "message": "export", "start": 1735582716137, "status": "failure", "end": 1735582716152, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1735582732349", "message": "clean-vm", "start": 1735582732349, "status": "success", "end": 1735582732376, "result": { "merge": false } }, { "id": "1735582732364", "message": "clean-vm", "start": 1735582732364, "status": "success", "end": 1735582732426, "result": { "merge": false } }, { "id": "1735582732943", "message": "clean-vm", "start": 1735582732943, "status": "success", "end": 1735582732990, "result": { "merge": false } }, { "id": "1735582732992", "message": "clean-vm", "start": 1735582732992, "status": "success", "end": 1735582733015, "result": { "merge": false } }, { "id": "1735582749464", "message": "clean-vm", "start": 1735582749464, "status": "success", "end": 1735582749500, "result": { "merge": false } }, { "id": "1735582749480", "message": "clean-vm", "start": 1735582749480, "status": "success", "end": 1735582749549, "result": { "merge": false } }, { "id": "1735582749851", "message": "clean-vm", "start": 1735582749851, "status": "success", "end": 1735582749924, "result": { "merge": false } }, { "id": "1735582749926", "message": "clean-vm", "start": 1735582749926, "status": "success", "end": 1735582749980, "result": { "merge": false } } ], "warnings": [ { "data": { "attempt": 1, "error": "all targets have failed, step: writer.prepare()" }, "message": "Retry the VM backup due to an error" }, { "data": { "attempt": 2, "error": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)" }, "message": "Retry the VM backup due to an error" } ], "end": 1735582749981, "result": { "code": "VM_LACKS_FEATURE", "params": [ "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a" ], "task": { "uuid": "58309b21-e833-44c8-7b90-dcfee090139a", "name_label": "Async.VM.clean_shutdown", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241230T18:19:09Z", "finished": "20241230T18:19:09Z", "status": "failure", "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "VM_LACKS_FEATURE", "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)", "name": "XapiError", "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)\n at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1068:24)\n at file:///root/xen-orchestra/packages/xen-api/index.mjs:1102:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1092:12)\n at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1265:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "type": "VM", "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6", "name_label": "Debian" }, "id": "1735582717167", "message": "backup VM", "start": 1735582717167, "status": "failure", "tasks": [ { "id": "1735582717172", "message": "clean-vm", "start": 1735582717172, "status": "success", "end": 1735582717244, "result": { "merge": false } }, { "id": "1735582717180", "message": "clean-vm", "start": 1735582717180, "status": "success", "end": 1735582717294, "result": { "merge": false } }, { "id": "1735582730015", "message": "snapshot", "start": 1735582730015, "status": "success", "end": 1735582731315, "result": "160221cd-fff0-d481-b05f-64c27735f740" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735582731316", "message": "export", "start": 1735582731316, "status": "failure", "end": 1735582731340, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735582731317", "message": "export", "start": 1735582731317, "status": "failure", "end": 1735582731333, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1735582733030", "message": "clean-vm", "start": 1735582733030, "status": "success", "end": 1735582733121, "result": { "merge": false } }, { "id": "1735582733038", "message": "clean-vm", "start": 1735582733038, "status": "success", "end": 1735582733207, "result": { "merge": false } }, { "id": "1735582733338", "message": "snapshot", "start": 1735582733338, "status": "success", "end": 1735582748462, "result": "f11f0d4c-0804-2bbe-960d-75c5c11305f1" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735582748462:0", "message": "export", "start": 1735582748462, "status": "failure", "end": 1735582748486, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735582748462:1", "message": "export", "start": 1735582748462, "status": "failure", "end": 1735582748469, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1735582749990", "message": "clean-vm", "start": 1735582749990, "status": "success", "end": 1735582750034, "result": { "merge": false } }, { "id": "1735582750019", "message": "clean-vm", "start": 1735582750019, "status": "success", "end": 1735582750124, "result": { "merge": false } }, { "id": "1735582750491", "message": "clean-vm", "start": 1735582750491, "status": "success", "end": 1735582750554, "result": { "merge": false } }, { "id": "1735582750581", "message": "clean-vm", "start": 1735582750581, "status": "success", "end": 1735582750631, "result": { "merge": false } } ], "warnings": [ { "data": { "attempt": 1, "error": "all targets have failed, step: writer.prepare()" }, "message": "Retry the VM backup due to an error" }, { "data": { "attempt": 2, "error": "all targets have failed, step: writer.prepare()" }, "message": "Retry the VM backup due to an error" } ], "end": 1735582750633, "result": { "code": "VM_LACKS_FEATURE", "params": [ "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956" ], "task": { "uuid": "fe77d196-a6f0-12ca-5795-3669fcf16201", "name_label": "Async.VM.clean_shutdown", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241230T18:19:10Z", "finished": "20241230T18:19:10Z", "status": "failure", "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "VM_LACKS_FEATURE", "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)", "name": "XapiError", "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)\n at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1068:24)\n at file:///root/xen-orchestra/packages/xen-api/index.mjs:1102:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1092:12)\n at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1265:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1735582750633 }I don't know, I will do some more testing.
And thanks again Oliver 🥳 -
I compared the 2 jobs to make sure they where exactly the same.
Ran the new job. Success.
Ran the old job. Failed.Removed all data from Long-term retention of backups in the old job
Number of daily backups kept7 to "blank"
Number of weekly backups kept2 to "blank"
Number of monthly backups kept1 to "blank"
Number of yearly backups kept1 to "blank"Ran the old job again. Success !
Put the data back to 7 2 1 1
Ran the old job. Failed.Hmmm.. I think there must be an undocumented feature here
I will run the new job alongside the old job an hour apart to see what will happen. -
I'm getting similar too, I get the
"entries must be sorted in asc order 2025-01 2025-52"error (see below) hope that's helpful.Based on the comment about removing the
Long-term retention of backupsdata, I just removed theNumber of weekly backups keptentry and re-ran, this allowed the backup to run. I kept myNumber of daily backups keptlimit which I haven't reached yet."message": "backup VM", "start": 1735640611472, "status": "failure", "tasks": [ { "id": "1735640611748", "message": "snapshot", "start": 1735640611748, "status": "success", "end": 1735640616464, "result": "1f91db46-fbfe-ba09-9f8b-1c350d7493fd" }, { "data": { "id": "59e1ac20-f03d-45b4-953d-6fb7277537c2", "type": "remote", "isFull": true }, "id": "1735640616501", "message": "export", "start": 1735640616501, "status": "failure", "end": 1735640616505, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///home/root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at FullRemoteWriter._run (file:///home/root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/FullRemoteWriter.mjs:39:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async FullRemoteWriter.run (file:///home/root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_AbstractFullWriter.mjs:6:14)" } -
Thank you for reporting this fix, after I removed the
Number of weekly backups keptentry my backups are working again. Last night all our backups failed with theentries must be sorted in asc order 2025-01 2025-52error. -
I have now, on the new year, added a new schedule to see if there will be any difference
-
@David Thanks, my backups suddenly fail since today, after removing all retention settings except “daily backup”, they work perfectly as last year.
-
@olivierlambert said in Long-term retention of backups:
@ph7 said in Long-term retention of backups:
VM_LACKS_FEATURE
This means there's no tools installed or working on that VM
The original job is still not running
I was thinking about this and came up with this cunning plan- Shut down the VM to see if there is still a problem with the tools
- Run the original job manually
No. it did not work, still failing
entries must be sorted in asc order 2025-01 2025-52{ "data": { "mode": "delta", "reportWhen": "failure", "backupReportTpl": "mjml" }, "id": "1735908223694", "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867", "jobName": "LongTerm-5+2", "message": "backup", "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7", "start": 1735908223694, "status": "failure", "infos": [ { "data": { "vms": [ "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6", "name_label": "Debian" }, "id": "1735908225556", "message": "backup VM", "start": 1735908225556, "status": "failure", "tasks": [ { "id": "1735908225574", "message": "clean-vm", "start": 1735908225574, "status": "success", "end": 1735908225783, "result": { "merge": false } }, { "id": "1735908225605", "message": "clean-vm", "start": 1735908225605, "status": "success", "end": 1735908225828, "result": { "merge": false } }, { "id": "1735908226496", "message": "snapshot", "start": 1735908226496, "status": "success", "end": 1735908231603, "result": "4cf9de45-ff3f-f1a1-8a86-7b5cb03c9dd5" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735908231603:0", "message": "export", "start": 1735908231603, "status": "failure", "end": 1735908231670, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735908231605", "message": "export", "start": 1735908231605, "status": "failure", "end": 1735908231657, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1735908232625", "message": "clean-vm", "start": 1735908232625, "status": "success", "end": 1735908232700, "result": { "merge": false } }, { "id": "1735908232656", "message": "clean-vm", "start": 1735908232656, "status": "success", "end": 1735908232767, "result": { "merge": false } }, { "id": "1735908232981", "message": "snapshot", "start": 1735908232981, "status": "success", "end": 1735908234120, "result": "f614fec5-87a5-c227-4e0c-8fa76acbb271" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735908234121", "message": "export", "start": 1735908234121, "status": "failure", "end": 1735908234135, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735908234121:0", "message": "export", "start": 1735908234121, "status": "failure", "end": 1735908234129, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "id": "1735908234974", "message": "clean-vm", "start": 1735908234974, "status": "success", "end": 1735908235063, "result": { "merge": false } }, { "id": "1735908234994", "message": "clean-vm", "start": 1735908234994, "status": "success", "end": 1735908235091, "result": { "merge": false } }, { "id": "1735908235345", "message": "snapshot", "start": 1735908235345, "status": "success", "end": 1735908236519, "result": "319c12f9-6f35-411d-2a99-b2c55f7d469b" }, { "data": { "id": "446f82eb-2794-4c01-bd71-e247abd6a39c", "isFull": true, "type": "remote" }, "id": "1735908236520", "message": "export", "start": 1735908236520, "status": "failure", "end": 1735908236533, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } }, { "data": { "id": "0927f4dc-4476-4ac3-959a-bc37e85956df", "isFull": true, "type": "remote" }, "id": "1735908236520:0", "message": "export", "start": 1735908236520, "status": "failure", "end": 1735908236527, "result": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "warnings": [ { "data": { "attempt": 1, "error": "all targets have failed, step: writer.prepare()" }, "message": "Retry the VM backup due to an error" }, { "data": { "attempt": 2, "error": "all targets have failed, step: writer.prepare()" }, "message": "Retry the VM backup due to an error" } ], "end": 1735908237352, "result": { "errors": [ { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" }, { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": false, "expected": true, "operator": "strictEqual", "message": "entries must be sorted in asc order 2025-01 2025-52", "name": "AssertionError", "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } ], "message": "all targets have failed, step: writer.prepare()", "name": "Error", "stack": "Error: all targets have failed, step: writer.prepare()\n at IncrementalXapiVmBackupRunner._callWriters (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:64:13)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async IncrementalXapiVmBackupRunner._copy (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:42:5)\n at async IncrementalXapiVmBackupRunner.run (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:396:9)\n at async file:///root/xen-orchestra/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38" } } ], "end": 1735908237354 }Maybe I'm wrong but I think we have a problem here. Could have something to do with the new year.
-
@ph7 This is being investigated by the XOA team. We will update this thread when more information is available.
