XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Long-term retention of backups

    Scheduled Pinned Locked Moved Solved Backup
    23 Posts 7 Posters 4.4k Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      ph7 @olivierlambert
      last edited by ph7

      @olivierlambert
      The last log is from the Debian VM only and with 1.0.0-proto-0.4.0 tools
      The one without tools was removed from the job

      Thanks for Your engagement, It's great with people that don't sleep ;o)

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        I cannot tell which VM is OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956 but you can, try to see to which VM it refers 🙂

        P 1 Reply Last reply Reply Quote 0
        • P Offline
          ph7 @olivierlambert
          last edited by ph7

          @olivierlambert

          I cannot tell which VM is OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956 but you can, try to see to which VM it refers 🙂

          It's the one without tools

          But in the third log I only ran the vm with tools and still got errors:
          "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a" and "AssertionError"

          I got no clue

          I installed vm1 (the "unistalled" one), updated and installed the 1.0.0-proto-0.4.0 tools
          made a new simple backup job with normal snapshot and it ran successfully
          I added vm2 and the job ran successfully again.

          Changed the snapshot mode to offline
          SuccessfulI

          I added the same schedule as in the first job
          retention 7
          Number of daily backups kept "blank"
          Number of weekly backups kept 2
          Number of monthly backups kept 1
          Number of yearly backups kept 1
          And it failed.

          Changed Number of daily backups kept "blank" to 7
          And Viola Success.
          It must have been deleted by mistake.
          Seems like it doesn't like the Number of daily backups kept to be blank

          I will change this in the first job and hopefully this is solved.

          And it's not
          I changed Number of daily backups kept "blank" to 7
          Ran the job, Failure.
          Rebooted everything included the host
          Ran the job. Failure again.

          {
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1735582708642",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735582708642,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "5f029e45-a01c-310b-9895-a9e58bf92b96",
          "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "5f029e45-a01c-310b-9895-a9e58bf92b96",
        "name_label": "PiHole 🗑️"
      },
      "id": "1735582710485",
      "message": "backup VM",
      "start": 1735582710485,
      "status": "failure",
      "tasks": [
        {
          "id": "1735582710508",
          "message": "clean-vm",
          "start": 1735582710508,
          "status": "success",
          "end": 1735582710559,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582710511",
          "message": "clean-vm",
          "start": 1735582710511,
          "status": "success",
          "end": 1735582710584,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582714860",
          "message": "snapshot",
          "start": 1735582714860,
          "status": "success",
          "end": 1735582716135,
          "result": "37392150-1519-8b53-dd16-bcbfb64a1972"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582716136",
          "message": "export",
          "start": 1735582716136,
          "status": "failure",
          "end": 1735582716159,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582716137",
          "message": "export",
          "start": 1735582716137,
          "status": "failure",
          "end": 1735582716152,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735582732349",
          "message": "clean-vm",
          "start": 1735582732349,
          "status": "success",
          "end": 1735582732376,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582732364",
          "message": "clean-vm",
          "start": 1735582732364,
          "status": "success",
          "end": 1735582732426,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582732943",
          "message": "clean-vm",
          "start": 1735582732943,
          "status": "success",
          "end": 1735582732990,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582732992",
          "message": "clean-vm",
          "start": 1735582732992,
          "status": "success",
          "end": 1735582733015,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749464",
          "message": "clean-vm",
          "start": 1735582749464,
          "status": "success",
          "end": 1735582749500,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749480",
          "message": "clean-vm",
          "start": 1735582749480,
          "status": "success",
          "end": 1735582749549,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749851",
          "message": "clean-vm",
          "start": 1735582749851,
          "status": "success",
          "end": 1735582749924,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749926",
          "message": "clean-vm",
          "start": 1735582749926,
          "status": "success",
          "end": 1735582749980,
          "result": {
            "merge": false
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735582749981,
      "result": {
        "code": "VM_LACKS_FEATURE",
        "params": [
          "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a"
        ],
        "task": {
          "uuid": "58309b21-e833-44c8-7b90-dcfee090139a",
          "name_label": "Async.VM.clean_shutdown",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20241230T18:19:09Z",
          "finished": "20241230T18:19:09Z",
          "status": "failure",
          "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "VM_LACKS_FEATURE",
            "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
        },
        "message": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)",
        "name": "XapiError",
        "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)\n    at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1068:24)\n    at file:///root/xen-orchestra/packages/xen-api/index.mjs:1102:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1092:12)\n    at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1265:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
      }
    },
    {
      "data": {
        "type": "VM",
        "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
        "name_label": "Debian"
      },
      "id": "1735582717167",
      "message": "backup VM",
      "start": 1735582717167,
      "status": "failure",
      "tasks": [
        {
          "id": "1735582717172",
          "message": "clean-vm",
          "start": 1735582717172,
          "status": "success",
          "end": 1735582717244,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582717180",
          "message": "clean-vm",
          "start": 1735582717180,
          "status": "success",
          "end": 1735582717294,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582730015",
          "message": "snapshot",
          "start": 1735582730015,
          "status": "success",
          "end": 1735582731315,
          "result": "160221cd-fff0-d481-b05f-64c27735f740"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582731316",
          "message": "export",
          "start": 1735582731316,
          "status": "failure",
          "end": 1735582731340,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582731317",
          "message": "export",
          "start": 1735582731317,
          "status": "failure",
          "end": 1735582731333,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735582733030",
          "message": "clean-vm",
          "start": 1735582733030,
          "status": "success",
          "end": 1735582733121,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582733038",
          "message": "clean-vm",
          "start": 1735582733038,
          "status": "success",
          "end": 1735582733207,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582733338",
          "message": "snapshot",
          "start": 1735582733338,
          "status": "success",
          "end": 1735582748462,
          "result": "f11f0d4c-0804-2bbe-960d-75c5c11305f1"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582748462:0",
          "message": "export",
          "start": 1735582748462,
          "status": "failure",
          "end": 1735582748486,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582748462:1",
          "message": "export",
          "start": 1735582748462,
          "status": "failure",
          "end": 1735582748469,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735582749990",
          "message": "clean-vm",
          "start": 1735582749990,
          "status": "success",
          "end": 1735582750034,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582750019",
          "message": "clean-vm",
          "start": 1735582750019,
          "status": "success",
          "end": 1735582750124,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582750491",
          "message": "clean-vm",
          "start": 1735582750491,
          "status": "success",
          "end": 1735582750554,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582750581",
          "message": "clean-vm",
          "start": 1735582750581,
          "status": "success",
          "end": 1735582750631,
          "result": {
            "merge": false
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735582750633,
      "result": {
        "code": "VM_LACKS_FEATURE",
        "params": [
          "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956"
        ],
        "task": {
          "uuid": "fe77d196-a6f0-12ca-5795-3669fcf16201",
          "name_label": "Async.VM.clean_shutdown",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20241230T18:19:10Z",
          "finished": "20241230T18:19:10Z",
          "status": "failure",
          "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "VM_LACKS_FEATURE",
            "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
        },
        "message": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)",
        "name": "XapiError",
        "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)\n    at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1068:24)\n    at file:///root/xen-orchestra/packages/xen-api/index.mjs:1102:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1092:12)\n    at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1265:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
      }
    }
  ],
  "end": 1735582750633
}

          I don't know, I will do some more testing.
          And thanks again Oliver 🥳

          P 1 Reply Last reply Reply Quote 0
          • P Offline
            ph7 @ph7
            last edited by ph7

            I compared the 2 jobs to make sure they where exactly the same.
            Ran the new job. Success.
            Ran the old job. Failed.

            Removed all data from Long-term retention of backups in the old job
            Number of daily backups kept 7 to "blank"
            Number of weekly backups kept 2 to "blank"
            Number of monthly backups kept 1 to "blank"
            Number of yearly backups kept 1 to "blank"

            Ran the old job again. Success !

            Put the data back to 7 2 1 1
            Ran the old job. Failed.

            Hmmm.. I think there must be an undocumented feature here 🙂
            I will run the new job alongside the old job an hour apart to see what will happen.

            1 Reply Last reply Reply Quote 0
            • D Offline
              David
              last edited by

              I'm getting similar too, I get the "entries must be sorted in asc order 2025-01 2025-52" error (see below) hope that's helpful.

              Based on the comment about removing the Long-term retention of backups data, I just removed the Number of weekly backups kept entry and re-ran, this allowed the backup to run. I kept my Number of daily backups kept limit which I haven't reached yet.

                    "message": "backup VM",
      "start": 1735640611472,
      "status": "failure",
      "tasks": [
        {
          "id": "1735640611748",
          "message": "snapshot",
          "start": 1735640611748,
          "status": "success",
          "end": 1735640616464,
          "result": "1f91db46-fbfe-ba09-9f8b-1c350d7493fd"
        },
        {
          "data": {
            "id": "59e1ac20-f03d-45b4-953d-6fb7277537c2",
            "type": "remote",
            "isFull": true
          },
          "id": "1735640616501",
          "message": "export",
          "start": 1735640616501,
          "status": "failure",
          "end": 1735640616505,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///home/root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at FullRemoteWriter._run (file:///home/root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/FullRemoteWriter.mjs:39:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async FullRemoteWriter.run (file:///home/root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_AbstractFullWriter.mjs:6:14)"
          }
              R C 2 Replies Last reply Reply Quote 0
              • R Offline
                rmaclachlan @David
                last edited by

                Thank you for reporting this fix, after I removed the Number of weekly backups kept entry my backups are working again. Last night all our backups failed with the entries must be sorted in asc order 2025-01 2025-52 error.

                1 Reply Last reply Reply Quote 0
                • P Offline
                  ph7
                  last edited by

                  I have now, on the new year, added a new schedule to see if there will be any difference

                  1 Reply Last reply Reply Quote 0
                  • C Offline
                    chr1st0ph9 @David
                    last edited by

                    @David Thanks, my backups suddenly fail since today, after removing all retention settings except “daily backup”, they work perfectly as last year.

                    1 Reply Last reply Reply Quote 0
                    • P Offline
                      ph7 @olivierlambert
                      last edited by

                      @olivierlambert said in Long-term retention of backups:

                      @ph7 said in Long-term retention of backups:

                      VM_LACKS_FEATURE

                      This means there's no tools installed or working on that VM

                      The original job is still not running
                      I was thinking about this and came up with this cunning plan 🙂

                      1. Shut down the VM to see if there is still a problem with the tools
                      2. Run the original job manually

                      No. it did not work, still failing entries must be sorted in asc order 2025-01 2025-52

                      {
  "data": {
    "mode": "delta",
    "reportWhen": "failure",
    "backupReportTpl": "mjml"
  },
  "id": "1735908223694",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735908223694,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
        "name_label": "Debian"
      },
      "id": "1735908225556",
      "message": "backup VM",
      "start": 1735908225556,
      "status": "failure",
      "tasks": [
        {
          "id": "1735908225574",
          "message": "clean-vm",
          "start": 1735908225574,
          "status": "success",
          "end": 1735908225783,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908225605",
          "message": "clean-vm",
          "start": 1735908225605,
          "status": "success",
          "end": 1735908225828,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908226496",
          "message": "snapshot",
          "start": 1735908226496,
          "status": "success",
          "end": 1735908231603,
          "result": "4cf9de45-ff3f-f1a1-8a86-7b5cb03c9dd5"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908231603:0",
          "message": "export",
          "start": 1735908231603,
          "status": "failure",
          "end": 1735908231670,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908231605",
          "message": "export",
          "start": 1735908231605,
          "status": "failure",
          "end": 1735908231657,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735908232625",
          "message": "clean-vm",
          "start": 1735908232625,
          "status": "success",
          "end": 1735908232700,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908232656",
          "message": "clean-vm",
          "start": 1735908232656,
          "status": "success",
          "end": 1735908232767,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908232981",
          "message": "snapshot",
          "start": 1735908232981,
          "status": "success",
          "end": 1735908234120,
          "result": "f614fec5-87a5-c227-4e0c-8fa76acbb271"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908234121",
          "message": "export",
          "start": 1735908234121,
          "status": "failure",
          "end": 1735908234135,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908234121:0",
          "message": "export",
          "start": 1735908234121,
          "status": "failure",
          "end": 1735908234129,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735908234974",
          "message": "clean-vm",
          "start": 1735908234974,
          "status": "success",
          "end": 1735908235063,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908234994",
          "message": "clean-vm",
          "start": 1735908234994,
          "status": "success",
          "end": 1735908235091,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908235345",
          "message": "snapshot",
          "start": 1735908235345,
          "status": "success",
          "end": 1735908236519,
          "result": "319c12f9-6f35-411d-2a99-b2c55f7d469b"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908236520",
          "message": "export",
          "start": 1735908236520,
          "status": "failure",
          "end": 1735908236533,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908236520:0",
          "message": "export",
          "start": 1735908236520,
          "status": "failure",
          "end": 1735908236527,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735908237352,
      "result": {
        "errors": [
          {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          },
          {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        ],
        "message": "all targets have failed, step: writer.prepare()",
        "name": "Error",
        "stack": "Error: all targets have failed, step: writer.prepare()\n    at IncrementalXapiVmBackupRunner._callWriters (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:64:13)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async IncrementalXapiVmBackupRunner._copy (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:42:5)\n    at async IncrementalXapiVmBackupRunner.run (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:396:9)\n    at async file:///root/xen-orchestra/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }
    }
  ],
  "end": 1735908237354
}

                      Maybe I'm wrong but I think we have a problem here. Could have something to do with the new year.

                      DanpD 1 Reply Last reply Reply Quote 0
                      • DanpD Offline
                        Danp Pro Support Team @ph7
                        last edited by

                        @ph7 This is being investigated by the XOA team. We will update this thread when more information is available.

                        A 1 Reply Last reply Reply Quote 1
                        • D DustinB referenced this topic on
                        • A Offline
                          al-indigo @Danp
                          last edited by

                          @Danp Any news?
                          I have no delta-backups since new year and I don't know what will happen if I set retention "Backup retention" to 0.
                          I don't see any options to disable retention temporarily so I disabled daily backups. We have full monthly backups, but daily and weekly stopped working and it's disturbing.
                          We use open-source edition, no offence or pressure intended; just curious since maybe it's reasonable to delete all backups after next monthly full-backup is complete

                          DanpD 1 Reply Last reply Reply Quote 0
                          • DanpD Offline
                            Danp Pro Support Team @al-indigo
                            last edited by

                            @al-indigo I believe that a fix should be out by EOM.

                            A 2 Replies Last reply Reply Quote 0
                            • A Offline
                              al-indigo @Danp
                              last edited by

                              @Danp Could you please tell what will happen if I set "0" in retention field and enable backups again? Will it erase all my delta-backups there?
                              54226b89-ae07-4056-925d-797d4ff50b0a-image.png

                              DanpD 1 Reply Last reply Reply Quote 0
                              • DanpD Offline
                                Danp Pro Support Team @al-indigo
                                last edited by

                                @al-indigo AFAIK, the backup retention settings from the schedule won't be affected by changing the LTR values to zeros.

                                1 Reply Last reply Reply Quote 0
                                • A Offline
                                  al-indigo @Danp
                                  last edited by

                                  @Danp Do you have any updates on fix release estimation?

                                  DanpD 1 Reply Last reply Reply Quote 0
                                  • DanpD Offline
                                    Danp Pro Support Team @al-indigo
                                    last edited by

                                    @al-indigo The fix is included in XOA 5.103.0, which was released last week.

                                    https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md

                                    [Backup/LTR] Fix computation for the last week of the year (PR #8269)

                                    A 1 Reply Last reply Reply Quote 1
                                    • A Offline
                                      al-indigo @Danp
                                      last edited by

                                      @Danp Thank you very much! Missed it

                                      1 Reply Last reply Reply Quote 1
                                      • olivierlambertO olivierlambert marked this topic as a question on
                                      • olivierlambertO olivierlambert has marked this topic as solved on

                                      Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                      Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                      With your input, this post could be even better 💗

                                      Register Login
                                      • First post
                                        Last post