XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Long-term retention of backups

    Scheduled Pinned Locked Moved Solved Backup
    23 Posts 7 Posters 902 Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      ph7 @ph7
      last edited by

      @Danp @olivierlambert

      On 25 dec, I "installed" a debian vm, but without installing anything, only the iso attached.
      I ran 7+ manual backups to see if the 7 retension was working, it did.
      I then let the job run on schedule from 26 dec 06:00 CET
      The result was Successful.

      {
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1735448400010",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735448400010,
  "status": "success",
  "infos": [
    {
      "data": {
        "vms": [
          "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
        "name_label": "Debian Bookworm 12"
      },
      "id": "1735448401748",
      "message": "backup VM",
      "start": 1735448401748,
      "status": "success",
      "tasks": [
        {
          "id": "1735448401758",
          "message": "clean-vm",
          "start": 1735448401758,
          "status": "success",
          "tasks": [
            {
              "id": "1735448401954",
              "message": "merge",
              "start": 1735448401954,
              "status": "success",
              "end": 1735448402090
            }
          ],
          "end": 1735448402140,
          "result": {
            "merge": true
          }
        },
        {
          "id": "1735448401828",
          "message": "clean-vm",
          "start": 1735448401828,
          "status": "success",
          "tasks": [
            {
              "id": "1735448401999",
              "message": "merge",
              "start": 1735448401999,
              "status": "success",
              "end": 1735448402201
            }
          ],
          "end": 1735448402262,
          "result": {
            "merge": true
          }
        },
        {
          "id": "1735448402589",
          "message": "snapshot",
          "start": 1735448402589,
          "status": "success",
          "end": 1735448403661,
          "result": "868bb4b2-1b4a-1ba3-9f04-00e86bb6098b"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": false,
            "type": "remote"
          },
          "id": "1735448403662",
          "message": "export",
          "start": 1735448403662,
          "status": "success",
          "tasks": [
            {
              "id": "1735448405593",
              "message": "transfer",
              "start": 1735448405593,
              "status": "success",
              "end": 1735448407141,
              "result": {
                "size": 145920
              }
            },
            {
              "id": "1735448407995",
              "message": "clean-vm",
              "start": 1735448407995,
              "status": "success",
              "end": 1735448408048,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1735448408063
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": false,
            "type": "remote"
          },
          "id": "1735448403663",
          "message": "export",
          "start": 1735448403663,
          "status": "success",
          "tasks": [
            {
              "id": "1735448405591",
              "message": "transfer",
              "start": 1735448405591,
              "status": "success",
              "end": 1735448407139,
              "result": {
                "size": 145920
              }
            },
            {
              "id": "1735448408063:0",
              "message": "clean-vm",
              "start": 1735448408063,
              "status": "success",
              "end": 1735448408110,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1735448408119
        }
      ],
      "end": 1735448408119
    }
  ],
  "end": 1735448408120
}

      Then I added a proper Debian VM with 1.0.0-proto-0.4.0 tools.
      I ran the job manually, Again Successful

      {
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1735505513119",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735505513119,
  "status": "success",
  "infos": [
    {
      "data": {
        "vms": [
          "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
          "5f029e45-a01c-310b-9895-a9e58bf92b96"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
        "name_label": "Debian"
      },
      "id": "1735505515013",
      "message": "backup VM",
      "start": 1735505515013,
      "status": "success",
      "tasks": [
        {
          "id": "1735505515063",
          "message": "clean-vm",
          "start": 1735505515063,
          "status": "success",
          "tasks": [
            {
              "id": "1735505515145",
              "message": "merge",
              "start": 1735505515145,
              "status": "success",
              "end": 1735505515281
            }
          ],
          "end": 1735505515317,
          "result": {
            "merge": true
          }
        },
        {
          "id": "1735505515064",
          "message": "clean-vm",
          "start": 1735505515064,
          "status": "success",
          "tasks": [
            {
              "id": "1735505515190",
              "message": "merge",
              "start": 1735505515190,
              "status": "success",
              "end": 1735505515397
            }
          ],
          "end": 1735505515459,
          "result": {
            "merge": true
          }
        },
        {
          "id": "1735505520519",
          "message": "snapshot",
          "start": 1735505520519,
          "status": "success",
          "end": 1735505521796,
          "result": "321f6c53-acca-8cac-1373-9537e3ce4836"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": false,
            "type": "remote"
          },
          "id": "1735505521798",
          "message": "export",
          "start": 1735505521798,
          "status": "success",
          "tasks": [
            {
              "id": "1735505523731",
              "message": "transfer",
              "start": 1735505523731,
              "status": "success",
              "end": 1735505525358,
              "result": {
                "size": 145920
              }
            },
            {
              "id": "1735505526203",
              "message": "clean-vm",
              "start": 1735505526203,
              "status": "success",
              "end": 1735505526288,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1735505526305
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": false,
            "type": "remote"
          },
          "id": "1735505521799",
          "message": "export",
          "start": 1735505521799,
          "status": "success",
          "tasks": [
            {
              "id": "1735505523729",
              "message": "transfer",
              "start": 1735505523729,
              "status": "success",
              "end": 1735505525356,
              "result": {
                "size": 145920
              }
            },
            {
              "id": "1735505526305:0",
              "message": "clean-vm",
              "start": 1735505526305,
              "status": "success",
              "end": 1735505526338,
              "result": {
                "merge": true
              }
            }
          ],
          "end": 1735505526344
        }
      ],
      "end": 1735505526345
    },
    {
      "data": {
        "type": "VM",
        "id": "5f029e45-a01c-310b-9895-a9e58bf92b96",
        "name_label": "PiHole 🗑️"
      },
      "id": "1735505526351",
      "message": "backup VM",
      "start": 1735505526351,
      "status": "success",
      "tasks": [
        {
          "id": "1735505526355",
          "message": "clean-vm",
          "start": 1735505526355,
          "status": "success",
          "end": 1735505526360,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735505526361",
          "message": "clean-vm",
          "start": 1735505526361,
          "status": "success",
          "end": 1735505526366,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735505535390",
          "message": "snapshot",
          "start": 1735505535390,
          "status": "success",
          "end": 1735505536723,
          "result": "f1af0fb7-5eb5-004d-64e2-37bef4db95c2"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735505536724",
          "message": "export",
          "start": 1735505536724,
          "status": "success",
          "tasks": [
            {
              "id": "1735505537852",
              "message": "transfer",
              "start": 1735505537852,
              "status": "success",
              "end": 1735505655199,
              "result": {
                "size": 5902978560
              }
            },
            {
              "id": "1735505657049",
              "message": "clean-vm",
              "start": 1735505657049,
              "status": "success",
              "end": 1735505657082,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1735505657086
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735505536724:0",
          "message": "export",
          "start": 1735505536724,
          "status": "success",
          "tasks": [
            {
              "id": "1735505537850",
              "message": "transfer",
              "start": 1735505537850,
              "status": "success",
              "end": 1735505655897,
              "result": {
                "size": 5902978560
              }
            },
            {
              "id": "1735505657087",
              "message": "clean-vm",
              "start": 1735505657087,
              "status": "success",
              "end": 1735505657107,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1735505657109
        }
      ],
      "end": 1735505657109
    }
  ],
  "end": 1735505657109
}

      Then it was time for the scheduled job
      And now it Failed

      {
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1735534080010",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735534080010,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
          "5f029e45-a01c-310b-9895-a9e58bf92b96"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
        "name_label": "Debian"
      },
      "id": "1735534083984",
      "message": "backup VM",
      "start": 1735534083984,
      "status": "failure",
      "tasks": [
        {
          "id": "1735534084013",
          "message": "clean-vm",
          "start": 1735534084013,
          "status": "success",
          "tasks": [
            {
              "id": "1735534084261",
              "message": "merge",
              "start": 1735534084261,
              "status": "success",
              "end": 1735534084505
            }
          ],
          "end": 1735534084565,
          "result": {
            "merge": true
          }
        },
        {
          "id": "1735534084018",
          "message": "clean-vm",
          "start": 1735534084018,
          "status": "success",
          "tasks": [
            {
              "id": "1735534084190",
              "message": "merge",
              "start": 1735534084190,
              "status": "success",
              "end": 1735534084397
            }
          ],
          "end": 1735534084447,
          "result": {
            "merge": true
          }
        },
        {
          "id": "1735534085223",
          "message": "clean-vm",
          "start": 1735534085223,
          "status": "success",
          "end": 1735534085295,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534085298",
          "message": "clean-vm",
          "start": 1735534085298,
          "status": "success",
          "end": 1735534085343,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534128548",
          "message": "clean-vm",
          "start": 1735534128548,
          "status": "success",
          "end": 1735534128621,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534128559",
          "message": "clean-vm",
          "start": 1735534128559,
          "status": "success",
          "end": 1735534128634,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534129252",
          "message": "clean-vm",
          "start": 1735534129252,
          "status": "success",
          "end": 1735534129327,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534129332",
          "message": "clean-vm",
          "start": 1735534129332,
          "status": "success",
          "end": 1735534129387,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534129399",
          "message": "clean-vm",
          "start": 1735534129399,
          "status": "success",
          "end": 1735534129485,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534129432",
          "message": "clean-vm",
          "start": 1735534129432,
          "status": "success",
          "end": 1735534129549,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534130061",
          "message": "clean-vm",
          "start": 1735534130061,
          "status": "success",
          "end": 1735534130164,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534130166",
          "message": "clean-vm",
          "start": 1735534130166,
          "status": "success",
          "end": 1735534130206,
          "result": {
            "merge": false
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735534130208,
      "result": {
        "code": "VM_LACKS_FEATURE",
        "params": [
          "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956"
        ],
        "task": {
          "uuid": "b31c85f3-e58d-53bc-55ff-7bc122e7c5e5",
          "name_label": "Async.VM.clean_shutdown",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20241230T04:48:49Z",
          "finished": "20241230T04:48:49Z",
          "status": "failure",
          "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "VM_LACKS_FEATURE",
            "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
        },
        "message": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)",
        "name": "XapiError",
        "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)\n    at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1047:24)\n    at file:///root/xen-orchestra/packages/xen-api/index.mjs:1081:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1071:12)\n    at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1244:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
      }
    },
    {
      "data": {
        "type": "VM",
        "id": "5f029e45-a01c-310b-9895-a9e58bf92b96",
        "name_label": "PiHole 🗑️"
      },
      "id": "1735534085350",
      "message": "backup VM",
      "start": 1735534085350,
      "status": "success",
      "tasks": [
        {
          "id": "1735534085357",
          "message": "clean-vm",
          "start": 1735534085357,
          "status": "success",
          "end": 1735534085396,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534085367",
          "message": "clean-vm",
          "start": 1735534085367,
          "status": "success",
          "end": 1735534085431,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735534090190",
          "message": "snapshot",
          "start": 1735534090190,
          "status": "success",
          "end": 1735534091533,
          "result": "16c70b87-c65c-4c5a-660d-30a67446d36f"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": false,
            "type": "remote"
          },
          "id": "1735534091535",
          "message": "export",
          "start": 1735534091535,
          "status": "success",
          "tasks": [
            {
              "id": "1735534093576",
              "message": "transfer",
              "start": 1735534093576,
              "status": "success",
              "end": 1735534126295,
              "result": {
                "size": 765799424
              }
            },
            {
              "id": "1735534128431",
              "message": "clean-vm",
              "start": 1735534128431,
              "status": "success",
              "end": 1735534128472,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1735534128476
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": false,
            "type": "remote"
          },
          "id": "1735534091535:0",
          "message": "export",
          "start": 1735534091535,
          "status": "success",
          "tasks": [
            {
              "id": "1735534093570",
              "message": "transfer",
              "start": 1735534093570,
              "status": "success",
              "end": 1735534127145,
              "result": {
                "size": 765799424
              }
            },
            {
              "id": "1735534128476:0",
              "message": "clean-vm",
              "start": 1735534128476,
              "status": "success",
              "end": 1735534128519,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1735534128520
        }
      ],
      "end": 1735534128521
    }
  ],
  "end": 1735534130209
}

      I then removed the vm without the tools to test if that was the problem and ran the job manually
      Failed again

      {
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1735559371587",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735559371587,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "5f029e45-a01c-310b-9895-a9e58bf92b96"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "5f029e45-a01c-310b-9895-a9e58bf92b96",
        "name_label": "PiHole 🗑️"
      },
      "id": "1735559373175",
      "message": "backup VM",
      "start": 1735559373175,
      "status": "failure",
      "tasks": [
        {
          "id": "1735559373184",
          "message": "clean-vm",
          "start": 1735559373184,
          "status": "success",
          "end": 1735559373261,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735559373193",
          "message": "clean-vm",
          "start": 1735559373193,
          "status": "success",
          "end": 1735559373276,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735559377708",
          "message": "snapshot",
          "start": 1735559377708,
          "status": "success",
          "end": 1735559378775,
          "result": "c27f03ee-f606-6b8f-1d56-564011f7d819"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735559378776",
          "message": "export",
          "start": 1735559378776,
          "status": "failure",
          "end": 1735559378799,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735559378777",
          "message": "export",
          "start": 1735559378777,
          "status": "failure",
          "end": 1735559378793,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735559379622",
          "message": "clean-vm",
          "start": 1735559379622,
          "status": "success",
          "end": 1735559379643,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735559379630",
          "message": "clean-vm",
          "start": 1735559379630,
          "status": "success",
          "end": 1735559379660,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735559379784",
          "message": "snapshot",
          "start": 1735559379784,
          "status": "success",
          "end": 1735559393280,
          "result": "f7929692-70fb-3a83-8c2a-b02d349abd9e"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735559393280:0",
          "message": "export",
          "start": 1735559393280,
          "status": "failure",
          "end": 1735559393289,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735559393280:1",
          "message": "export",
          "start": 1735559393280,
          "status": "failure",
          "end": 1735559393284,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735559394101",
          "message": "clean-vm",
          "start": 1735559394101,
          "status": "success",
          "end": 1735559394121,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735559394110",
          "message": "clean-vm",
          "start": 1735559394110,
          "status": "success",
          "end": 1735559394148,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735559394495",
          "message": "clean-vm",
          "start": 1735559394495,
          "status": "success",
          "end": 1735559394538,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735559394540",
          "message": "clean-vm",
          "start": 1735559394540,
          "status": "success",
          "end": 1735559394562,
          "result": {
            "merge": false
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735559394564,
      "result": {
        "code": "VM_LACKS_FEATURE",
        "params": [
          "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a"
        ],
        "task": {
          "uuid": "b11403fb-1455-886a-dc28-c962ac295f40",
          "name_label": "Async.VM.clean_shutdown",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20241230T11:49:54Z",
          "finished": "20241230T11:49:54Z",
          "status": "failure",
          "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "VM_LACKS_FEATURE",
            "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
        },
        "message": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)",
        "name": "XapiError",
        "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)\n    at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1047:24)\n    at file:///root/xen-orchestra/packages/xen-api/index.mjs:1081:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1071:12)\n    at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1244:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
      }
    }
  ],
  "end": 1735559394565
}

      Happy reading on the new year

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by olivierlambert

        It failed when XO asked the VM with OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956 to do a Async.VM.clean_shutdown. Because without tools, you cannot make a clean shutdown. I don't remember in which scenario you need that action during a backup, probably related to auto healthcheck.

        P 1 Reply Last reply Reply Quote 0
        • P Offline
          ph7 @olivierlambert
          last edited by ph7

          @olivierlambert
          The last log is from the Debian VM only and with 1.0.0-proto-0.4.0 tools
          The one without tools was removed from the job

          Thanks for Your engagement, It's great with people that don't sleep ;o)

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            I cannot tell which VM is OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956 but you can, try to see to which VM it refers 🙂

            P 1 Reply Last reply Reply Quote 0
            • P Offline
              ph7 @olivierlambert
              last edited by ph7

              @olivierlambert

              I cannot tell which VM is OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956 but you can, try to see to which VM it refers 🙂

              It's the one without tools

              But in the third log I only ran the vm with tools and still got errors:
              "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a" and "AssertionError"

              I got no clue

              I installed vm1 (the "unistalled" one), updated and installed the 1.0.0-proto-0.4.0 tools
              made a new simple backup job with normal snapshot and it ran successfully
              I added vm2 and the job ran successfully again.

              Changed the snapshot mode to offline
              SuccessfulI

              I added the same schedule as in the first job
              retention 7
              Number of daily backups kept "blank"
              Number of weekly backups kept 2
              Number of monthly backups kept 1
              Number of yearly backups kept 1
              And it failed.

              Changed Number of daily backups kept "blank" to 7
              And Viola Success.
              It must have been deleted by mistake.
              Seems like it doesn't like the Number of daily backups kept to be blank

              I will change this in the first job and hopefully this is solved.

              And it's not
              I changed Number of daily backups kept "blank" to 7
              Ran the job, Failure.
              Rebooted everything included the host
              Ran the job. Failure again.

              {
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1735582708642",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735582708642,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "5f029e45-a01c-310b-9895-a9e58bf92b96",
          "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "5f029e45-a01c-310b-9895-a9e58bf92b96",
        "name_label": "PiHole 🗑️"
      },
      "id": "1735582710485",
      "message": "backup VM",
      "start": 1735582710485,
      "status": "failure",
      "tasks": [
        {
          "id": "1735582710508",
          "message": "clean-vm",
          "start": 1735582710508,
          "status": "success",
          "end": 1735582710559,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582710511",
          "message": "clean-vm",
          "start": 1735582710511,
          "status": "success",
          "end": 1735582710584,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582714860",
          "message": "snapshot",
          "start": 1735582714860,
          "status": "success",
          "end": 1735582716135,
          "result": "37392150-1519-8b53-dd16-bcbfb64a1972"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582716136",
          "message": "export",
          "start": 1735582716136,
          "status": "failure",
          "end": 1735582716159,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582716137",
          "message": "export",
          "start": 1735582716137,
          "status": "failure",
          "end": 1735582716152,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735582732349",
          "message": "clean-vm",
          "start": 1735582732349,
          "status": "success",
          "end": 1735582732376,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582732364",
          "message": "clean-vm",
          "start": 1735582732364,
          "status": "success",
          "end": 1735582732426,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582732943",
          "message": "clean-vm",
          "start": 1735582732943,
          "status": "success",
          "end": 1735582732990,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582732992",
          "message": "clean-vm",
          "start": 1735582732992,
          "status": "success",
          "end": 1735582733015,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749464",
          "message": "clean-vm",
          "start": 1735582749464,
          "status": "success",
          "end": 1735582749500,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749480",
          "message": "clean-vm",
          "start": 1735582749480,
          "status": "success",
          "end": 1735582749549,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749851",
          "message": "clean-vm",
          "start": 1735582749851,
          "status": "success",
          "end": 1735582749924,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582749926",
          "message": "clean-vm",
          "start": 1735582749926,
          "status": "success",
          "end": 1735582749980,
          "result": {
            "merge": false
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735582749981,
      "result": {
        "code": "VM_LACKS_FEATURE",
        "params": [
          "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a"
        ],
        "task": {
          "uuid": "58309b21-e833-44c8-7b90-dcfee090139a",
          "name_label": "Async.VM.clean_shutdown",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20241230T18:19:09Z",
          "finished": "20241230T18:19:09Z",
          "status": "failure",
          "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "VM_LACKS_FEATURE",
            "OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
        },
        "message": "VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)",
        "name": "XapiError",
        "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:c5030229-917f-b22f-7be6-532cc4cccf5a)\n    at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1068:24)\n    at file:///root/xen-orchestra/packages/xen-api/index.mjs:1102:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1092:12)\n    at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1265:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
      }
    },
    {
      "data": {
        "type": "VM",
        "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
        "name_label": "Debian"
      },
      "id": "1735582717167",
      "message": "backup VM",
      "start": 1735582717167,
      "status": "failure",
      "tasks": [
        {
          "id": "1735582717172",
          "message": "clean-vm",
          "start": 1735582717172,
          "status": "success",
          "end": 1735582717244,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582717180",
          "message": "clean-vm",
          "start": 1735582717180,
          "status": "success",
          "end": 1735582717294,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582730015",
          "message": "snapshot",
          "start": 1735582730015,
          "status": "success",
          "end": 1735582731315,
          "result": "160221cd-fff0-d481-b05f-64c27735f740"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582731316",
          "message": "export",
          "start": 1735582731316,
          "status": "failure",
          "end": 1735582731340,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582731317",
          "message": "export",
          "start": 1735582731317,
          "status": "failure",
          "end": 1735582731333,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735582733030",
          "message": "clean-vm",
          "start": 1735582733030,
          "status": "success",
          "end": 1735582733121,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582733038",
          "message": "clean-vm",
          "start": 1735582733038,
          "status": "success",
          "end": 1735582733207,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582733338",
          "message": "snapshot",
          "start": 1735582733338,
          "status": "success",
          "end": 1735582748462,
          "result": "f11f0d4c-0804-2bbe-960d-75c5c11305f1"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582748462:0",
          "message": "export",
          "start": 1735582748462,
          "status": "failure",
          "end": 1735582748486,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735582748462:1",
          "message": "export",
          "start": 1735582748462,
          "status": "failure",
          "end": 1735582748469,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735582749990",
          "message": "clean-vm",
          "start": 1735582749990,
          "status": "success",
          "end": 1735582750034,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582750019",
          "message": "clean-vm",
          "start": 1735582750019,
          "status": "success",
          "end": 1735582750124,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582750491",
          "message": "clean-vm",
          "start": 1735582750491,
          "status": "success",
          "end": 1735582750554,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735582750581",
          "message": "clean-vm",
          "start": 1735582750581,
          "status": "success",
          "end": 1735582750631,
          "result": {
            "merge": false
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735582750633,
      "result": {
        "code": "VM_LACKS_FEATURE",
        "params": [
          "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956"
        ],
        "task": {
          "uuid": "fe77d196-a6f0-12ca-5795-3669fcf16201",
          "name_label": "Async.VM.clean_shutdown",
          "name_description": "",
          "allowed_operations": [],
          "current_operations": {},
          "created": "20241230T18:19:10Z",
          "finished": "20241230T18:19:10Z",
          "status": "failure",
          "resident_on": "OpaqueRef:ed939ef0-4062-e338-ea05-6fc86232259e",
          "progress": 1,
          "type": "<none/>",
          "result": "",
          "error_info": [
            "VM_LACKS_FEATURE",
            "OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956"
          ],
          "other_config": {},
          "subtask_of": "OpaqueRef:NULL",
          "subtasks": [],
          "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 742))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1653))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1662))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2034))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
        },
        "message": "VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)",
        "name": "XapiError",
        "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:f3db3478-5acb-3723-5d15-652879bcf956)\n    at XapiError.wrap (file:///root/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///root/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///root/xen-orchestra/packages/xen-api/index.mjs:1068:24)\n    at file:///root/xen-orchestra/packages/xen-api/index.mjs:1102:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1092:12)\n    at Xapi._watchEvents (file:///root/xen-orchestra/packages/xen-api/index.mjs:1265:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
      }
    }
  ],
  "end": 1735582750633
}

              I don't know, I will do some more testing.
              And thanks again Oliver 🥳

              P 1 Reply Last reply Reply Quote 0
              • P Offline
                ph7 @ph7
                last edited by ph7

                I compared the 2 jobs to make sure they where exactly the same.
                Ran the new job. Success.
                Ran the old job. Failed.

                Removed all data from Long-term retention of backups in the old job
                Number of daily backups kept 7 to "blank"
                Number of weekly backups kept 2 to "blank"
                Number of monthly backups kept 1 to "blank"
                Number of yearly backups kept 1 to "blank"

                Ran the old job again. Success !

                Put the data back to 7 2 1 1
                Ran the old job. Failed.

                Hmmm.. I think there must be an undocumented feature here 🙂
                I will run the new job alongside the old job an hour apart to see what will happen.

                1 Reply Last reply Reply Quote 0
                • D Offline
                  David
                  last edited by

                  I'm getting similar too, I get the "entries must be sorted in asc order 2025-01 2025-52" error (see below) hope that's helpful.

                  Based on the comment about removing the Long-term retention of backups data, I just removed the Number of weekly backups kept entry and re-ran, this allowed the backup to run. I kept my Number of daily backups kept limit which I haven't reached yet.

                        "message": "backup VM",
      "start": 1735640611472,
      "status": "failure",
      "tasks": [
        {
          "id": "1735640611748",
          "message": "snapshot",
          "start": 1735640611748,
          "status": "success",
          "end": 1735640616464,
          "result": "1f91db46-fbfe-ba09-9f8b-1c350d7493fd"
        },
        {
          "data": {
            "id": "59e1ac20-f03d-45b4-953d-6fb7277537c2",
            "type": "remote",
            "isFull": true
          },
          "id": "1735640616501",
          "message": "export",
          "start": 1735640616501,
          "status": "failure",
          "end": 1735640616505,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///home/root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at FullRemoteWriter._run (file:///home/root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/FullRemoteWriter.mjs:39:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async FullRemoteWriter.run (file:///home/root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/_AbstractFullWriter.mjs:6:14)"
          }
                  R C 2 Replies Last reply Reply Quote 0
                  • R Offline
                    rmaclachlan @David
                    last edited by

                    Thank you for reporting this fix, after I removed the Number of weekly backups kept entry my backups are working again. Last night all our backups failed with the entries must be sorted in asc order 2025-01 2025-52 error.

                    1 Reply Last reply Reply Quote 0
                    • P Offline
                      ph7
                      last edited by

                      I have now, on the new year, added a new schedule to see if there will be any difference

                      1 Reply Last reply Reply Quote 0
                      • C Offline
                        chr1st0ph9 @David
                        last edited by

                        @David Thanks, my backups suddenly fail since today, after removing all retention settings except “daily backup”, they work perfectly as last year.

                        1 Reply Last reply Reply Quote 0
                        • P Offline
                          ph7 @olivierlambert
                          last edited by

                          @olivierlambert said in Long-term retention of backups:

                          @ph7 said in Long-term retention of backups:

                          VM_LACKS_FEATURE

                          This means there's no tools installed or working on that VM

                          The original job is still not running
                          I was thinking about this and came up with this cunning plan 🙂

                          1. Shut down the VM to see if there is still a problem with the tools
                          2. Run the original job manually

                          No. it did not work, still failing entries must be sorted in asc order 2025-01 2025-52

                          {
  "data": {
    "mode": "delta",
    "reportWhen": "failure",
    "backupReportTpl": "mjml"
  },
  "id": "1735908223694",
  "jobId": "6a12854b-cc07-4269-b780-fe87df6cd867",
  "jobName": "LongTerm-5+2",
  "message": "backup",
  "scheduleId": "4f8cc819-2cf7-42ac-94ae-ee005727ecf7",
  "start": 1735908223694,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "1b75faf0-f4db-5aa9-d204-9bee6f2abfd6",
        "name_label": "Debian"
      },
      "id": "1735908225556",
      "message": "backup VM",
      "start": 1735908225556,
      "status": "failure",
      "tasks": [
        {
          "id": "1735908225574",
          "message": "clean-vm",
          "start": 1735908225574,
          "status": "success",
          "end": 1735908225783,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908225605",
          "message": "clean-vm",
          "start": 1735908225605,
          "status": "success",
          "end": 1735908225828,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908226496",
          "message": "snapshot",
          "start": 1735908226496,
          "status": "success",
          "end": 1735908231603,
          "result": "4cf9de45-ff3f-f1a1-8a86-7b5cb03c9dd5"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908231603:0",
          "message": "export",
          "start": 1735908231603,
          "status": "failure",
          "end": 1735908231670,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908231605",
          "message": "export",
          "start": 1735908231605,
          "status": "failure",
          "end": 1735908231657,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735908232625",
          "message": "clean-vm",
          "start": 1735908232625,
          "status": "success",
          "end": 1735908232700,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908232656",
          "message": "clean-vm",
          "start": 1735908232656,
          "status": "success",
          "end": 1735908232767,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908232981",
          "message": "snapshot",
          "start": 1735908232981,
          "status": "success",
          "end": 1735908234120,
          "result": "f614fec5-87a5-c227-4e0c-8fa76acbb271"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908234121",
          "message": "export",
          "start": 1735908234121,
          "status": "failure",
          "end": 1735908234135,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908234121:0",
          "message": "export",
          "start": 1735908234121,
          "status": "failure",
          "end": 1735908234129,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "id": "1735908234974",
          "message": "clean-vm",
          "start": 1735908234974,
          "status": "success",
          "end": 1735908235063,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908234994",
          "message": "clean-vm",
          "start": 1735908234994,
          "status": "success",
          "end": 1735908235091,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1735908235345",
          "message": "snapshot",
          "start": 1735908235345,
          "status": "success",
          "end": 1735908236519,
          "result": "319c12f9-6f35-411d-2a99-b2c55f7d469b"
        },
        {
          "data": {
            "id": "446f82eb-2794-4c01-bd71-e247abd6a39c",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908236520",
          "message": "export",
          "start": 1735908236520,
          "status": "failure",
          "end": 1735908236533,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        },
        {
          "data": {
            "id": "0927f4dc-4476-4ac3-959a-bc37e85956df",
            "isFull": true,
            "type": "remote"
          },
          "id": "1735908236520:0",
          "message": "export",
          "start": 1735908236520,
          "status": "failure",
          "end": 1735908236527,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        },
        {
          "data": {
            "attempt": 2,
            "error": "all targets have failed, step: writer.prepare()"
          },
          "message": "Retry the VM backup due to an error"
        }
      ],
      "end": 1735908237352,
      "result": {
        "errors": [
          {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          },
          {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "strictEqual",
            "message": "entries must be sorted in asc order 2025-01 2025-52",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: entries must be sorted in asc order 2025-01 2025-52\n    at getOldEntries (file:///root/xen-orchestra/@xen-orchestra/backups/_getOldEntries.mjs:116:18)\n    at IncrementalRemoteWriter._prepare (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:96:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"
          }
        ],
        "message": "all targets have failed, step: writer.prepare()",
        "name": "Error",
        "stack": "Error: all targets have failed, step: writer.prepare()\n    at IncrementalXapiVmBackupRunner._callWriters (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:64:13)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async IncrementalXapiVmBackupRunner._copy (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:42:5)\n    at async IncrementalXapiVmBackupRunner.run (file:///root/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:396:9)\n    at async file:///root/xen-orchestra/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }
    }
  ],
  "end": 1735908237354
}

                          Maybe I'm wrong but I think we have a problem here. Could have something to do with the new year.

                          DanpD 1 Reply Last reply Reply Quote 0
                          • DanpD Offline
                            Danp Pro Support Team @ph7
                            last edited by

                            @ph7 This is being investigated by the XOA team. We will update this thread when more information is available.

                            A 1 Reply Last reply Reply Quote 1
                            • D DustinB referenced this topic on
                            • A Offline
                              al-indigo @Danp
                              last edited by

                              @Danp Any news?
                              I have no delta-backups since new year and I don't know what will happen if I set retention "Backup retention" to 0.
                              I don't see any options to disable retention temporarily so I disabled daily backups. We have full monthly backups, but daily and weekly stopped working and it's disturbing.
                              We use open-source edition, no offence or pressure intended; just curious since maybe it's reasonable to delete all backups after next monthly full-backup is complete

                              DanpD 1 Reply Last reply Reply Quote 0
                              • DanpD Offline
                                Danp Pro Support Team @al-indigo
                                last edited by

                                @al-indigo I believe that a fix should be out by EOM.

                                A 2 Replies Last reply Reply Quote 0
                                • A Offline
                                  al-indigo @Danp
                                  last edited by

                                  @Danp Could you please tell what will happen if I set "0" in retention field and enable backups again? Will it erase all my delta-backups there?
                                  54226b89-ae07-4056-925d-797d4ff50b0a-image.png

                                  DanpD 1 Reply Last reply Reply Quote 0
                                  • DanpD Offline
                                    Danp Pro Support Team @al-indigo
                                    last edited by

                                    @al-indigo AFAIK, the backup retention settings from the schedule won't be affected by changing the LTR values to zeros.

                                    1 Reply Last reply Reply Quote 0
                                    • A Offline
                                      al-indigo @Danp
                                      last edited by

                                      @Danp Do you have any updates on fix release estimation?

                                      DanpD 1 Reply Last reply Reply Quote 0
                                      • DanpD Offline
                                        Danp Pro Support Team @al-indigo
                                        last edited by

                                        @al-indigo The fix is included in XOA 5.103.0, which was released last week.

                                        https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md

                                        [Backup/LTR] Fix computation for the last week of the year (PR #8269)

                                        A 1 Reply Last reply Reply Quote 1
                                        • A Offline
                                          al-indigo @Danp
                                          last edited by

                                          @Danp Thank you very much! Missed it

                                          1 Reply Last reply Reply Quote 1
                                          • olivierlambertO olivierlambert marked this topic as a question on
                                          • olivierlambertO olivierlambert has marked this topic as solved on
                                          • First post
                                            Last post