Delta backup footer1 !== footer2 error

[Alexander 0]

@olivierlambert Yesterday I deleted snapshots of delta backups and it made full backups successful. Today I see errors again with incremental.

[olivierlambert]

Please rebuild everything on master and try again. It's hard to investigate when we do not control entirely the environment.

[Alexander 0]

@olivierlambert I have rebuild xen orchestra with last master.
Now I recreating backup job. Will see if there will be same issues.

[Alexander 0]

@olivierlambert and today I have first failed backup with same error.

[olivierlambert]

To rule out an XO problem (but something specific on your setup), please try with an XOA on latest If you need another trial round for that, let me know I'll unlock it!

[Alexander 0]

@olivierlambert I see today's commit with changes to merge process.
May this help?

[olivierlambert]

Feel free to try and report then thanks!

[Alexander 0]

@olivierlambert I've set up XOA trial. Will see if problem appear.

[Alexander 0]

@olivierlambert
I got the same error on XOA appliance.

{
  "data": {
    "mode": "delta",
    "reportWhen": "always"
  },
  "id": "1620250200011",
  "jobId": "f44dc556-32ba-4bbc-8568-06d5d26b6835",
  "jobName": "discourse-BUp",
  "message": "backup",
  "scheduleId": "bb7b53e4-ca32-48f8-afbb-1ce36ddc84f2",
  "start": 1620250200011,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "a1a3832c-b16a-a85e-2fdd-ae585087f629"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "a1a3832c-b16a-a85e-2fdd-ae585087f629"
      },
      "id": "1620250200710",
      "message": "backup VM",
      "start": 1620250200710,
      "status": "failure",
      "tasks": [
        {
          "id": "1620250205577",
          "message": "snapshot",
          "start": 1620250205577,
          "status": "success",
          "end": 1620250210934,
          "result": "569701f3-2665-32c1-ef1c-2b8b1377d88b"
        },
        {
          "data": {
            "id": "84dd7240-14e8-4233-befa-f95e7d62e74e",
            "isFull": false,
            "type": "remote"
          },
          "id": "1620250210935",
          "message": "export",
          "start": 1620250210935,
          "status": "failure",
          "tasks": [
            {
              "id": "1620250211055",
              "message": "transfer",
              "start": 1620250211055,
              "status": "failure",
              "end": 1620250232442,
              "result": {
                "generatedMessage": false,
                "code": "ERR_ASSERTION",
                "actual": false,
                "expected": true,
                "operator": "==",
                "message": "footer1 !== footer2",
                "name": "AssertionError",
                "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n    at Vhd.readHeaderAndFooter (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/dist/vhd.js:118:27)\n    at async checkVhd (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_checkVhd.js:4:3)\n    at async RemoteAdapter.outputStream (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/RemoteAdapter.js:517:7)\n    at async /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_DeltaBackupWriter.js:196:11\n    at async Promise.all (index 0)\n    at async /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_DeltaBackupWriter.js:171:7"
              }
            }
          ],
          "end": 1620250232442,
          "result": {
            "generatedMessage": false,
            "code": "ERR_ASSERTION",
            "actual": false,
            "expected": true,
            "operator": "==",
            "message": "footer1 !== footer2",
            "name": "AssertionError",
            "stack": "AssertionError [ERR_ASSERTION]: footer1 !== footer2\n    at Vhd.readHeaderAndFooter (/usr/local/lib/node_modules/xo-server/node_modules/vhd-lib/dist/vhd.js:118:27)\n    at async checkVhd (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_checkVhd.js:4:3)\n    at async RemoteAdapter.outputStream (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/RemoteAdapter.js:517:7)\n    at async /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_DeltaBackupWriter.js:196:11\n    at async Promise.all (index 0)\n    at async /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_DeltaBackupWriter.js:171:7"
          }
        }
      ],
      "end": 1620250233919,
      "result": {
        "message": "task has already ended",
        "name": "Error",
        "stack": "Error: task has already ended\n    at Task.logAfterEnd (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/Task.js:5:9)\n    at Task.onLog (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/Task.js:59:37)\n    at Task.#log (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/Task.js:139:16)\n    at new Task (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/Task.js:75:14)\n    at Function.run (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/Task.js:33:12)\n    at DeltaBackupWriter._deleteOldEntries (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_DeltaBackupWriter.js:115:17)\n    at DeltaBackupWriter.cleanup (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_DeltaBackupWriter.js:110:18)\n    at /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/Task.js:129:32\n    at /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/Task.js:103:24\n    at Zone.run (/usr/local/lib/node_modules/xo-server/node_modules/node-zone/index.js:80:23)"
      }
    }
  ],
  "end": 1620250233919
}

I have created a bug ticket with a link to this topic.

[Alexander 0]

@olivierlambert problem was fixed. Thank you for help.

[MrNaz]

@alexander-0
I am having this exact same issue. Can you tell me what you did to fix it? Or was it fixed when you updated to the latest version?

[Alexander 0]

@mrnaz the fixed it in version 5.58.1. So update helped.