Backup success or failure ?

[peo]

I'm continuing to investigate a persistent (but random) backup issue I've encountered during my use of xcp-ng. I’ve previously raised this as part of other backup-related threads. The backup reports for some of my jobs continues to indicate a failure (specifically for "overall status" and "remote status"), even though the backup transfer itself appears to complete successfully.

The email backup report:

When testing the remote from the settings, it is working without any problem.

To provide more context, I recently performed a restore health check and a full restore to a new VM after the original machine received an OS and Docker update. This involved reverting the machine back to its pre-update state. This restore process was successful. Notably, when I used xo-cli to re-run the backup on this machine (from a sequence of 6 machines, all of which failed with ”Error: invalid HTTP header in response body"), it was the only one reporting this kind of error.

I’ve attached the full JSON log for this job.

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1748358247453",
  "jobId": "b9142eba-4954-414a-bbc0-180b071a6490",
  "jobName": "Docker",
  "message": "backup",
  "scheduleId": "148eb0e1-1a22-4966-9143-ce53849875de",
  "start": 1748358247453,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "2af0192f-3cb3-daa2-33e6-ec8f18b0cfb1"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "2af0192f-3cb3-daa2-33e6-ec8f18b0cfb1",
        "name_label": "Docker"
      },
      "id": "1748358250243",
      "message": "backup VM",
      "start": 1748358250243,
      "status": "failure",
      "tasks": [
        {
          "id": "1748358250247",
          "message": "clean-vm",
          "start": 1748358250247,
          "status": "success",
          "end": 1748358253304,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1748358253543",
          "message": "snapshot",
          "start": 1748358253543,
          "status": "success",
          "end": 1748358263363,
          "result": "f938e63e-3d5c-517c-f487-a28e2d93d44f"
        },
        {
          "data": {
            "id": "2b919467-704c-4e35-bac9-2d6a43118bda",
            "isFull": true,
            "type": "remote"
          },
          "id": "1748358263364",
          "message": "export",
          "start": 1748358263364,
          "status": "failure",
          "tasks": [
            {
              "id": "1748358268282",
              "message": "transfer",
              "start": 1748358268282,
              "status": "success",
              "end": 1748358860438,
              "result": {
                "size": 38249955328
              }
            },
            {
              "id": "1748358884098",
              "message": "clean-vm",
              "start": 1748358884098,
              "status": "failure",
              "end": 1748358895353,
              "result": {
                "name": "InternalError",
                "$fault": "client",
                "$metadata": {
                  "httpStatusCode": 500,
                  "requestId": "074282D2C6F6946B",
                  "extendedRequestId": "MDc0MjgyRDJDNkY2OTQ2QjA3NDI4MkQyQzZGNjk0NkIwNzQyODJEMkM2RjY5NDZCMDc0MjgyRDJDNkY2OTQ2Qg==",
                  "attempts": 3,
                  "totalRetryDelay": 89
                },
                "Code": "InternalError",
                "message": "Internal Error",
                "stack": "InternalError: Internal Error\n    at throwDefaultError (/opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@smithy/smithy-client/dist-cjs/index.js:867:20)\n    at /opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@smithy/smithy-client/dist-cjs/index.js:876:5\n    at de_CommandError (/opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@aws-sdk/client-s3/dist-cjs/index.js:4952:14)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@smithy/middleware-serde/dist-cjs/index.js:35:20\n    at async /opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:484:18\n    at async /opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@smithy/middleware-retry/dist-cjs/index.js:320:38\n    at async /opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:110:22\n    at async /opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:137:14\n    at async /opt/xo/xo-builds/xen-orchestra-202505261454/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:33:22"
              }
            }
          ],
          "end": 1748358895354
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:f5151a82-9e27-e821-fb17-067c7c1de856"
          },
          "message": "Snapshot data has been deleted"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:d8613b53-fe2b-014a-da5b-db943ec55b67"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1748358895355
    }
  ],
  "end": 1748358895355
}