Error mirroring full backups to backblaze b2

[poddingue]

Quick one to close this off: florent's fix for the encrypted-remote mirror alignment landed in a numbered release now, XO 6.6.2 (2026-07-09).
The PR was https://github.com/vatesfr/xen-orchestra/pull/10061, so it's out of master and into something you can just update to.
Did your retest come good on it?
I'm also curious whether you still need the minPartSize=100000000 setting with the fix in place, or whether that's redundant now, since it'd help us work out whether it's worth writing into the B2 docs.

[pedro_udifar]

@poddingue Hi,

The backup mirror job is still running, and I'm waiting for the larger VMs to complete, but so far it looks good - one backup, larger than 50GB was successfully mirrored without specifying minPartSize.

[pedro_udifar]

@poddingue Hi!

Unfortunately, another error came up on the larger vms. The job also seems to be painfully slow. Slower than usual I mean.
Also, it seems that job changes are not being commited. I've changed the job so it wouldn't retry in case of failure and the log shows it still trying 3 times (that was the old setting, and probably explains the job taking longer than usual).

{
  "id": "0mrjahr8e-ssawlbveq5",
  "start": 1783951215134,
  "status": "failure",
  "tasks": [
    {
      "id": "0mrjahs3e-dbfaqajeze",
      "start": 1783951216250,
      "status": "failure",
      "tasks": [
        {
          "id": "0mrjahs8s-2yb6i3fny6f",
          "start": 1783951216444,
          "status": "failure",
          "end": 1783959266484,
          "result": {
            "$fault": "server",
            "$metadata": {
              "httpStatusCode": 500,
              "requestId": "c0a9f30190109f99",
              "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
              "attempts": 3,
              "totalRetryDelay": 197
            },
            "name": "InternalError",
            "Code": "InternalError",
            "message": "internal incident",
            "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
          },
          "message": "transfer",
          "data": {
            "progress": 40
          }
        }
      ],
      "end": 1783959266485,
      "result": {
        "$fault": "server",
        "$metadata": {
          "httpStatusCode": 500,
          "requestId": "c0a9f30190109f99",
          "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
          "attempts": 3,
          "totalRetryDelay": 197
        },
        "name": "InternalError",
        "Code": "InternalError",
        "message": "internal incident",
        "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
      },
      "message": "export",
      "data": {
        "id": "14b59f44-4517-411e-9e39-4eb7136928d0",
        "type": "remote",
        "isFull": true
      }
    },
    {
      "id": "0mrjfabp3-c8mf2pnq8yf",
      "start": 1783959266487,
      "status": "success",
      "warnings": [
        {
          "data": {
            "path": "xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/cache.json.gz",
            "actual": 0,
            "expected": 1
          },
          "message": "unexpected number of entries in backup cache"
        }
      ],
      "end": 1783959266989,
      "result": {
        "merge": false,
        "size": 0
      },
      "message": "clean-vm"
    }
  ],
  "end": 1783959266989,
  "result": {
    "$fault": "server",
    "$metadata": {
      "httpStatusCode": 500,
      "requestId": "c0a9f30190109f99",
      "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
      "attempts": 3,
      "totalRetryDelay": 197
    },
    "name": "InternalError",
    "Code": "InternalError",
    "message": "internal incident",
    "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
  },
  "message": "backup VM",
  "data": {
    "id": "60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee",
    "type": "VM",
    "progress": 0
  }
},

[pierrebrunet]

@pedro_udifar Hi,
This is a b2 side error. It could be just a transient error, did you try to restart the backup?

[pedro_udifar]

This post is deleted!

[pedro_udifar]

This post is deleted!

[pedro_udifar]

@pierrebrunet Hello!

I've checked B2 logs and found some errors like you had foreseen. I had a look and this kind of 500 INTERNAL ERROR several times on other jobs that seem ok. How many retries does XO attempt to send the part when this happens?

I'll do another run that includes only a few large VMs. BTW, is there a way to interrupt the job withou restarting XO ?

REST.PUT.PART Weekly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva "PUT /zzzzzkly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva?partNumber=3954&uploadId=4_ze092693d1b65962e9fe6051b_f25969add7e42e384_d20260713_m140016_c003_v0312040_t0013_u01783951216751&x-id=UploadPart HTTP/1.1" 500 INTERNAL_ERROR 145 - 3762 3762 "-" "aws-sdk-js/3.1065.0 ua/2.1 os/linux#5.15.0-185-generic lang/js md/nodejs#24.18.0 api/s3#3.1065.0 m/N,a,c,E,e xen-orchestra-fs-4.9.2" - - - TLS_AES_256_GCM_SHA384 AuthHeader s3.eu-central-003.backblazeb2.com TLSv1.3 - -

[pedro_udifar]

@pierrebrunet

Large VMs still fail with the same error (internal incident), the process is not reliable - don't know if it's a B2 problem or XO.
When I have some time, I'll do another run setting minPartSize=100000000 and see if it helps.

[pierrebrunet]

@pedro_udifar Hi,
You can try again with minPartSize=100000000 indeed. But I would advise you to contact B2 directly. They should have a clearer error to show instead of a 500.
Thanks for your patience and your help

[pedro_udifar]

@pierrebrunet Hi!

I'm on vacation now, so sorry for the delay. In the meantime I've been analyzing these issues and it seems these are somewhat expected and still happen with minpartsize set to 100M.

My next step is to increase the AWS_MAX_ATTEMPTS from the default 3 attempts to 10. This might reduce the number of failed VMs and avoid the need to retry the full VM backup.

From AI research, it seems rclone uses 10 attempts instead of the AWS SDK default.

Here are some stats of my weekly full backup to B2:
"00 Weekly Full Backblaze" ran xxxx → xxxxx — 34.1 hours total, moving ~2.86 TiB across 89 VMs. Final status: failure (more on that below, because it's misleading) -> this is related to XO reporting the backup as a failure even though the follow up retry succeded which is a known issue.

Every one of the 7 first-attempt errors shows attempts: 3 in the SDK metadata — the default budget was exhausted in all cases -> 7 of 89 VMs failed on the initial attempt.

Error mix: 5× Backblaze InternalError 500, 1× ECONNRESET, 1× EPIPE

The 7 failed first attempts burned ~7.1 hours of cumulative transfer time

Even though XO does not expose AWS_MAX_ATTEMPTS parameter I might be able to set it using an environment variable. I'll give a try when I'm back at the office.

Tks,
Pedro