Error mirroring full backups to backblaze b2

pedro_udifar

Replication is from local NFS (Ztsd compressed, not encripted) to a S3 Backblaze (with encription).

pierrebrunet

@pedro_udifar Hi,
We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream

This should fix the stream issue and fix the warning.

florent

both fixes are now on master AND released as a patch

thank you @pedro_udifar

pedro_udifar

@florent

Sorry, missed your previous post. I will return to master and test it (over the weekend).

Thank you!

poddingue

Quick one to close this off: florent's fix for the encrypted-remote mirror alignment landed in a numbered release now, XO 6.6.2 (2026-07-09).
The PR was https://github.com/vatesfr/xen-orchestra/pull/10061, so it's out of master and into something you can just update to.
Did your retest come good on it?
I'm also curious whether you still need the minPartSize=100000000 setting with the fix in place, or whether that's redundant now, since it'd help us work out whether it's worth writing into the B2 docs.

pedro_udifar

@poddingue Hi,

The backup mirror job is still running, and I'm waiting for the larger VMs to complete, but so far it looks good - one backup, larger than 50GB was successfully mirrored without specifying minPartSize.

pedro_udifar

@poddingue Hi!

Unfortunately, another error came up on the larger vms. The job also seems to be painfully slow. Slower than usual I mean.
Also, it seems that job changes are not being commited. I've changed the job so it wouldn't retry in case of failure and the log shows it still trying 3 times (that was the old setting, and probably explains the job taking longer than usual).

{
  "id": "0mrjahr8e-ssawlbveq5",
  "start": 1783951215134,
  "status": "failure",
  "tasks": [
    {
      "id": "0mrjahs3e-dbfaqajeze",
      "start": 1783951216250,
      "status": "failure",
      "tasks": [
        {
          "id": "0mrjahs8s-2yb6i3fny6f",
          "start": 1783951216444,
          "status": "failure",
          "end": 1783959266484,
          "result": {
            "$fault": "server",
            "$metadata": {
              "httpStatusCode": 500,
              "requestId": "c0a9f30190109f99",
              "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
              "attempts": 3,
              "totalRetryDelay": 197
            },
            "name": "InternalError",
            "Code": "InternalError",
            "message": "internal incident",
            "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
          },
          "message": "transfer",
          "data": {
            "progress": 40
          }
        }
      ],
      "end": 1783959266485,
      "result": {
        "$fault": "server",
        "$metadata": {
          "httpStatusCode": 500,
          "requestId": "c0a9f30190109f99",
          "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
          "attempts": 3,
          "totalRetryDelay": 197
        },
        "name": "InternalError",
        "Code": "InternalError",
        "message": "internal incident",
        "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
      },
      "message": "export",
      "data": {
        "id": "14b59f44-4517-411e-9e39-4eb7136928d0",
        "type": "remote",
        "isFull": true
      }
    },
    {
      "id": "0mrjfabp3-c8mf2pnq8yf",
      "start": 1783959266487,
      "status": "success",
      "warnings": [
        {
          "data": {
            "path": "xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/cache.json.gz",
            "actual": 0,
            "expected": 1
          },
          "message": "unexpected number of entries in backup cache"
        }
      ],
      "end": 1783959266989,
      "result": {
        "merge": false,
        "size": 0
      },
      "message": "clean-vm"
    }
  ],
  "end": 1783959266989,
  "result": {
    "$fault": "server",
    "$metadata": {
      "httpStatusCode": 500,
      "requestId": "c0a9f30190109f99",
      "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
      "attempts": 3,
      "totalRetryDelay": 197
    },
    "name": "InternalError",
    "Code": "InternalError",
    "message": "internal incident",
    "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
  },
  "message": "backup VM",
  "data": {
    "id": "60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee",
    "type": "VM",
    "progress": 0
  }
},

pierrebrunet

@pedro_udifar Hi,
This is a b2 side error. It could be just a transient error, did you try to restart the backup?

pedro_udifar

This post is deleted!

pedro_udifar

This post is deleted!

pedro_udifar

@pierrebrunet Hello!

I've checked B2 logs and found some errors like you had foreseen. I had a look and this kind of 500 INTERNAL ERROR several times on other jobs that seem ok. How many retries does XO attempt to send the part when this happens?

I'll do another run that includes only a few large VMs. BTW, is there a way to interrupt the job withou restarting XO ?

REST.PUT.PART Weekly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva "PUT /zzzzzkly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva?partNumber=3954&uploadId=4_ze092693d1b65962e9fe6051b_f25969add7e42e384_d20260713_m140016_c003_v0312040_t0013_u01783951216751&x-id=UploadPart HTTP/1.1" 500 INTERNAL_ERROR 145 - 3762 3762 "-" "aws-sdk-js/3.1065.0 ua/2.1 os/linux#5.15.0-185-generic lang/js md/nodejs#24.18.0 api/s3#3.1065.0 m/N,a,c,E,e xen-orchestra-fs-4.9.2" - - - TLS_AES_256_GCM_SHA384 AuthHeader s3.eu-central-003.backblazeb2.com TLSv1.3 - -