XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Error mirroring full backups to backblaze b2

    Scheduled Pinned Locked Moved Backup
    27 Posts 4 Posters 2.8k Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      pedro_udifar
      last edited by

      Still getting these warnings, on the VMs that don't need to be uploaded. Thought this might go away also.

      838c12d7-315d-4e4d-bc76-acb4dd9e74d9-image.jpeg

      1 Reply Last reply Reply Quote 0
      • P Offline
        pedro_udifar
        last edited by

        Hello Florent,

        The new code is still failing...

        a7b27068-0bbb-4ea6-b0e7-74f8058775f3-image.jpeg

        	```
	{
		"id": "0mr3li2kt-c10ifxuuw68",
		"start": 1783002246797,
		"status": "failure",
		"tasks": [
			{
				"id": "0mr3li2ud-r1vgn4rnv5d",
				"start": 1783002247141,
				"status": "failure",
				"tasks": [
					{
						"id": "0mr3li30l-6n1cr5a6o7v",
						"start": 1783002247365,
						"status": "failure",
						"end": 1783004124397,
						"result": {
							"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
							"name": "Error",
							"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
						},
						"message": "transfer",
						"data": {
							"progress": 0
						}
					}
				],
				"end": 1783004124397,
				"result": {
					"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
					"name": "Error",
					"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
				},
				"message": "export",
				"data": {
					"id": "14b59f44-4517-411e-9e39-4eb7136928d0",
					"type": "remote",
					"isFull": true
				}
			},
			{
				"id": "0mr3mmbcf-yc4lpd0z5p",
				"start": 1783004124399,
				"status": "success",
				"warnings": [
					{
						"data": {
							"path": "xo-vm-backups/1c842f34-f7f8-d3cd-ba24-ffe51d6ae075/cache.json.gz",
							"actual": 0,
							"expected": 1
						},
						"message": "unexpected number of entries in backup cache"
					},
					{
						"data": {
							"error": {
								"generatedMessage": false,
								"code": "ERR_ASSERTION",
								"actual": true,
								"expected": false,
								"operator": "strictEqual",
								"diff": "simple"
							}
						},
						"message": "Issue while checking XVA"
					},
					{
						"data": {
							"error": {
								"generatedMessage": false,
								"code": "ERR_ASSERTION",
								"actual": true,
								"expected": false,
								"operator": "strictEqual",
								"diff": "simple"
							}
						},
						"message": "Checksum file not valid, not blocking"
					},
					{
						"data": {
							"path": "/xo-vm-backups/1c842f34-f7f8-d3cd-ba24-ffe51d6ae075/20260624T231818Z.xva"
						},
						"message": "XVA might be broken"
					},
					{
						"data": {
							"files": [
								"20260624T231818Z.json",
								"20260624T231818Z.xva",
								"20260624T231818Z.xva.checksum"
							]
						},
						"message": "This files may be corrupted but not yet to be removed"
					}
				],
				"end": 1783004124983,
				"result": {
					"merge": false,
					"size": 0
				},
				"message": "clean-vm"
			}
		],
		"end": 1783004124983,
		"result": {
			"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
			"name": "Error",
			"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
		},
		"message": "backup VM",
		"data": {
			"id": "1c842f34-f7f8-d3cd-ba24-ffe51d6ae075",
			"type": "VM",
			"progress": 0
		}
	},

        Best regards,
        Pedro

        florentF 1 Reply Last reply Reply Quote 0
        • florentF Offline
          florent Vates 🪐 XO Team @pedro_udifar
          last edited by

          @pedro_udifar that is strange that we have exactly the same error, even though we should have added 28 bytes

          I pushed a little debug that will log additional lines in journalctl

          'will output stream ', 'stream is encrypted' and 'stream is updated '

          Can you update your branch, rebuild, rerun the transfer and post the log ( with the attached values ) ?

          regards

          P 1 Reply Last reply Reply Quote 0
          • P Offline
            pedro_udifar @florent
            last edited by

            @florent

            Sure, but I can only do it over the weekend.
            Also, I'll try to check if the script built the correct branch. First time using the script to build anything other than master.

            On a side note: recently some other strange errors started to show up "Body timeout Error". These are plain full backups to a remote with disaster recovery for another cluster, that always worked fine. Also, in this case, the timeout defined within the job doesn't cancel the job and the remaining jobs in the sequence don't run. Haven't had the time to dig into this - when I do, I'll start a new forum post.

            790bd845-1932-4f33-abfd-36441d720aa1-image.jpeg

            Cheers,
            Pedro

            1 Reply Last reply Reply Quote 0
            • P Offline
              pedro_udifar
              last edited by

              Hello again!

              c5189e53-c2d1-4de5-a26d-243ddd41e5db-image.jpeg

              Here are the logs:
              b2_mirror_error.txt

              Hope it helps!

              P 1 Reply Last reply Reply Quote 1
              • P Offline
                pierrebrunet Vates 🪐 XO Team @pedro_udifar
                last edited by

                @pedro_udifar Hi, just to be sure, this is a replication job with two s3? If yes, is the source s3 encrypted or is it only the target?

                P 1 Reply Last reply Reply Quote 0
                • P Offline
                  pedro_udifar @pierrebrunet
                  last edited by

                  @pierrebrunet Hi!

                  Replication is from local NFS (Ztsd compressed, not encripted) to a S3 Backblaze (with encription).

                  P 1 Reply Last reply Reply Quote 0
                  • P Offline
                    pierrebrunet Vates 🪐 XO Team @pedro_udifar
                    last edited by

                    @pedro_udifar Hi,
                    We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream

                    This should fix the stream issue and fix the warning.

                    florentF 1 Reply Last reply Reply Quote 0
                    • florentF Offline
                      florent Vates 🪐 XO Team @pierrebrunet
                      last edited by

                      both fixes are now on master AND released as a patch

                      thank you @pedro_udifar

                      P 1 Reply Last reply Reply Quote 0
                      • P Offline
                        pedro_udifar @florent
                        last edited by

                        @florent

                        Sorry, missed your previous post. I will return to master and test it (over the weekend).

                        Thank you!

                        poddingueP 1 Reply Last reply Reply Quote 0
                        • poddingueP Online
                          poddingue Vates 🪐 @pedro_udifar
                          last edited by

                          Quick one to close this off: florent's fix for the encrypted-remote mirror alignment landed in a numbered release now, XO 6.6.2 (2026-07-09).
                          The PR was https://github.com/vatesfr/xen-orchestra/pull/10061, so it's out of master and into something you can just update to.
                          Did your retest come good on it?
                          I'm also curious whether you still need the minPartSize=100000000 setting with the fix in place, or whether that's redundant now, since it'd help us work out whether it's worth writing into the B2 docs.

                          P 2 Replies Last reply Reply Quote 0
                          • P Offline
                            pedro_udifar @poddingue
                            last edited by

                            @poddingue Hi,

                            The backup mirror job is still running, and I'm waiting for the larger VMs to complete, but so far it looks good - one backup, larger than 50GB was successfully mirrored without specifying minPartSize.

                            a5158a60-6607-4679-ac2d-ee9f84c02e4e-image.jpeg

                            1 Reply Last reply Reply Quote 0
                            • P Offline
                              pedro_udifar @poddingue
                              last edited by

                              @poddingue Hi!

                              Unfortunately, another error came up on the larger vms. The job also seems to be painfully slow. Slower than usual I mean.
                              Also, it seems that job changes are not being commited. I've changed the job so it wouldn't retry in case of failure and the log shows it still trying 3 times (that was the old setting, and probably explains the job taking longer than usual).

                              a917295e-ea92-4173-bc5a-7c9652a8d2ad-image.jpeg

                              {
  "id": "0mrjahr8e-ssawlbveq5",
  "start": 1783951215134,
  "status": "failure",
  "tasks": [
    {
      "id": "0mrjahs3e-dbfaqajeze",
      "start": 1783951216250,
      "status": "failure",
      "tasks": [
        {
          "id": "0mrjahs8s-2yb6i3fny6f",
          "start": 1783951216444,
          "status": "failure",
          "end": 1783959266484,
          "result": {
            "$fault": "server",
            "$metadata": {
              "httpStatusCode": 500,
              "requestId": "c0a9f30190109f99",
              "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
              "attempts": 3,
              "totalRetryDelay": 197
            },
            "name": "InternalError",
            "Code": "InternalError",
            "message": "internal incident",
            "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
          },
          "message": "transfer",
          "data": {
            "progress": 40
          }
        }
      ],
      "end": 1783959266485,
      "result": {
        "$fault": "server",
        "$metadata": {
          "httpStatusCode": 500,
          "requestId": "c0a9f30190109f99",
          "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
          "attempts": 3,
          "totalRetryDelay": 197
        },
        "name": "InternalError",
        "Code": "InternalError",
        "message": "internal incident",
        "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
      },
      "message": "export",
      "data": {
        "id": "14b59f44-4517-411e-9e39-4eb7136928d0",
        "type": "remote",
        "isFull": true
      }
    },
    {
      "id": "0mrjfabp3-c8mf2pnq8yf",
      "start": 1783959266487,
      "status": "success",
      "warnings": [
        {
          "data": {
            "path": "xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/cache.json.gz",
            "actual": 0,
            "expected": 1
          },
          "message": "unexpected number of entries in backup cache"
        }
      ],
      "end": 1783959266989,
      "result": {
        "merge": false,
        "size": 0
      },
      "message": "clean-vm"
    }
  ],
  "end": 1783959266989,
  "result": {
    "$fault": "server",
    "$metadata": {
      "httpStatusCode": 500,
      "requestId": "c0a9f30190109f99",
      "extendedRequestId": "aMJgyljkPZEdiEDW0Nk9lTmbINpI1FGIP",
      "attempts": 3,
      "totalRetryDelay": 197
    },
    "name": "InternalError",
    "Code": "InternalError",
    "message": "internal incident",
    "stack": "InternalError: internal incident\n    at ProtocolLib.getErrorSchemaOrThrowBaseException (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:70:67)\n    at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1856:65)\n    at S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:341:24)\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:25:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:367:20\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/checksums/dist-cjs/index.js:337:24\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:64:28\n    at async /opt/xo/xo-builds/xen-orchestra-202607120728/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/submodules/s3/index.js:91:20"
  },
  "message": "backup VM",
  "data": {
    "id": "60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee",
    "type": "VM",
    "progress": 0
  }
},
                              P 1 Reply Last reply Reply Quote 0
                              • P Offline
                                pierrebrunet Vates 🪐 XO Team @pedro_udifar
                                last edited by

                                @pedro_udifar Hi,
                                This is a b2 side error. It could be just a transient error, did you try to restart the backup?

                                P 3 Replies Last reply Reply Quote 0
                                • P Offline
                                  pedro_udifar @pierrebrunet
                                  last edited by

                                  This post is deleted!
                                  1 Reply Last reply Reply Quote 0
                                  • P Offline
                                    pedro_udifar @pierrebrunet
                                    last edited by

                                    This post is deleted!
                                    1 Reply Last reply Reply Quote 0
                                    • P Offline
                                      pedro_udifar @pierrebrunet
                                      last edited by

                                      @pierrebrunet Hello!

                                      I've checked B2 logs and found some errors like you had foreseen. I had a look and this kind of 500 INTERNAL ERROR several times on other jobs that seem ok. How many retries does XO attempt to send the part when this happens?

                                      I'll do another run that includes only a few large VMs. BTW, is there a way to interrupt the job withou restarting XO ?

                                      REST.PUT.PART Weekly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva "PUT /zzzzzkly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva?partNumber=3954&uploadId=4_ze092693d1b65962e9fe6051b_f25969add7e42e384_d20260713_m140016_c003_v0312040_t0013_u01783951216751&x-id=UploadPart HTTP/1.1" 500 INTERNAL_ERROR 145 - 3762 3762 "-" "aws-sdk-js/3.1065.0 ua/2.1 os/linux#5.15.0-185-generic lang/js md/nodejs#24.18.0 api/s3#3.1065.0 m/N,a,c,E,e xen-orchestra-fs-4.9.2" - - - TLS_AES_256_GCM_SHA384 AuthHeader s3.eu-central-003.backblazeb2.com TLSv1.3 - -

                                      P 1 Reply Last reply Reply Quote 0
                                      • P Offline
                                        pedro_udifar @pedro_udifar
                                        last edited by

                                        @pierrebrunet

                                        Large VMs still fail with the same error (internal incident), the process is not reliable - don't know if it's a B2 problem or XO.
                                        When I have some time, I'll do another run setting minPartSize=100000000 and see if it helps.

                                        P 1 Reply Last reply Reply Quote 0
                                        • P Offline
                                          pierrebrunet Vates 🪐 XO Team @pedro_udifar
                                          last edited by

                                          @pedro_udifar Hi,
                                          You can try again with minPartSize=100000000 indeed. But I would advise you to contact B2 directly. They should have a clearer error to show instead of a 500.
                                          Thanks for your patience and your help

                                          1 Reply Last reply Reply Quote 0

                                          Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                          Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                          With your input, this post could be even better 💗

                                          Register Login
                                          • First post
                                            Last post