Error mirroring full backups to backblaze b2

pedro_udifar

Hello forum!

I recently begun mirroring my full backups to backblaze and I've been getting the error "maximum size allowed is (your minPartSize for the chunk)"

This error was supposed to be fixed, earlier this year, but I'm still getting it.
Actually PR #9396 might only fix direct backups and not mirror backups...
I'm running xo built last week. Object lock is set to 0 at backblaze for the time being, but I plan to turn it whenr this problem is over.

This may be caused because I'm mirroring from a local nfs remote (not encripted) to backcblaze remote (encripted) ? Maybe this scenario wasn't tested.

Anyway, everytime I run the mirror job the larger VMs would fail and all VMs would show warnings about issues checking XVA:

I've just bumped minPartSize to 100 000 000 (following the recomendation from Backblaze and hope that the maximum size allowed error goes away, though the warnings will still be there.

Thank you,
Pedro

poddingue

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(backups): size estimation of full backups #9396

florent

@poddingue said:

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

nice catch
and the mirror path is easier to fix . Thaks @pedro_udifar and @poddingue we will fix it asap

will it be possible to test a potential fix @pedro_udifar , since it's often specific to a provider ( and need huge VM to work )?

florent

now that I am in front of the source code , it is already getting the real size, so it should compute the right size

are the source or destination encrypted ?

florent

@pedro_udifar we have found a clue , if the destination is encrypted and the source is not

can you test this branch ? https://github.com/vatesfr/xen-orchestra/pull/10061
if you are using a xoa, we can deploy the fix if you open a support tunnel

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(fs): compute the output stream precisely when encrypted #10061

pedro_udifar

Hello Poddingue and Florent,

I started duplicating "default" minPartSize but the larger VMs we're still failing to mirror. After that, I found that Bacblaze has a recommended chunk size of minPartSize=100000000 and, with this setting, all my VMs completed without that specific error.
(My largest VM compressed backup is about 450 GB)

https://www.backblaze.com/docs/cloud-storage-large-files
In my case, the recommendPartSize returned was 100000000.

Since I was also getting the other warnings "Issue checking XVA" and "XVA might be broken", most o f the times I would wipe de backblaze bucket and start with a new job no XO (so that I could be sure it was not about the retention settings of this mirror job).

Replication source is on a local NFS Ztsd compressed, not encripted.
Replication destinations is Backblaze with encription. Object lock is turned off (XO can delete and change the files without restrictions)

I'm still getting the XVA warnings and my next step was to run a test turning encription on, on the local 1st tier nfs remote.

The sizes reported locally and on backblaze have very different sizes, which I guess is to be expected on this scenario.

Local:

Backblaze:

In my planned backup pipeline, I am planning to turn on object lock on backblaze, making sure it's aligned with mirror retention settings on XO thoug I'm not sure this will work nice with metadata json files. What do you guys think?

I'll gladly test the fix when it's ready. Is already commited?

Thank you for your help,
Pedro

florent

@pedro_udifar object lock works quite well with xo
it is automatically detected if the s3 user has enough privilege to see if object lock is enabled

the main point is that it won't create "cache.json.gz" files since theses files must be updated, so the backup listing is a little slower

note that XO retention should be at least 1 more than object lock . At worst you will have error , saying that XO coudn't delete an old backup

The fix is on a separated branch : fix_s3_nonencrypted_to_encrypted_xva ( PR : https://github.com/vatesfr/xen-orchestra/pull/10061 )

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(fs): compute the output stream precisely when encrypted #10061

pedro_udifar

@florent said:

fix_s3_nonencrypted_to_encrypted_xva

Hello Florent,

Branch built and testing
Test procedure: removed the minPartSize=100000000 from remote definition, deleted the b2 backup and restarted the mirror backup. There are some other VMs that will also be uploaded to B2 so it will take a while. Expect feedback tomorrow.

Thank you,
Pedro

pedro_udifar

Still getting these warnings, on the VMs that don't need to be uploaded. Thought this might go away also.

pedro_udifar

Hello Florent,

The new code is still failing...

	```
	{
		"id": "0mr3li2kt-c10ifxuuw68",
		"start": 1783002246797,
		"status": "failure",
		"tasks": [
			{
				"id": "0mr3li2ud-r1vgn4rnv5d",
				"start": 1783002247141,
				"status": "failure",
				"tasks": [
					{
						"id": "0mr3li30l-6n1cr5a6o7v",
						"start": 1783002247365,
						"status": "failure",
						"end": 1783004124397,
						"result": {
							"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
							"name": "Error",
							"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
						},
						"message": "transfer",
						"data": {
							"progress": 0
						}
					}
				],
				"end": 1783004124397,
				"result": {
					"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
					"name": "Error",
					"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
				},
				"message": "export",
				"data": {
					"id": "14b59f44-4517-411e-9e39-4eb7136928d0",
					"type": "remote",
					"isFull": true
				}
			},
			{
				"id": "0mr3mmbcf-yc4lpd0z5p",
				"start": 1783004124399,
				"status": "success",
				"warnings": [
					{
						"data": {
							"path": "xo-vm-backups/1c842f34-f7f8-d3cd-ba24-ffe51d6ae075/cache.json.gz",
							"actual": 0,
							"expected": 1
						},
						"message": "unexpected number of entries in backup cache"
					},
					{
						"data": {
							"error": {
								"generatedMessage": false,
								"code": "ERR_ASSERTION",
								"actual": true,
								"expected": false,
								"operator": "strictEqual",
								"diff": "simple"
							}
						},
						"message": "Issue while checking XVA"
					},
					{
						"data": {
							"error": {
								"generatedMessage": false,
								"code": "ERR_ASSERTION",
								"actual": true,
								"expected": false,
								"operator": "strictEqual",
								"diff": "simple"
							}
						},
						"message": "Checksum file not valid, not blocking"
					},
					{
						"data": {
							"path": "/xo-vm-backups/1c842f34-f7f8-d3cd-ba24-ffe51d6ae075/20260624T231818Z.xva"
						},
						"message": "XVA might be broken"
					},
					{
						"data": {
							"files": [
								"20260624T231818Z.json",
								"20260624T231818Z.xva",
								"20260624T231818Z.xva.checksum"
							]
						},
						"message": "This files may be corrupted but not yet to be removed"
					}
				],
				"end": 1783004124983,
				"result": {
					"merge": false,
					"size": 0
				},
				"message": "clean-vm"
			}
		],
		"end": 1783004124983,
		"result": {
			"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
			"name": "Error",
			"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
		},
		"message": "backup VM",
		"data": {
			"id": "1c842f34-f7f8-d3cd-ba24-ffe51d6ae075",
			"type": "VM",
			"progress": 0
		}
	},

Best regards,
Pedro

florent

@pedro_udifar that is strange that we have exactly the same error, even though we should have added 28 bytes

I pushed a little debug that will log additional lines in journalctl

'will output stream ', 'stream is encrypted' and 'stream is updated '

Can you update your branch, rebuild, rerun the transfer and post the log ( with the attached values ) ?

regards

pedro_udifar

@florent

Sure, but I can only do it over the weekend.
Also, I'll try to check if the script built the correct branch. First time using the script to build anything other than master.

On a side note: recently some other strange errors started to show up "Body timeout Error". These are plain full backups to a remote with disaster recovery for another cluster, that always worked fine. Also, in this case, the timeout defined within the job doesn't cancel the job and the remaining jobs in the sequence don't run. Haven't had the time to dig into this - when I do, I'll start a new forum post.

Cheers,
Pedro

pedro_udifar

Hello again!

Here are the logs:
b2_mirror_error.txt

Hope it helps!

pierrebrunet

@pedro_udifar Hi, just to be sure, this is a replication job with two s3? If yes, is the source s3 encrypted or is it only the target?

pedro_udifar

@pierrebrunet Hi!

Replication is from local NFS (Ztsd compressed, not encripted) to a S3 Backblaze (with encription).

pierrebrunet

@pedro_udifar Hi,
We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream

This should fix the stream issue and fix the warning.

florent

both fixes are now on master AND released as a patch

thank you @pedro_udifar

pedro_udifar

@florent

Sorry, missed your previous post. I will return to master and test it (over the weekend).

Thank you!

poddingue

Quick one to close this off: florent's fix for the encrypted-remote mirror alignment landed in a numbered release now, XO 6.6.2 (2026-07-09).
The PR was https://github.com/vatesfr/xen-orchestra/pull/10061, so it's out of master and into something you can just update to.
Did your retest come good on it?
I'm also curious whether you still need the minPartSize=100000000 setting with the fix in place, or whether that's redundant now, since it'd help us work out whether it's worth writing into the B2 docs.

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(fs): compute the output stream precisely when encrypted #10061

pedro_udifar

@poddingue Hi,

The backup mirror job is still running, and I'm waiting for the larger VMs to complete, but so far it looks good - one backup, larger than 50GB was successfully mirrored without specifying minPartSize.