Error mirroring full backups to backblaze b2

[florent]

@pedro_udifar object lock works quite well with xo
it is automatically detected if the s3 user has enough privilege to see if object lock is enabled

the main point is that it won't create "cache.json.gz" files since theses files must be updated, so the backup listing is a little slower

note that XO retention should be at least 1 more than object lock . At worst you will have error , saying that XO coudn't delete an old backup

The fix is on a separated branch : fix_s3_nonencrypted_to_encrypted_xva ( PR : https://github.com/vatesfr/xen-orchestra/pull/10061 )

fbeauchamp opened this pull request in vatesfr/xen-orchestra

open fix(fs): compute the output stream precisely when encrypted #10061

[pedro_udifar]

@florent said:

fix_s3_nonencrypted_to_encrypted_xva

Hello Florent,

Branch built and testing
Test procedure: removed the minPartSize=100000000 from remote definition, deleted the b2 backup and restarted the mirror backup. There are some other VMs that will also be uploaded to B2 so it will take a while. Expect feedback tomorrow.

Thank you,
Pedro

[pedro_udifar]

Still getting these warnings, on the VMs that don't need to be uploaded. Thought this might go away also.

[pedro_udifar]

Hello Florent,

The new code is still failing...

	```
	{
		"id": "0mr3li2kt-c10ifxuuw68",
		"start": 1783002246797,
		"status": "failure",
		"tasks": [
			{
				"id": "0mr3li2ud-r1vgn4rnv5d",
				"start": 1783002247141,
				"status": "failure",
				"tasks": [
					{
						"id": "0mr3li30l-6n1cr5a6o7v",
						"start": 1783002247365,
						"status": "failure",
						"end": 1783004124397,
						"result": {
							"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
							"name": "Error",
							"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
						},
						"message": "transfer",
						"data": {
							"progress": 0
						}
					}
				],
				"end": 1783004124397,
				"result": {
					"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
					"name": "Error",
					"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
				},
				"message": "export",
				"data": {
					"id": "14b59f44-4517-411e-9e39-4eb7136928d0",
					"type": "remote",
					"isFull": true
				}
			},
			{
				"id": "0mr3mmbcf-yc4lpd0z5p",
				"start": 1783004124399,
				"status": "success",
				"warnings": [
					{
						"data": {
							"path": "xo-vm-backups/1c842f34-f7f8-d3cd-ba24-ffe51d6ae075/cache.json.gz",
							"actual": 0,
							"expected": 1
						},
						"message": "unexpected number of entries in backup cache"
					},
					{
						"data": {
							"error": {
								"generatedMessage": false,
								"code": "ERR_ASSERTION",
								"actual": true,
								"expected": false,
								"operator": "strictEqual",
								"diff": "simple"
							}
						},
						"message": "Issue while checking XVA"
					},
					{
						"data": {
							"error": {
								"generatedMessage": false,
								"code": "ERR_ASSERTION",
								"actual": true,
								"expected": false,
								"operator": "strictEqual",
								"diff": "simple"
							}
						},
						"message": "Checksum file not valid, not blocking"
					},
					{
						"data": {
							"path": "/xo-vm-backups/1c842f34-f7f8-d3cd-ba24-ffe51d6ae075/20260624T231818Z.xva"
						},
						"message": "XVA might be broken"
					},
					{
						"data": {
							"files": [
								"20260624T231818Z.json",
								"20260624T231818Z.xva",
								"20260624T231818Z.xva.checksum"
							]
						},
						"message": "This files may be corrupted but not yet to be removed"
					}
				],
				"end": 1783004124983,
				"result": {
					"merge": false,
					"size": 0
				},
				"message": "clean-vm"
			}
		],
		"end": 1783004124983,
		"result": {
			"message": "read 52428800012 bytes, maximum size allowed  is 52428800000 ",
			"name": "Error",
			"stack": "Error: read 52428800012 bytes, maximum size allowed  is 52428800000 \n    at Transform.transform [as _transform] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/fs/dist/s3.js:214:20)\n    at Transform._write (node:internal/streams/transform:171:8)\n    at writeOrBuffer (node:internal/streams/writable:570:12)\n    at _write (node:internal/streams/writable:499:10)\n    at Writable.write (node:internal/streams/writable:508:10)\n    at PassThrough.ondata (node:internal/streams/readable:1012:24)\n    at PassThrough.emit (node:events:509:28)\n    at PassThrough.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202607021439/@xen-orchestra/log/configure.js:52:17)\n    at addChunk (node:internal/streams/readable:563:12)\n    at readableAddChunkPushObjectMode (node:internal/streams/readable:540:3)"
		},
		"message": "backup VM",
		"data": {
			"id": "1c842f34-f7f8-d3cd-ba24-ffe51d6ae075",
			"type": "VM",
			"progress": 0
		}
	},

Best regards,
Pedro

[florent]

@pedro_udifar that is strange that we have exactly the same error, even though we should have added 28 bytes

I pushed a little debug that will log additional lines in journalctl

'will output stream ', 'stream is encrypted' and 'stream is updated '

Can you update your branch, rebuild, rerun the transfer and post the log ( with the attached values ) ?

regards

[pedro_udifar]

@florent

Sure, but I can only do it over the weekend.
Also, I'll try to check if the script built the correct branch. First time using the script to build anything other than master.

On a side note: recently some other strange errors started to show up "Body timeout Error". These are plain full backups to a remote with disaster recovery for another cluster, that always worked fine. Also, in this case, the timeout defined within the job doesn't cancel the job and the remaining jobs in the sequence don't run. Haven't had the time to dig into this - when I do, I'll start a new forum post.

Cheers,
Pedro

[pedro_udifar]

Hello again!

Here are the logs:
b2_mirror_error.txt

Hope it helps!

[pierrebrunet]

@pedro_udifar Hi, just to be sure, this is a replication job with two s3? If yes, is the source s3 encrypted or is it only the target?

[pedro_udifar]

@pierrebrunet Hi!

Replication is from local NFS (Ztsd compressed, not encripted) to a S3 Backblaze (with encription).

[pierrebrunet]

@pedro_udifar Hi,
We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream

This should fix the stream issue and fix the warning.