Error mirroring full backups to backblaze b2
-
This post is deleted! -
This post is deleted! -
@pierrebrunet Hello!
I've checked B2 logs and found some errors like you had foreseen. I had a look and this kind of 500 INTERNAL ERROR several times on other jobs that seem ok. How many retries does XO attempt to send the part when this happens?
I'll do another run that includes only a few large VMs. BTW, is there a way to interrupt the job withou restarting XO ?
REST.PUT.PART Weekly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva "PUT /zzzzzkly/xo-vm-backups/60b4d2a7-3f91-f0f9-b0a0-f834df3ff9ee/20260705T070618Z.xva?partNumber=3954&uploadId=4_ze092693d1b65962e9fe6051b_f25969add7e42e384_d20260713_m140016_c003_v0312040_t0013_u01783951216751&x-id=UploadPart HTTP/1.1" 500 INTERNAL_ERROR 145 - 3762 3762 "-" "aws-sdk-js/3.1065.0 ua/2.1 os/linux#5.15.0-185-generic lang/js md/nodejs#24.18.0 api/s3#3.1065.0 m/N,a,c,E,e xen-orchestra-fs-4.9.2" - - - TLS_AES_256_GCM_SHA384 AuthHeader s3.eu-central-003.backblazeb2.com TLSv1.3 - -
-
Large VMs still fail with the same error (internal incident), the process is not reliable - don't know if it's a B2 problem or XO.
When I have some time, I'll do another run setting minPartSize=100000000 and see if it helps. -
@pedro_udifar Hi,
You can try again with minPartSize=100000000 indeed. But I would advise you to contact B2 directly. They should have a clearer error to show instead of a 500.
Thanks for your patience and your help -
@pierrebrunet Hi!
I'm on vacation now, so sorry for the delay. In the meantime I've been analyzing these issues and it seems these are somewhat expected and still happen with minpartsize set to 100M.
My next step is to increase the AWS_MAX_ATTEMPTS from the default 3 attempts to 10. This might reduce the number of failed VMs and avoid the need to retry the full VM backup.
From AI research, it seems rclone uses 10 attempts instead of the AWS SDK default.
Here are some stats of my weekly full backup to B2:
"00 Weekly Full Backblaze" ran xxxx → xxxxx — 34.1 hours total, moving ~2.86 TiB across 89 VMs. Final status: failure (more on that below, because it's misleading) -> this is related to XO reporting the backup as a failure even though the follow up retry succeded which is a known issue.Every one of the 7 first-attempt errors shows attempts: 3 in the SDK metadata — the default budget was exhausted in all cases -> 7 of 89 VMs failed on the initial attempt.
Error mix: 5× Backblaze InternalError 500, 1× ECONNRESET, 1× EPIPE
The 7 failed first attempts burned ~7.1 hours of cumulative transfer time
Even though XO does not expose AWS_MAX_ATTEMPTS parameter I might be able to set it using an environment variable. I'll give a try when I'm back at the office.
Tks,
Pedro -
P poddingue marked this topic as a question
-
Hello,
The initial issue, non encrypred to encrypted mirroring is now working.
The second issue was about general errors when sending backups to S3. I've tried and was able to increase AWS_MAX_ATTEMPTS to 10 but I'm still having errors and VMs that fail the first backup and have to be retried. So these do not seem to be transient errors that can be fixed by increasing AWS_MAX_ATTEMPTS.
Thank you,
Pedro -
@pedro_udifar Hi,
Do you still have errors 500 when you are talking about general errors? -
This week I only got 500 errors affecting 5 VMs. These 5 VMs were retried at VM-level and one fo these failed again (it failed to backup since I only have set 1 retry).
I'm now looking into connection and bandwitdh to B2.
I'm only reaching 200 Mbps to B2 with my current ISP so I'll tune concurrency so I reach those 200 Mbps with fewer VMs in parallel, making each single VM transfer faster.
Concurrency was set to 8, I'll try with 3 VMs in parallel this weekend.Tks,
Pedro -
Hello,
To wrap up this last issue: the 500 errors backing up to S3 B2 are network contention related. There is some kind of contention with my link to B2 and having 8 VMs backing up simultaneously sometimes triggers these errors. Last weekend I reduced concurrency to 3 and every VM was successfully backed up - no VM level retries. I can't tell if some file parts were retried since these are only logged when the AWS_MAX_ATTEMPS are exhausted and the VM backup fails.
So, these are not B2 issues and the default AWS_MAX_ATTEMPS is probably fine for most scenarios. I'll leave my at 10 following rclone defaults.Thank you,
Pedro -
P pedro has marked this topic as solved
-
Thanks a lot for this feedback, @pedro!
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login