Testing S3 Remotes (I can sort of break it!)
-
Apologies if some of this doesn't make sense, as I have been running on very little sleep.
I have been testing the S3 remote for use in conjunction with delta backups. I have a host running Minio on TrueNAS to expose an S3 compatible bucket. It appears to die in a certain case. I have several single disk VMs which back up to it consistently with zero failures, but I have one other on the same host and storage, with the only real difference between those successful backed up is the failure VM has two disks. It is failing 100% of the time. Each time I run it, the backup job creates the snapshot with no issues, then when it reaches the transfer phase, disk 0 shows up in the task list with 4% progress right away, but disk 1 hangs at 0%. This goes for about 8 minutes and times out, giving me "Error Calling AWS.S3.upload: http connection timed out" in the logs. If I test the remote immediately after the job fails, it tests OK. I think the two disks uploading at the same time breaks the connection to the remote for some reason. I am curious to try this with a real AWS S3 bucket, and also a backblaze B2, but I haven't tried that yet.
TL;DR I suppose this testing makes my findings a bit of a feature request: Is there a way to set backups to serialize tasks instead of running what appears to be two in parallel? I think that would solve my issue found in testing, and I have had other use cases for backup jobs running tasks one at a time, such as a low bandwidth connection in order to increase the chance of success. I suppose it could go the other way as well in some edge cases: Perhaps a very high-performance setup may wish to do eight tasks in parallel in order to complete backup jobs as quickly as possible.
I just thought I would share my experience and am interested to hear others' experiences with the S3 remotes beta.
-
-
By the way, this testing was on xo-server 5.80.0 and xo-web 5.84.0. My initial tests were on earlier versions.
I just tried on Backblaze B2 as well, and I am coming up with errors while testing the remote.
Error Error calling AWS.S3.putObject: Non-whitespace before first tag. Line: 0 Column: 1 Char: B Test step writeInitially I had an issue creating the remote, because Backblaze allows uppercase characters in bucket names, and XO would not allow that input.
-
@schmalztech it will not be working correctly yer for very long time
i was trying to setup backups on backblaze in past year, didnt worked, I think s3 backup function is not production ready - but despite of this it was putted on xoa like it would be functioning correctly... -
@akurzawa Well, it does say beta. I was trying different use cases and posting my results in case it's being actively worked on.
It does work very reliably on my minio instance on my NAS so long as it's doing only one disk.
-
Reping @nraynaud
