S3 Backup - maximum number of parts

deligatedgeek

Finally got round to following up on this.

I found a better solution using DEFAULT_CHUNK_SIZE, which is normally set to 4MB, with MAX_PART_NUMBER being 1000, this limits the largest object backed up to 3rd party S3 implementations as 4GB.

Increasing the DEFAULT_CHUNK_SIZE in the S3 code, or having a setting in configuration, either file or web GUI would allow backups of VMs larger than 4GB, without the memory required to track several thousand object chunks.

Regards,

Mark

olivierlambert

That's very interesting, let me make sure @florent got this

florent

@deligatedgeek are you modifying the chunk for delta or full backup ?
because a lot of part in the incremental code assume that blocks are aligned to 2MB

For full backup ( one big xva file per VM) the code should handle file with a size bigger than 4GB , by dynamically increasing the chunk size. We do not increase the chunk size by default because it also increase a lot the memory consumption. The minimum size of a chunk is given by MIN_PART_SIZE ( 5MB as per AWS doc ) and the maximum size is MAX_PART_SIZE ( 5GB) . The maximum file is 50TB , the biggest full backup I saw in the wild was a few hundred GB

deligatedgeek

@florent The actual problem we had to solve was that AWS S3 allows 10000 chunks, whereas other implementations by default only allow 1000, so s3 library calculates small chunk size.
We tried configuring swift proxy multipart and swift S3 to 10000, but it causes other problems.
We also ran into the same issue using Mender with Minio as a backend failure with files over 10G.

Understanding this more now and the solution could simply be to specify someone a maximum number of chunks, which would be 10000 by default by could be set to 1000.

We settled on our solution because of this code in the s3 js files

chunk_size = max(MIN_PART_SIZE, ceil(file_size / 10000))

It was easier to set MIN_PART_SIZE, than modify the code to have a parameter instead of 10000. And we patch the code after each XO update.

Either solution would work for us, thanks for following up on this.

florent

@deligatedgeek would it be ok to do a quick iteration where you need to use the command line to add some magics to your backup repository settings ?

first you will need to change your branch to feat_s3_part_settings ( if you are using xo from source)
Then use xo-cli to update the settings of your remote :

xo-cli remote.getAll

find the right one, then xo-cli remote.set id=<the uuid you get at the previous step> url='<the previous url >&maxPartNumber=1000

you can also add maxPartSize and minPartSize if needed

florent

I insist on the fact that this will only have impact on the FULL backups jobs , and not on the incremental ones

deligatedgeek

@florent I have tested in our dev environment backing up a 30GB VM, which compressed into 22GB and was uploaded in 739 parts of 40MB each.

I will test a larger VM next week but I think this was successful test is already conclusive.

Thank You

florent

@deligatedgeek we will try to push this branch to master before the end of the month
(we will have a lot of documentation to write on what is solves and what it don't solves)

deligatedgeek

@florent That would be great. Thanks again!

poddingue

Late follow-up on this one. The branch Florent mentioned did land: maxPartNumber, maxPartSize and minPartSize shipped in XO 6.3.0 on 2026-03-31 (https://github.com/vatesfr/xen-orchestra/pull/9561), so the xo-cli remote.set recipe from his post above should work on a released build now rather than only on feat_s3_part_settings.

His next post is the one worth re-reading, because the limitation is easy to lose in a long thread: it only changes full backup jobs, not incremental ones.
One thing I went and checked because I wasn't sure myself, there's still no field for these in the web UI, so putting them on the remote URL is genuinely the way to set them rather than a leftover from the branch.