<![CDATA[S3 Backup - maximum number of parts]]>We use S3 swift proxy for swift storage and have built some infrastructure around it.

When we tried S3 for backup we got this error
xo-server[144792]: error: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive

Swift defaults to max number of parts of 1000, as do several commercial S3 implementations, and we have tried increasing but it caused instability in one of applications built around swift SLO.

The solution I've implemented is to reduce the MAX_PART_NUMBER in
@xen-orchestra/fs/dist/s3.js to 1000, but this be overwritten on each upgrade.

Could an option be added into the AWS S3 parameters to set this? or even just being able to override the value in a configuration file somewhere.

]]>https://xcp-ng.org/forum/topic/10587/s3-backup-maximum-number-of-partsRSS for NodeTue, 12 May 2026 18:13:18 GMTFri, 07 Mar 2025 16:51:29 GMT60<![CDATA[Reply to S3 Backup - maximum number of parts on Mon, 09 Mar 2026 16:24:13 GMT]]>@florent That would be great. Thanks again!

]]>https://xcp-ng.org/forum/post/103294https://xcp-ng.org/forum/post/103294Mon, 09 Mar 2026 16:24:13 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Mon, 09 Mar 2026 16:12:19 GMT]]>@deligatedgeek we will try to push this branch to master before the end of the month
(we will have a lot of documentation to write on what is solves and what it don't solves)

]]>https://xcp-ng.org/forum/post/103292https://xcp-ng.org/forum/post/103292Mon, 09 Mar 2026 16:12:19 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Fri, 06 Mar 2026 17:45:01 GMT]]>@florent I have tested in our dev environment backing up a 30GB VM, which compressed into 22GB and was uploaded in 739 parts of 40MB each.

I will test a larger VM next week but I think this was successful test is already conclusive.

Thank You

]]>https://xcp-ng.org/forum/post/103264https://xcp-ng.org/forum/post/103264Fri, 06 Mar 2026 17:45:01 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Thu, 05 Mar 2026 08:37:29 GMT]]>I insist on the fact that this will only have impact on the FULL backups jobs , and not on the incremental ones

]]>https://xcp-ng.org/forum/post/103206https://xcp-ng.org/forum/post/103206Thu, 05 Mar 2026 08:37:29 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Wed, 04 Mar 2026 16:40:19 GMT]]>@deligatedgeek would it be ok to do a quick iteration where you need to use the command line to add some magics to your backup repository settings ?

first you will need to change your branch to feat_s3_part_settings ( if you are using xo from source)
Then use xo-cli to update the settings of your remote :

xo-cli remote.getAll

find the right one, then xo-cli remote.set id=<the uuid you get at the previous step> url='<the previous url >&maxPartNumber=1000

you can also add maxPartSize and minPartSize if needed

]]>https://xcp-ng.org/forum/post/103185https://xcp-ng.org/forum/post/103185Wed, 04 Mar 2026 16:40:19 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Tue, 03 Mar 2026 20:03:21 GMT]]>@florent The actual problem we had to solve was that AWS S3 allows 10000 chunks, whereas other implementations by default only allow 1000, so s3 library calculates small chunk size.
We tried configuring swift proxy multipart and swift S3 to 10000, but it causes other problems.
We also ran into the same issue using Mender with Minio as a backend failure with files over 10G.

Understanding this more now and the solution could simply be to specify someone a maximum number of chunks, which would be 10000 by default by could be set to 1000.

We settled on our solution because of this code in the s3 js files

chunk_size = max(MIN_PART_SIZE, ceil(file_size / 10000))

It was easier to set MIN_PART_SIZE, than modify the code to have a parameter instead of 10000. And we patch the code after each XO update.

Either solution would work for us, thanks for following up on this.

]]>https://xcp-ng.org/forum/post/103147https://xcp-ng.org/forum/post/103147Tue, 03 Mar 2026 20:03:21 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Tue, 03 Mar 2026 09:10:24 GMT]]>@deligatedgeek are you modifying the chunk for delta or full backup ?
because a lot of part in the incremental code assume that blocks are aligned to 2MB

For full backup ( one big xva file per VM) the code should handle file with a size bigger than 4GB , by dynamically increasing the chunk size. We do not increase the chunk size by default because it also increase a lot the memory consumption. The minimum size of a chunk is given by MIN_PART_SIZE ( 5MB as per AWS doc ) and the maximum size is MAX_PART_SIZE ( 5GB) . The maximum file is 50TB , the biggest full backup I saw in the wild was a few hundred GB

]]>https://xcp-ng.org/forum/post/103119https://xcp-ng.org/forum/post/103119Tue, 03 Mar 2026 09:10:24 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Sat, 28 Feb 2026 20:33:51 GMT]]>That's very interesting, let me make sure @florent got this

]]>https://xcp-ng.org/forum/post/103084https://xcp-ng.org/forum/post/103084Sat, 28 Feb 2026 20:33:51 GMT<![CDATA[Reply to S3 Backup - maximum number of parts on Fri, 27 Feb 2026 19:17:20 GMT]]>Hi @olivierlambert,

Finally got round to following up on this.

I found a better solution using DEFAULT_CHUNK_SIZE, which is normally set to 4MB, with MAX_PART_NUMBER being 1000, this limits the largest object backed up to 3rd party S3 implementations as 4GB.

Increasing the DEFAULT_CHUNK_SIZE in the S3 code, or having a setting in configuration, either file or web GUI would allow backups of VMs larger than 4GB, without the memory required to track several thousand object chunks.

Regards,

Mark

]]>https://xcp-ng.org/forum/post/103068https://xcp-ng.org/forum/post/103068Fri, 27 Feb 2026 19:17:20 GMT