S3 Backup - maximum number of parts

[deligatedgeek]

We use S3 swift proxy for swift storage and have built some infrastructure around it.

When we tried S3 for backup we got this error
xo-server[144792]: error: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive

Swift defaults to max number of parts of 1000, as do several commercial S3 implementations, and we have tried increasing but it caused instability in one of applications built around swift SLO.

The solution I've implemented is to reduce the MAX_PART_NUMBER in
@xen-orchestra/fs/dist/s3.js to 1000, but this be overwritten on each upgrade.

Could an option be added into the AWS S3 parameters to set this? or even just being able to override the value in a configuration file somewhere.

[deligatedgeek]

Hi @olivierlambert,

Finally got round to following up on this.

I found a better solution using DEFAULT_CHUNK_SIZE, which is normally set to 4MB, with MAX_PART_NUMBER being 1000, this limits the largest object backed up to 3rd party S3 implementations as 4GB.

Increasing the DEFAULT_CHUNK_SIZE in the S3 code, or having a setting in configuration, either file or web GUI would allow backups of VMs larger than 4GB, without the memory required to track several thousand object chunks.

Regards,

Mark