Backup to S3 aborted what permissions are required?
-
@jensolsson-se that is weird, there is no real error in the logs you posted, and the testing function actually writes on the S3 bucket, so I really think a lot of things went right.
I am trying to think of a bug that would have gone around the error reporting system. The websocket message is a bit suspicious.
-
@nraynaud Is there anything I can try to narrow down the problem?
-
@nraynaud What does isFull: true mean? Could this be the issue?
Did my permissions look correct?
Also wonder if there are any other logs anywhere I could find, or if there is something maybe I could try from the console to narrow down the problem?
Kind regards
Jens -
-
isFull: truemeans that it's a full copy (because there were no previous valid delta chain).The issue is not related to this or to permissions, it appears that S3 protocol has constraints not really compatible with our use case (more info), we are investigating but it does not look great so far.
-
@julien-f Ah of course, If I would have thought about that isFull line a few minutes I would probably figure that out
Regarding S3 I think it sounds a bit strange. I read the linked article, but for what I understand there should not be a problem uploading a multipart object to S3 not knowing the total filesize. https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
Do you really need to sign the url? I think the credentials can be used to upload without signing the URL. Or is this used so that the xcp-ng servers are actually transferring directly to S3 ? I also found this article which implies that every part of the multipart upload could be signed individually if signed urls are preferred:
https://www.altostra.com/blog/multipart-uploads-with-s3-presigned-urlDon't know what API you are using with S3, but using the command line, if the source is a stream one could do something like
<command streaming to stdout> | aws s3 cp - s3://bucket/...I know that S3 needs an expected size of the upload. This is to calculate the number of parts so that it does not reach the limit of 5GB per part or too many parts, but setting the expected upload size to 5TB would probably work.
https://loige.co/aws-command-line-s3-content-from-stdin-or-to-stdout/Does this makes sense or did I fully misunderstood the problem?
Kind regards
Jens -
@jensolsson-se Hi Jens, you are on the right track. The other thing is that 5TB/10000 leads to big fragment size that are a bit much to keep in memory.
-
@nraynaud So memory requirements are ~500 MB and that is too much?
Is there a requirement that there need to be one single file in the destination or would it be possible to set the fragment size to 100 MB and if the file will be over 1 TB just create a new file? .0 .1 .2 .3 and so on ?
Maybe I am now braking some convention here but I guess it should work in a good way?
Also read that it is possible to copy objects into parts
https://aws.amazon.com/blogs/developer/efficient-amazon-s3-object-concatenation-using-the-aws-sdk-for-ruby/I am thinking if this would make it possible to make a multipart upload of 10 MB parts into one big 100 GB file
Then take 50 x 100 GB files and combine them into one big 5 TB file and as I understand it this can be done 100% within S3. No upload/download
Kind regards
Jens -
@jensolsson-se yes, we have though of complicated solutions too, but we haven't yet really dug into it, because this is a backup situation, we'd like the state of things and failure modes to be manageable.
-
@nraynaud Makes sense to keep it simple.
But this means that S3 backup in XO is currently broken, right, and I need to find some other way to back up my VMs for now.
-
@jensolsson-se Can you use SMB, NFS or local backups? I don't think S3 has ever worked for anyone.
-
@nraynaud yes i use nfs today. But would love to send it to the cloud somewhere as well.
