XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup to S3 aborted what permissions are required?

    Xen Orchestra
    backup aws
    4
    24
    2253
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jensolsson.se @nraynaud
      last edited by

      @nraynaud What does isFull: true mean? Could this be the issue?

      Did my permissions look correct?

      Also wonder if there are any other logs anywhere I could find, or if there is something maybe I could try from the console to narrow down the problem?

      Kind regards
      Jens

      1 Reply Last reply Reply Quote 0
      • olivierlambertO
        olivierlambert Vates ๐Ÿช Co-Founder๐Ÿฆธ CEO ๐Ÿง‘โ€๐Ÿ’ผ
        last edited by

        I think @nraynaud and @julien-f are trying another library than original S3 SDK from AWS because it's problematic.

        If I understand correctly ๐Ÿ™‚

        julien-fJ 1 Reply Last reply Reply Quote 0
        • julien-fJ
          julien-f Vates ๐Ÿช Co-Founder๐Ÿฆธ XO Team ๐Ÿ”ญ @olivierlambert
          last edited by julien-f

          isFull: true means that it's a full copy (because there were no previous valid delta chain).

          The issue is not related to this or to permissions, it appears that S3 protocol has constraints not really compatible with our use case (more info), we are investigating but it does not look great so far.

          J 1 Reply Last reply Reply Quote 0
          • J
            jensolsson.se @julien-f
            last edited by

            @julien-f Ah of course, If I would have thought about that isFull line a few minutes I would probably figure that out ๐Ÿ™‚

            Regarding S3 I think it sounds a bit strange. I read the linked article, but for what I understand there should not be a problem uploading a multipart object to S3 not knowing the total filesize. https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html

            Do you really need to sign the url? I think the credentials can be used to upload without signing the URL. Or is this used so that the xcp-ng servers are actually transferring directly to S3 ? I also found this article which implies that every part of the multipart upload could be signed individually if signed urls are preferred:
            https://www.altostra.com/blog/multipart-uploads-with-s3-presigned-url

            Don't know what API you are using with S3, but using the command line, if the source is a stream one could do something like
            <command streaming to stdout> | aws s3 cp - s3://bucket/...

            I know that S3 needs an expected size of the upload. This is to calculate the number of parts so that it does not reach the limit of 5GB per part or too many parts, but setting the expected upload size to 5TB would probably work.
            https://loige.co/aws-command-line-s3-content-from-stdin-or-to-stdout/

            Does this makes sense or did I fully misunderstood the problem?

            Kind regards
            Jens

            nraynaudN 1 Reply Last reply Reply Quote 0
            • nraynaudN
              nraynaud XCP-ng Team ๐Ÿš€ @jensolsson.se
              last edited by

              @jensolsson-se Hi Jens, you are on the right track. The other thing is that 5TB/10000 leads to big fragment size that are a bit much to keep in memory.

              J 1 Reply Last reply Reply Quote 0
              • J
                jensolsson.se @nraynaud
                last edited by jensolsson.se

                @nraynaud So memory requirements are ~500 MB and that is too much?

                Is there a requirement that there need to be one single file in the destination or would it be possible to set the fragment size to 100 MB and if the file will be over 1 TB just create a new file? .0 .1 .2 .3 and so on ?

                Maybe I am now braking some convention here but I guess it should work in a good way?

                Also read that it is possible to copy objects into parts
                https://aws.amazon.com/blogs/developer/efficient-amazon-s3-object-concatenation-using-the-aws-sdk-for-ruby/

                I am thinking if this would make it possible to make a multipart upload of 10 MB parts into one big 100 GB file

                Then take 50 x 100 GB files and combine them into one big 5 TB file and as I understand it this can be done 100% within S3. No upload/download

                Kind regards
                Jens

                nraynaudN 1 Reply Last reply Reply Quote 0
                • nraynaudN
                  nraynaud XCP-ng Team ๐Ÿš€ @jensolsson.se
                  last edited by

                  @jensolsson-se yes, we have though of complicated solutions too, but we haven't yet really dug into it, because this is a backup situation, we'd like the state of things and failure modes to be manageable.

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    jensolsson.se @nraynaud
                    last edited by jensolsson.se

                    @nraynaud Makes sense to keep it simple.

                    But this means that S3 backup in XO is currently broken, right, and I need to find some other way to back up my VMs for now.

                    nraynaudN 1 Reply Last reply Reply Quote 0
                    • nraynaudN
                      nraynaud XCP-ng Team ๐Ÿš€ @jensolsson.se
                      last edited by

                      @jensolsson-se Can you use SMB, NFS or local backups? I don't think S3 has ever worked for anyone.

                      J 1 Reply Last reply Reply Quote 0
                      • J
                        jensolsson.se @nraynaud
                        last edited by

                        @nraynaud yes i use nfs today. But would love to send it to the cloud somewhere as well.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post