XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Testing S3 Remotes (I can sort of break it!)

    Scheduled Pinned Locked Moved Xen Orchestra
    6 Posts 3 Posters 727 Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • SchmalzTechS Offline
      SchmalzTech
      last edited by

      Apologies if some of this doesn't make sense, as I have been running on very little sleep.

      I have been testing the S3 remote for use in conjunction with delta backups. I have a host running Minio on TrueNAS to expose an S3 compatible bucket. It appears to die in a certain case. I have several single disk VMs which back up to it consistently with zero failures, but I have one other on the same host and storage, with the only real difference between those successful backed up is the failure VM has two disks. It is failing 100% of the time. Each time I run it, the backup job creates the snapshot with no issues, then when it reaches the transfer phase, disk 0 shows up in the task list with 4% progress right away, but disk 1 hangs at 0%. This goes for about 8 minutes and times out, giving me "Error Calling AWS.S3.upload: http connection timed out" in the logs. If I test the remote immediately after the job fails, it tests OK. I think the two disks uploading at the same time breaks the connection to the remote for some reason. I am curious to try this with a real AWS S3 bucket, and also a backblaze B2, but I haven't tried that yet.

      TL;DR I suppose this testing makes my findings a bit of a feature request: Is there a way to set backups to serialize tasks instead of running what appears to be two in parallel? I think that would solve my issue found in testing, and I have had other use cases for backup jobs running tasks one at a time, such as a low bandwidth connection in order to increase the chance of success. I suppose it could go the other way as well in some edge cases: Perhaps a very high-performance setup may wish to do eight tasks in parallel in order to complete backup jobs as quickly as possible.

      I just thought I would share my experience and am interested to hear others' experiences with the S3 remotes beta.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Pinging @nraynaud and/or @julien-f

        1 Reply Last reply Reply Quote 0
        • SchmalzTechS Offline
          SchmalzTech
          last edited by SchmalzTech

          By the way, this testing was on xo-server 5.80.0 and xo-web 5.84.0. My initial tests were on earlier versions.

          I just tried on Backblaze B2 as well, and I am coming up with errors while testing the remote.

          
          Error
          Error calling AWS.S3.putObject: Non-whitespace before first tag. Line: 0 Column: 1 Char: B
          Test step
          write
          

          Initially I had an issue creating the remote, because Backblaze allows uppercase characters in bucket names, and XO would not allow that input.

          akurzawaA 1 Reply Last reply Reply Quote 0
          • akurzawaA Offline
            akurzawa @SchmalzTech
            last edited by

            @schmalztech it will not be working correctly yer for very long time 😛 i was trying to setup backups on backblaze in past year, didnt worked, I think s3 backup function is not production ready - but despite of this it was putted on xoa like it would be functioning correctly...

            SchmalzTechS 1 Reply Last reply Reply Quote 0
            • SchmalzTechS Offline
              SchmalzTech @akurzawa
              last edited by

              @akurzawa Well, it does say beta. I was trying different use cases and posting my results in case it's being actively worked on.

              It does work very reliably on my minio instance on my NAS so long as it's doing only one disk.

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Reping @nraynaud

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post