XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    S3 Backup "Please Match The Requested Format"

    Scheduled Pinned Locked Moved Xen Orchestra
    38 Posts 4 Posters 3.9k Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V Offline
      vincentp @florent
      last edited by

      @florent said in S3 Backup "Please Match The Requested Format":

      @vincentp Did you try activating the NBD option ?

      Can you back one VM on a NFS on a separate job to check if it's really reading speed that is slow, or if it's the S3 and CR writer ?

      No, I don't have any shared storage at that site so cannot test NFS.

      I do have NDB enabled on the direct 10G connection between the two hosts.

      florentF 1 Reply Last reply Reply Quote 0
      • florentF Offline
        florent Vates 🪐 XO Team @vincentp
        last edited by

        @vincentp did you also enable it in the xapioptions of xo-server / the relevant proxy ?

        1 Reply Last reply Reply Quote 0
        • planedropP Offline
          planedrop Top contributor @vincentp
          last edited by

          @vincentp Seems to me the full backups taking a long time isn't a huge issue considering you should mostly just have deltas from there on out, other than periodic fulls once in a long while.

          I'm going to do more testing myself here soon to see if I can get things to speed up more, will test NBD.

          Also @florent I am not seeing the writeBlockConcurrency option within the config files, is it one that I should add or should it already be there?

          florentF 1 Reply Last reply Reply Quote 0
          • florentF Offline
            florent Vates 🪐 XO Team @planedrop
            last edited by

            @planedrop you should add it in the [backups] section

            planedropP 1 Reply Last reply Reply Quote 0
            • planedropP Offline
              planedrop Top contributor @florent
              last edited by

              @florent I'll give this a shot and see how performance is. Do you have a recommended number to set on this? 8?

              florentF 1 Reply Last reply Reply Quote 0
              • florentF Offline
                florent Vates 🪐 XO Team @planedrop
                last edited by

                @planedrop not reallky for now

                1 Reply Last reply Reply Quote 1
                • planedropP Offline
                  planedrop Top contributor @vincentp
                  last edited by

                  @vincentp Wanted to get some more info from you regarding your backups that were taking a long time.

                  Was the > 24hrs mark the entire backup process or just the data transfer process?

                  If you were using Delta backups and then had the backup retention in the schedule set to 1, it is still honored (even if the schedule itself was disabled), so what might have been happening is that it was merging the delta chain every single time you did a backup which takes a long time.

                  I did some more testing (nothing with huge VMs yet, but with 4 100GB ish ones) and setting the backup retention to 7 (so it only merges things into the Delta every 7 days) made it so the deltas were much faster to complete, otherwise the backup process would get "stuck" while it's merging the deltas into the full, which is the part that takes the longest from my testing.

                  Of course, once you hit that 7 retention mark, then every backup from there on out has to merge 1 delta into the full so not sure this is a solution but was just curious.

                  I'm still doing testing but I think the merge process is going to be the largest issue, so much so that it might actually be faster to NOT use Delta's and instead use full backups and just do them less frequently.

                  V 1 Reply Last reply Reply Quote 0
                  • planedropP Offline
                    planedrop Top contributor
                    last edited by

                    @florent any idea on a way to make the delta merges faster? I know there was some work in the pipeline for that a while back but don't recall what came of it. As of right now the actual uploads of the deltas are plenty fast but the merge process is super slow (for some VMs the upload happens in like 10 minutes but merge takes over an hour).

                    Would the writeblockconcurrency setting help with merges at all?

                    1 Reply Last reply Reply Quote 0
                    • V Offline
                      vincentp @planedrop
                      last edited by

                      @planedrop TBH I didn't monitor for the full 24hrs - but the transfer speeds were just too slow. The same happens with CR between hosts - way too slow to be usable.

                      Backup performance on xcp/xo is a major problem - currently battling with this at another site where we just installed a truenas server with dual 25G direct connections between the xcp host and the truenas server (with NDB - and we see < 100MiB/s transfers (nfs share on truenas scale latest) - this time the xcp host is a dual eypc 7543 with all nvme and tons of ram.

                      This thread did help a bit - top on the host was showing stunnel using a lot of cpu - switching the connection from xo to the server tp be http removed stunnel from the equation but it's still too slow - and I did read somewhere that in the next xcp/xen version http is going away and everything will be https only.

                      I'm currently researching backups on proxmox to see if they have the same sort of performance issues. I really like xcp and xo but when we end up with backups taking so long that they overlap with the next backup schedule that is not usable.

                      planedropP 1 Reply Last reply Reply Quote 0
                      • planedropP Offline
                        planedrop Top contributor @vincentp
                        last edited by

                        @vincentp I do agree backups being as slow as they are, especially for S3, is a huge issue.

                        I don't personally find backups over SMB (10GbE in this case) too slow to be usable by any means, many TB of VMs can be backed up in a single night without issues, but remote stuff is very slow.

                        I'm curious to hear about your CR experience though, as that is something I'm planning to deploy here soon. Where the hosts on the same subnet and had fast connections or was it a remote host with like IPSec handling the connectivity?

                        V 1 Reply Last reply Reply Quote 0
                        • V Offline
                          vincentp @planedrop
                          last edited by vincentp

                          @planedrop I haven't used smb for backups but will test that - currently testing with nfs which I imagine would be faster than smb though.

                          I have 3 sites, each with a separate XO instance managing them

                          LA DC

                          • 2 xcp hosts (dual E5-2620v4)
                          • no shared storage
                          • 10G direct connection between hosts

                          I tried CR between the hosts, and it works well.. once you get past the full backup - but those take 24hrs (800GB vms),
                          S3 Backup - same experience - backups seem to work well once you get past the full backup - but that sometimes takes longer than 24hrs and runs into the next backup schedule. XO has access to the 10G interface and I confirmed it's using that.

                          Sydney DC

                          • single xcp host (dual eypc)
                          • truenas scale (installed yesterday) - dual 25G direct connections to the xcp

                          Testing full backups to nfs on truenas over 25G network(NDB Enabled) with the shared option to use multiple files enabled. Also XO is connecting via http to avoid stunnel which was using a huge amount of cpu when I first tried it, and it has access to a 25G interface and I confirmed via netdata that the correct interface is being used for the backups

                          Small vm test.
                          Duration: 9 minutes
                          Size: 41.48 GiB
                          Speed: 81.03 MiB/s

                          More testing required but its frustrating that we cannot get better speeds than this - could have saved some money and just used 1G cards in the machines!

                          HomeLab

                          • single xcp host
                          • truenas server to be commissioned this week

                          just doing disaster recovery backups to 2nd local storage on the machine for now (and just as slow) - will use the truenas once I get it in the rack this week.

                          The bottleneck with the backups is definitely not the network, even on the homelab machine that backs up to local storage I'm only seeing around 60MiB/s

                          florentF 1 Reply Last reply Reply Quote 0
                          • florentF Offline
                            florent Vates 🪐 XO Team @vincentp
                            last edited by florent

                            @vincentp If you backup multiple VMs in parallel, does the total speed stays at 80MB/s or does it scales with the number of VM ?

                            NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode

                            V 2 Replies Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by olivierlambert

                              80MiB/s between LA and Sydney is already pretty impressive, knowing the latency between those. When you write a lot of small blocks, each block have to way for a round trip before being ACK. This takes a lot of time.

                              Higher the latency, longer the backup, except if we choose bigger blocks, which isn't trivial.

                              When using NBD, it should be a lot better however since we can have more blocks worked in parallel. I achieved a huge bump with more blocks at the same time.

                              Also, you can also try XO from the source on a physical machine to check the difference 🙂 (vs XO in a VM)

                              There's many many many ways to get faster, what's important is to measure each modification boost, because this might help to identify bottlenecks 🙂

                              V 1 Reply Last reply Reply Quote 0
                              • V Offline
                                vincentp @olivierlambert
                                last edited by

                                @olivierlambert said in S3 Backup "Please Match The Requested Format":

                                80MiB/s between LA and Sydney is already pretty impressive

                                No, that's local to sydney - no backups occuring between sites.

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  Do you have NBD enabled on the network used by XO to backup?

                                  V 1 Reply Last reply Reply Quote 0
                                  • V Offline
                                    vincentp @olivierlambert
                                    last edited by

                                    @olivierlambert said in S3 Backup "Please Match The Requested Format":

                                    Do you have NBD enabled on the network used by XO to backup?

                                    yes.

                                    I will try the insecure nbd - ok without encryption since its a direct connection between the machines

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      @florent is there a way to easily raise the blocks in parallel to see the impact for @vincentp ?

                                      1 Reply Last reply Reply Quote 0
                                      • V Offline
                                        vincentp @florent
                                        last edited by

                                        @florent said in S3 Backup "Please Match The Requested Format":

                                        @vincentp If you backup multiple VMs in parallel, does the total speed stays at 80MB/s or does it scales with the number of VM ?

                                        Yes it does.

                                        NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode

                                        I ran 2 full backups together - netdata and truenas are showed 1.45Gb/s so that's definitely an improvement - although it's using a lot of cpu

                                          PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
                                        23882 root      20   0 1022156 101484  20560 S 105.0  1.3  18:08.94 xapi
                                        25349 root      20   0   31884   3872   2660 R  66.0  0.0   0:25.54 tapdisk
                                        10163 root      20   0   32696   4896   2900 R  28.7  0.1   2:25.91 tapdisk
                                        

                                        VM1
                                        Duration: 10 minutes
                                        Size: 41.51 GiB
                                        Speed: 67.6 MiB/s

                                        VM2
                                        Duration: 10 minutes
                                        Size: 23.25 GiB
                                        Speed: 39.13 MiB/s

                                        Some other backups are running at the moment, when they are done I will measure VM1 by itself to compare to the earlier one I posted.

                                        1 Reply Last reply Reply Quote 1
                                        • V Offline
                                          vincentp @florent
                                          last edited by

                                          @florent said in S3 Backup "Please Match The Requested Format":

                                          NBD also use encryption by default. You can use it unencrypted by removing the NBD purpose on the network and adding insecure_nbd https://docs.citrix.com/en-us/citrix-hypervisor/developer/changed-block-tracking-guide/enabling-nbd.html#enabling-an-insecure-nbd-connection-for-a-network-notls-mode

                                          Duration: 9 minutes
                                          Size: 41.52 GiB
                                          Speed: 75.26 MiB/s

                                          So didn't really make any meaningful difference (was 80MiB/s previously).

                                          florentF 1 Reply Last reply Reply Quote 0
                                          • florentF Offline
                                            florent Vates 🪐 XO Team @vincentp
                                            last edited by florent

                                            @vincentp
                                            if you are on master, can you apply the setting from here : https://xcp-ng.org/forum/topic/7209/slow-backups-updated-xo-source-issue/8 ?

                                            V 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post