XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Disater recovery backups crazy slow

    Scheduled Pinned Locked Moved Backup
    27 Posts 4 Posters 4.9k Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      comdirect
      last edited by

      xen version.png

      300gig from ssd to 2TB sata drive takes 6 hrs. i can backup inside vm to a iscsi in 1 hr.

      i have two vm servers both with the same issues. cant seem to see why there is such a bottle neck on the backup?

      when i choose delta backup transfer rate is 10Mbps cant seem to see why so slow.

      i attched a SR drive as a second HDD and used a cloning software and it cloned over 100MBs so i know its not drives.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Online
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Hi,

        It's really hard to answer with such scarce information. First, be sure to be fully up to date on master, and/or try with XOA.

        C 1 Reply Last reply Reply Quote 0
        • C Offline
          comdirect @olivierlambert
          last edited by

          @olivierlambert so interesting enough , i changed to CR and not DR and backup is fast.

          1 Reply Last reply Reply Quote 0
          • L Offline
            ludovic78
            last edited by

            Hello Comdirect and Olivier,

            I think I have about the same problem, what do you mean by CR and DR?
            I have four hosts and two storage servers with Truenas core 13 10gig network for migration and backup.
            I have 12 crucial 2tb ssd's and 6 8bt wd red pro hdd's .
            Whether I use stripe vdev pools or mirror with or without cache, I always have slow performance.
            But when I do a Crystaldiskmark test on a machine these look good

            6cde0e46-44a9-4c46-98c7-f8dbf70d9a4c-image.png
            This is with a 1vdev 2x8tb mirror and ssd cache.
            We would love to use it in production but until this problem gets resolved it is a no go.
            I am probably missing the elephant in the room.

            kind regards,
            Ludo

            P 1 Reply Last reply Reply Quote 0
            • P Offline
              probain @ludovic78
              last edited by

              @ludovic78
              Try and set the Sync-setting to Disabled, on the target dataset that you're sharing. See if that does any difference.
              56c0d2ef-7bd1-4f47-80d0-afe9b65e92e1-image.png

              This makes a GIANT difference for me.

              L 1 Reply Last reply Reply Quote 0
              • L Offline
                ludovic78 @probain
                last edited by

                @probain
                I know this can give better performance, but I cannot use this in a production environment since in case of power outage or server failure there is loss of data or even corruption of vm's.
                Would there be a guide available somewhere on how to get the most out of a setup?

                Thanks you for your reply!

                P 1 Reply Last reply Reply Quote 0
                • olivierlambertO Online
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  DR is using full everytime and XAPI will generate the XVA (compression can even make it slower). CR will just expose the disks, and export is faster. So it's very likely the bottleneck you have on your setup is due to XVA generation from XAPI.

                  L 1 Reply Last reply Reply Quote 0
                  • L Offline
                    ludovic78 @olivierlambert
                    last edited by

                    @olivierlambert said in Disater recovery backups crazy slow:

                    Thanks for the reply!

                    For my tests I use xen from source
                    Xen Orchestra e5608 and master a0c39
                    I also have slow disk migration it is so not only when I do backup.
                    Just did another migration test from a
                    4 x vdev mirror ssd to a 1 vdev 2x hdd with ssd cache averaged 30MB/s speed
                    Two days ago I had 150MB/s in the same test nothing was changed on the installation

                    How can I know if I am in DR or CR?

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Online
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      It is visible in your backup job, edit and check the type 🙂

                      1 Reply Last reply Reply Quote 0
                      • P Offline
                        probain @ludovic78
                        last edited by

                        @ludovic78
                        In the case of such a power outage, the backup would fail anyway.
                        And I'm also assuming that you have a separate share/dataset for those specific backups.
                        Coble it together with the Health Checks, and it is semi-production at least.
                        If it is for such a highly critical environment, that it wouldn't tolerate more than that. Then obviously you should open a support ticket 🙂

                        L 1 Reply Last reply Reply Quote 0
                        • L Offline
                          ludovic78 @probain
                          last edited by

                          @probain On the receiving remote (smb) the sync is off I also perform a health check on a stripe ssd dedicated to that where it is also off.
                          This is a snip of the machine the XOA is running on, what you see is the activity during the health check, I have no idea why this is behaving this way.
                          This also happens while taking the backup itself, this is not always the case, if it remains stable I have about 80MB/s speed.
                          Considering I have in production vm's that are 700GB in size this is way too slow.
                          417a4ee5-ac69-4233-acec-6ec4774f9b75-image.png

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Online
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            With CR you will only replicate the new blocks, so after the initial "sync", future runs will be very fast.

                            C 1 Reply Last reply Reply Quote 0
                            • C Offline
                              comdirect @olivierlambert
                              last edited by

                              @olivierlambert it first run so it's a full copy and 300 gigs took 1.5 hrs. With dr first run take 3-4. I do backups inside to iscsi very fast also. Just DR seems to take twice as long and I'm fully updated. It is from sources but fully updated. I even downloaded the official and tried trial and dr is still slow.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Online
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Please switch to CR to get rid of the XVA generation bottleneck first.

                                L C 2 Replies Last reply Reply Quote 0
                                • L Offline
                                  ludovic78 @olivierlambert
                                  last edited by

                                  @olivierlambert Just to make sure we are talking about the same thing cr is continous replication and dr disaster recovery right?

                                  1 Reply Last reply Reply Quote -1
                                  • olivierlambertO Online
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    Yes. Similar function, just done differently (XVA export vs VDI export).

                                    1 Reply Last reply Reply Quote 0
                                    • C Offline
                                      comdirect @olivierlambert
                                      last edited by

                                      @olivierlambert that's what I did and performance is fine. Soon as I choose dr (disaster recovery) it takes 2 5 times longer.

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Online
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        So don't use it. When you use DR, are you using compression or not?

                                        1 Reply Last reply Reply Quote 0
                                        • L Offline
                                          ludovic78
                                          last edited by

                                          It behaves differently and puts much less stress on the cpu
                                          Unfortunately, the speed is still a problem.

                                          78cd3dd0-ae21-4dc3-99d9-6f3536fe5791-image.png

                                          1 Reply Last reply Reply Quote 0
                                          • L Offline
                                            ludovic78
                                            last edited by

                                            I will put my xoa on another host maybe this will help

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post