XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    In delta backup force the full backup task in a specific day

    Scheduled Pinned Locked Moved Backup
    21 Posts 6 Posters 2.4k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • robytR Offline
      robyt
      last edited by

      Hi, i've a problem with delta..
      I've a QNAP NAS for the backup storage with nfs access (NAS + Extension bay, two raid5 volumes of 21tb each)
      When xoa perform a full backup.. it's VERY VERY slow, 1-2 days for the big VM:
      Delta FtPA Jan 4, 2024, 04:58:02 PM Jan 10, 2024, 03:00:26 AM 5 days Transfer: 2.18 TiB
      The problem is in full backup, in delta i can do small delta with more frequency, all bakcup is with NBD protocol
      Is there a method to start the full in a specific day of week?

      julien-fJ 1 Reply Last reply Reply Quote 0
      • julien-fJ Offline
        julien-f Vates 🪐 Co-Founder XO Team @robyt
        last edited by

        @robyt Hi, you can have multiple schedules for a single job and you can use the Force full backup option on one of them 🙂

        robytR 1 Reply Last reply Reply Quote 0
        • robytR Offline
          robyt @julien-f
          last edited by

          @julien-f said in In delta backup force the full backup task in a specific day:

          @robyt Hi, you can have multiple schedules for a single job and you can use the Force full backup option on one of them 🙂

          Nice..
          I must:
          Set a large number for full interval (ie if in a week i can do 30 deltas i set this parameter to 40)
          Set for delta schedule all days except saturday and sunday
          Create a scedule for full backup at saturtay

          It's ok?

          julien-fJ 1 Reply Last reply Reply Quote 0
          • julien-fJ Offline
            julien-f Vates 🪐 Co-Founder XO Team @robyt
            last edited by

            @robyt I would suggest to either use Force full backup or Full backup interval, not both. In your case, it seems Force full backup is more appropriate.

            Yes, creating a schedule for all days but Saturday and Sunday, and another one with Force full backup on Saturday is ok 🙂

            Just make sure that these two schedules are in the same Delta Backup job.

            robytR C 2 Replies Last reply Reply Quote 0
            • robytR Offline
              robyt @julien-f
              last edited by

              @julien-f said in In delta backup force the full backup task in a specific day:

              @robyt I would suggest to either use Force full backup or Full backup interval, not both. In your case, it seems Force full backup is more appropriate.

              Yes, creating a schedule for all days but Saturday and Sunday, and another one with Force full backup on Saturday is ok 🙂

              Just make sure that these two schedules are in the same Delta Backup job.

              i'm doing it and work fine.. but the full process is very slow: started saturday 13th and is not ended.
              There is not tasks but i see in XOA (via ssh) the process

              node /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_backupWorker.mjs
              

              that write at 99% of io capacity
              the process started at Jan 13

              root      7760  5.1  5.5 12028572 684916 ?     Sl   Jan13 235:53  \_ /usr/local/bin/node /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_backupWorker.mjs
              root     19227 10.8  4.1 11855200 512836 ?     Sl   Jan13 458:11  \_ /usr/local/bin/node /usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/backups/_backupWorker.mjs
              

              The nas is not fast, i know.. but the process is very very slow

              C 1 Reply Last reply Reply Quote 0
              • C Offline
                CJ @robyt
                last edited by

                @robyt You may want to change your concurrency to 1 instead of the default 2. That way you won't have multiple backups trying to copy to the NAS at the same time.

                Also, remember that the backups relay through XO so you could be network constrained depending on the host and SR networking.

                robytR 1 Reply Last reply Reply Quote 0
                • robytR Offline
                  robyt @CJ
                  last edited by robyt

                  @CJ said in In delta backup force the full backup task in a specific day:

                  @robyt You may want to change your concurrency to 1 instead of the default 2. That way you won't have multiple backups trying to copy to the NAS at the same time.

                  Also, remember that the backups relay through XO so you could be network constrained depending on the host and SR networking.

                  the huge vserver is a single vm.
                  how many NBD connection i can use?
                  the nas is on gigabit lan, the problem is not the transfer (i see "nothing" in tasks but the work after the transfer

                  florentF 1 Reply Last reply Reply Quote 0
                  • C Offline
                    chr1st0ph9 @julien-f
                    last edited by

                    @julien-f

                    1 Reply Last reply Reply Quote 0
                    • florentF Offline
                      florent Vates 🪐 XO Team @robyt
                      last edited by

                      @robyt hi, I tested with up to 16 NBD connection per disk . If the xapi don't accept the connection, it will continue the backup with the open one

                      It should be faster, though. What are the latencies between XOA and the host, and XOA and the NAS ?

                      Are you using a block based storage ? The merging of such backups may be very expensive either way

                      robytR 1 Reply Last reply Reply Quote 0
                      • robytR Offline
                        robyt @florent
                        last edited by

                        @florent said in In delta backup force the full backup task in a specific day:

                        @robyt hi, I tested with up to 16 NBD connection per disk . If the xapi don't accept the connection, it will continue the backup with the open one

                        It should be faster, though. What are the latencies between XOA and the host, and XOA and the NAS ?

                        Are you using a block based storage ? The merging of such backups may be very expensive either way

                        Hi, disk is a single sr?
                        Some server has 7(8 sr for a single VM, the nas (a small QNAP) goes under a high load.
                        I try to lower NBD to 2 for disk and check the status of nas

                        robytR 1 Reply Last reply Reply Quote 0
                        • robytR Offline
                          robyt @robyt
                          last edited by robyt

                          @robyt said in In delta backup force the full backup task in a specific day:

                          @florent said in In delta backup force the full backup task in a specific day:

                          @robyt hi, I tested with up to 16 NBD connection per disk . If the xapi don't accept the connection, it will continue the backup with the open one

                          It should be faster, though. What are the latencies between XOA and the host, and XOA and the NAS ?

                          Are you using a block based storage ? The merging of such backups may be very expensive either way

                          Hi, disk is a single sr?
                          Some server has 7(8 sr for a single VM, the nas (a small QNAP) goes under a high load.
                          I try to lower NBD to 2 for disk and check the status of nas

                          addenda:
                          the nas is shared with nfs connection (1 gigabit)
                          The transfer is good, but when the transfer ended and start the other operation the load on nas goes over 30 and slow down all transfer

                          florentF 1 Reply Last reply Reply Quote 0
                          • florentF Offline
                            florent Vates 🪐 XO Team @robyt
                            last edited by florent

                            @robyt I think I misunderstood, you want to decrease the concurrency ?
                            the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

                            the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

                            that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

                            robytR 1 Reply Last reply Reply Quote 0
                            • robytR Offline
                              robyt @florent
                              last edited by

                              @florent said in In delta backup force the full backup task in a specific day:

                              @robyt I think I misunderstood, you want to decrease the concurrency ?
                              the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

                              the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

                              that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

                              Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
                              at this moment remain only the big, the other was finished
                              after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
                              of course i've a lot of DW nfs daemon in nas

                              florentF 1 Reply Last reply Reply Quote 0
                              • florentF Offline
                                florent Vates 🪐 XO Team @robyt
                                last edited by

                                @robyt said in In delta backup force the full backup task in a specific day:

                                @florent said in In delta backup force the full backup task in a specific day:

                                @robyt I think I misunderstood, you want to decrease the concurrency ?
                                the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

                                the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

                                that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

                                Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
                                at this moment remain only the big, the other was finished
                                after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
                                of course i've a lot of DW nfs daemon in nas

                                after the backup there is the merge process, that will colaesce/merge the older backups. This one is very expensive.
                                If your NAS allow it, you can switch the remote to block based storage, in the bottom of the remote settings : "Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)"

                                Please note that the next backup after enabling/disabling it will be a full for all VM.

                                robytR 1 Reply Last reply Reply Quote 0
                                • robytR Offline
                                  robyt @florent
                                  last edited by olivierlambert

                                  @florent said in In delta backup force the full backup task in a specific day:

                                  @robyt said in In delta backup force the full backup task in a specific day:

                                  @florent said in In delta backup force the full backup task in a specific day:

                                  @robyt I think I misunderstood, you want to decrease the concurrency ?
                                  the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

                                  the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

                                  that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

                                  Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
                                  at this moment remain only the big, the other was finished
                                  after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
                                  of course i've a lot of DW nfs daemon in nas

                                  after the backup there is the merge process, that will colaesce/merge the older backups. This one is very expensive.
                                  If your NAS allow it, you can switch the remote to block based storage, in the bottom of the remote settings : "Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)"

                                  Please note that the next backup after enabling/disabling it will be a full for all VM.

                                  Hi, i've tried some months ago but is very very slow
                                  For the big VM i've over 2 tb of data, may ext4 manage two million files ?

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Online
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by olivierlambert

                                    A quick computation: ext4 by default will be able to handle 4 billion files.

                                    Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.

                                    If you need to store more than this, then you need to go for XFS or ZFS.

                                    robytR 2 Replies Last reply Reply Quote 0
                                    • robytR Offline
                                      robyt @olivierlambert
                                      last edited by

                                      @olivierlambert said in In delta backup force the full backup task in a specific day:

                                      A quick computation: ext4 by default will be able to handle 4 billion files.

                                      Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.

                                      If you need to store more than this, then you need to for XFS or ZFS.

                                      ok, i tried with a small VM in separated backup job

                                      1 Reply Last reply Reply Quote 0
                                      • robytR Offline
                                        robyt @olivierlambert
                                        last edited by

                                        @olivierlambert said in In delta backup force the full backup task in a specific day:

                                        A quick computation: ext4 by default will be able to handle 4 billion files.

                                        Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.

                                        If you need to store more than this, then you need to go for XFS or ZFS.

                                        it's too slow. 😞
                                        I've enable the block mode, i start a backup and the write speed computed by nas is 6/7 mB/sec
                                        but if i mount (when the backup process is active) the nas on a linux server and transfer a 1GB block i see 18,3 mB/sec on linux, the nas compute `24 mB/Sec
                                        When the linux transfer end the speed go down to 6/7 mB/Sec
                                        i'm confused..
                                        How i can interrupt the backup process?
                                        I need to restart XOA vm?

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO Online
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by olivierlambert

                                          Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)

                                          robytR 2 Replies Last reply Reply Quote 0
                                          • robytR Offline
                                            robyt @olivierlambert
                                            last edited by

                                            @olivierlambert said in In delta backup force the full backup task in a specific day:

                                            Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)

                                            yes, i've tried with and without block mode and nbd is enabled; the network is on NDB nic in pool

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post