XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    In delta backup force the full backup task in a specific day

    Scheduled Pinned Locked Moved Backup
    21 Posts 6 Posters 2.4k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • florentF Offline
      florent Vates 🪐 XO Team @robyt
      last edited by florent

      @robyt I think I misunderstood, you want to decrease the concurrency ?
      the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

      the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

      that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

      robytR 1 Reply Last reply Reply Quote 0
      • robytR Offline
        robyt @florent
        last edited by

        @florent said in In delta backup force the full backup task in a specific day:

        @robyt I think I misunderstood, you want to decrease the concurrency ?
        the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

        the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

        that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

        Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
        at this moment remain only the big, the other was finished
        after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
        of course i've a lot of DW nfs daemon in nas

        florentF 1 Reply Last reply Reply Quote 0
        • florentF Offline
          florent Vates 🪐 XO Team @robyt
          last edited by

          @robyt said in In delta backup force the full backup task in a specific day:

          @florent said in In delta backup force the full backup task in a specific day:

          @robyt I think I misunderstood, you want to decrease the concurrency ?
          the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

          the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

          that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

          Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
          at this moment remain only the big, the other was finished
          after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
          of course i've a lot of DW nfs daemon in nas

          after the backup there is the merge process, that will colaesce/merge the older backups. This one is very expensive.
          If your NAS allow it, you can switch the remote to block based storage, in the bottom of the remote settings : "Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)"

          Please note that the next backup after enabling/disabling it will be a full for all VM.

          robytR 1 Reply Last reply Reply Quote 0
          • robytR Offline
            robyt @florent
            last edited by olivierlambert

            @florent said in In delta backup force the full backup task in a specific day:

            @robyt said in In delta backup force the full backup task in a specific day:

            @florent said in In delta backup force the full backup task in a specific day:

            @robyt I think I misunderstood, you want to decrease the concurrency ?
            the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)

            the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM

            that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS

            Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
            at this moment remain only the big, the other was finished
            after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
            of course i've a lot of DW nfs daemon in nas

            after the backup there is the merge process, that will colaesce/merge the older backups. This one is very expensive.
            If your NAS allow it, you can switch the remote to block based storage, in the bottom of the remote settings : "Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)"

            Please note that the next backup after enabling/disabling it will be a full for all VM.

            Hi, i've tried some months ago but is very very slow
            For the big VM i've over 2 tb of data, may ext4 manage two million files ?

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Online
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by olivierlambert

              A quick computation: ext4 by default will be able to handle 4 billion files.

              Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.

              If you need to store more than this, then you need to go for XFS or ZFS.

              robytR 2 Replies Last reply Reply Quote 0
              • robytR Offline
                robyt @olivierlambert
                last edited by

                @olivierlambert said in In delta backup force the full backup task in a specific day:

                A quick computation: ext4 by default will be able to handle 4 billion files.

                Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.

                If you need to store more than this, then you need to for XFS or ZFS.

                ok, i tried with a small VM in separated backup job

                1 Reply Last reply Reply Quote 0
                • robytR Offline
                  robyt @olivierlambert
                  last edited by

                  @olivierlambert said in In delta backup force the full backup task in a specific day:

                  A quick computation: ext4 by default will be able to handle 4 billion files.

                  Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.

                  If you need to store more than this, then you need to go for XFS or ZFS.

                  it's too slow. 😞
                  I've enable the block mode, i start a backup and the write speed computed by nas is 6/7 mB/sec
                  but if i mount (when the backup process is active) the nas on a linux server and transfer a 1GB block i see 18,3 mB/sec on linux, the nas compute `24 mB/Sec
                  When the linux transfer end the speed go down to 6/7 mB/Sec
                  i'm confused..
                  How i can interrupt the backup process?
                  I need to restart XOA vm?

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Online
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by olivierlambert

                    Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)

                    robytR 2 Replies Last reply Reply Quote 0
                    • robytR Offline
                      robyt @olivierlambert
                      last edited by

                      @olivierlambert said in In delta backup force the full backup task in a specific day:

                      Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)

                      yes, i've tried with and without block mode and nbd is enabled; the network is on NDB nic in pool

                      1 Reply Last reply Reply Quote 0
                      • robytR Offline
                        robyt @olivierlambert
                        last edited by

                        @olivierlambert said in In delta backup force the full backup task in a specific day:

                        Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)

                        0e57de89-11e1-4a42-a34c-162d4151f754-immagine.png
                        is locked to 37%... ho i can stop the backup job without damage old backups?

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post