In delta backup force the full backup task in a specific day
-
@CJ said in In delta backup force the full backup task in a specific day:
@robyt You may want to change your concurrency to 1 instead of the default 2. That way you won't have multiple backups trying to copy to the NAS at the same time.
Also, remember that the backups relay through XO so you could be network constrained depending on the host and SR networking.
the huge vserver is a single vm.
how many NBD connection i can use?
the nas is on gigabit lan, the problem is not the transfer (i see "nothing" in tasks but the work after the transfer -
-
@robyt hi, I tested with up to 16 NBD connection per disk . If the xapi don't accept the connection, it will continue the backup with the open one
It should be faster, though. What are the latencies between XOA and the host, and XOA and the NAS ?
Are you using a block based storage ? The merging of such backups may be very expensive either way
-
@florent said in In delta backup force the full backup task in a specific day:
@robyt hi, I tested with up to 16 NBD connection per disk . If the xapi don't accept the connection, it will continue the backup with the open one
It should be faster, though. What are the latencies between XOA and the host, and XOA and the NAS ?
Are you using a block based storage ? The merging of such backups may be very expensive either way
Hi, disk is a single sr?
Some server has 7(8 sr for a single VM, the nas (a small QNAP) goes under a high load.
I try to lower NBD to 2 for disk and check the status of nas -
@robyt said in In delta backup force the full backup task in a specific day:
@florent said in In delta backup force the full backup task in a specific day:
@robyt hi, I tested with up to 16 NBD connection per disk . If the xapi don't accept the connection, it will continue the backup with the open one
It should be faster, though. What are the latencies between XOA and the host, and XOA and the NAS ?
Are you using a block based storage ? The merging of such backups may be very expensive either way
Hi, disk is a single sr?
Some server has 7(8 sr for a single VM, the nas (a small QNAP) goes under a high load.
I try to lower NBD to 2 for disk and check the status of nasaddenda:
the nas is shared with nfs connection (1 gigabit)
The transfer is good, but when the transfer ended and start the other operation the load on nas goes over 30 and slow down all transfer -
@robyt I think I misunderstood, you want to decrease the concurrency ?
the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM
that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS
-
@florent said in In delta backup force the full backup task in a specific day:
@robyt I think I misunderstood, you want to decrease the concurrency ?
the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM
that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS
Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
at this moment remain only the big, the other was finished
after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
of course i've a lot of DW nfs daemon in nas -
@robyt said in In delta backup force the full backup task in a specific day:
@florent said in In delta backup force the full backup task in a specific day:
@robyt I think I misunderstood, you want to decrease the concurrency ?
the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM
that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS
Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
at this moment remain only the big, the other was finished
after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
of course i've a lot of DW nfs daemon in nasafter the backup there is the merge process, that will colaesce/merge the older backups. This one is very expensive.
If your NAS allow it, you can switch the remote to block based storage, in the bottom of the remote settings : "Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)"Please note that the next backup after enabling/disabling it will be a full for all VM.
-
@florent said in In delta backup force the full backup task in a specific day:
@robyt said in In delta backup force the full backup task in a specific day:
@florent said in In delta backup force the full backup task in a specific day:
@robyt I think I misunderstood, you want to decrease the concurrency ?
the settings is not exposed in the UX or in the config file, but it can be changed in the code if you're ready to test ( if this is an XOA, please open a ticket I will patch for you)the file is in <xo server directory>/node_modules/@xen-orchestra/backups/_runners/VmsXapi.mjs and you can put 1 in diskPerVmConcurrency: 0 It will limit the number of disk transferred in parallel per VM
that way, by limiting VM concurrency, and the disk per vm , you will limit the numbe of thread writing to the NAS
Hi, in UI i've "Number of NBD connexion per disk": i've 4 for disk, for last backup i've decreased the value to 1 for 3 backup job and 2 for the big backup job
at this moment remain only the big, the other was finished
after the copy to the vm i see a lot of nfs daemon and the load of nas goes high (over 30)
of course i've a lot of DW nfs daemon in nasafter the backup there is the merge process, that will colaesce/merge the older backups. This one is very expensive.
If your NAS allow it, you can switch the remote to block based storage, in the bottom of the remote settings : "Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)"Please note that the next backup after enabling/disabling it will be a full for all VM.
Hi, i've tried some months ago but is very very slow
For the big VM i've over 2 tb of data, may ext4 manage two million files ? -
A quick computation: ext4 by default will be able to handle 4 billion files.
Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.
If you need to store more than this, then you need to go for XFS or ZFS.
-
@olivierlambert said in In delta backup force the full backup task in a specific day:
A quick computation: ext4 by default will be able to handle 4 billion files.
Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.
If you need to store more than this, then you need to for XFS or ZFS.
ok, i tried with a small VM in separated backup job
-
@olivierlambert said in In delta backup force the full backup task in a specific day:
A quick computation: ext4 by default will be able to handle 4 billion files.
Since we store 2MiB blocks, it means up 8 billions MiB stored max. Which is roughly equal to 7PiB worth of data.
If you need to store more than this, then you need to go for XFS or ZFS.
it's too slow.
I've enable the block mode, i start a backup and the write speed computed by nas is 6/7 mB/sec
but if i mount (when the backup process is active) the nas on a linux server and transfer a 1GB block i see 18,3 mB/sec on linux, the nas compute `24 mB/Sec
When the linux transfer end the speed go down to 6/7 mB/Sec
i'm confused..
How i can interrupt the backup process?
I need to restart XOA vm? -
Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)
-
@olivierlambert said in In delta backup force the full backup task in a specific day:
Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)
yes, i've tried with and without block mode and nbd is enabled; the network is on NDB nic in pool
-
@olivierlambert said in In delta backup force the full backup task in a specific day:
Block mode with NBD enabled? That sounds too slow, especially for a first backup (which will be full and won't require any merge)
is locked to 37%... ho i can stop the backup job without damage old backups?
