Long backup times via NFS to Data Domain from Xen Orchestra

MajorP93

@florent said in Long backup times via NFS to Data Domain from Xen Orchestra:

interesting

can you try to do a perfomance test while using block storage ?

This will store backup are multiple small ( typically 1MB ) files , that are easy to deduplicated, and the merge process will be moving / deleting files instead of modifying one big monolithic file per disk. It could sidestep the hydratation process.
This is the mode by default on S3 / azure, and will probably be the mode by default everywhere in the future, given its advantages

(Note for later : don't use XO encryption at rest if you need dedup, since even the same block encrypted twice will give different results)

I am not sure if it is a good idea to use that feature as default.

We just switched away from it in our XCP-ng / XO environment as it tanked performance really hard.
The issue is with using NBD, delta backups will open at least 1 data stream per virtual disk. Even with concurrency set to 1 in the backup job there will be a "spam" of small files on the remote when there is a virtual machine with a lot of virtual disks attached to it (around 10 virtual disks).

Data blocks feature resulted in transfer speeds to our NAS going down to 300 Mbit/s for VMs that have many disks.
After disabling data blocks feature transfer speed went up to 2.8 Gbit/s.

Instead I would like to see brotli compression becoming default for delta backups no matter if data blocks is turned on or not. Also encryption for remotes that do not use data blocks would be awesome. This way people can combine good performance with security.

florent

@MajorP93
interesting, Note that this is orthogonal to NBD.
I note that there is probably more work to do to improve the performance and will retest VM with a lot of disk
Performance is really depending on the underlying storage.
compression and encryption can't be done in "legacy mode" , since we won't be able to merge block in place in this case.

MajorP93

@florent said in Long backup times via NFS to Data Domain from Xen Orchestra:

@MajorP93
interesting, Note that this is orthogonal to NBD.
I note that there is probably more work to do to improve the performance and will retest VM with a lot of disk
Performance is really depending on the underlying storage.
compression and encryption can't be done in "legacy mode" , since we won't be able to merge block in place in this case.

I see thanks for the insights.
The problem that we saw could also be solved if you guys would add another config parameter to the delta backup job: disk concurrency per VM.
That way it would be possible to backup only e.g. 2 out of 10 virtual disks at the time.

florent

@MajorP93 this settings exists (not in the ui )

you can create a configuration file named /etc/xo-server/config.diskConcurrency.toml if you use a xoa

containing

[backups]
diskPerVmConcurrency = 2

Forza

@florent said in Long backup times via NFS to Data Domain from Xen Orchestra:

@MajorP93 this settings exists (not in the ui )

you can create a configuration file named /etc/xo-server/config.diskConcurrency.toml if you use a xoa

containing
[backups]
diskPerVmConcurrency = 2

That is great. Can we get it as a UI option too?

Pilow

@florent what if we use XO Proxies ?

MajorP93

@florent said in Long backup times via NFS to Data Domain from Xen Orchestra:

@MajorP93 this settings exists (not in the ui )

you can create a configuration file named /etc/xo-server/config.diskConcurrency.toml if you use a xoa

containing
[backups]
diskPerVmConcurrency = 2

Hey, does this also work for XO from sources users?

It would be great indeed if there was an UI option for this.

best regards

florent

@Pilow said in Long backup times via NFS to Data Domain from Xen Orchestra:

@florent what if we use XO Proxies ?

te the conf should be on the proxy is /etc/xo-proxy/

florent

@MajorP93 the config should be in ~/.config/xo-server/ of the user running xo-server

It is noted

MajorP93

@florent said in Long backup times via NFS to Data Domain from Xen Orchestra:

@MajorP93 the config should be in ~/.config/xo-server/ of the user running xo-server

It is noted

Thanks.

I tried what you said and placed the "config.diskConcurrency.toml" in "~/.config/xo-server/".

However my XO instance seems to not pick up the config. For VMs that have a lot of disks still all disks are being backed up at the same time. (E.g. 9 disks at the same time instead of the 2 defined in the config file).

root@2d4da229555e:~/.config/xo-server# pwd
/root/.config/xo-server
root@2d4da229555e:~/.config/xo-server# ls -alh
total 16K
drwxr-xr-x 2 root root 4.0K Nov 18 10:36 .
drwxr-xr-x 3 root root 4.0K Nov 17 09:03 ..
-rw-r--r-- 1 root root   35 Nov 18 10:36 config.diskConcurrency.toml
-rw------- 1 root root   72 Nov 17 09:03 config.z-auto.json
root@2d4da229555e:~/.config/xo-server# cat config.diskConcurrency.toml 
[backups]
diskPerVmConcurrency = 2

@florent is it also possible to append the [backups] section with the option you specified to the main config.toml file?