Very slow Backup speed when using "Continuous Replication" to NFS target

0nelight

Hi XCP Community,

I use self-build Xen-Orchestra latest version; commit: 1961d; running as a VM
xcp-ng 8.2.1 on an

B450 AORUS ELITE Mainboard
AMD Ryzen 5 3600XT 6-Core Processor

XCP-NG ist installed on a single SSD with 256GB.

Additonally there is an Adaptec-PCI-Raidcard connected with a Raid1 of 2x4TB HDDs.

Local-Storage on the XCP-NG host ist EXT4! Dom0 Ram is 1,6 GB.

I am connected to a RaspberryPi4 USB-HDD with NFS for Backups.

For Test Purposes, every VM including Dom0 is on the single SSD only (you can Ignore the Raid1).

Following things run as fast as expected:

Normal "Backup" of a random VM from Xen-Orchestra to the RaspberryPi4 - no matter if hosted on the SSD or on the Raid1.
Test-File-Transfer via Command Line into mounted NFS share on Dom0, Xen-Orchestra-VM or random VM.
iperf benchmark for network connection is fine between Dom0<->XO-VM; Dom0<->RapsberryPi4; XO-VM<->RaspberryPi4; RandomVM<->Dom0 and RandomVM<->XO-VM

Networkspeed I get is 1 Gbit/s, HDD-Speed is far over 60MB/s.

But I don't get this numbers when using "Continuous Replication".
I only get 4-6 Mbyte/s speed when I want to Backup the "RandomVM" to the NFS-Share on the Raspberry. (No matter if the VM is "ON" or "OFF")

I see via htop that there seems to be two processes involved "tapdisk" and "vhd-tool serve" when doing a "Continuous Replication". In the "Disk Write" Column of htop I only get approximatly 5.27 to 6.60 M/s for these two processes. Tapdisk is always a few Kbytes faster than the "vhd-tool serve".

In the Logfiles I cannot find anything which would help me further.

Any Ideas to find the bottleneck?

Things tried:
Used "http://" for connecting the server with XO. -> No change.

Thanks!

0nelight

@Andrew
Yess!
After specifying "async" on the NFS-Server I get 70 Mbytes/Second
Is there any downside for using "async" in this context?
Thank you!

Andrew

@0nelight NFS sync on the server?

0nelight

@Andrew
Yess!
After specifying "async" on the NFS-Server I get 70 Mbytes/Second
Is there any downside for using "async" in this context?
Thank you!

Andrew

@0nelight The risk is data loss during an unplanned reboot/crash/power cycle, but for backup use that's not a big issue. 70MBytes/sec is a normal speed for backups.

Using a SR on async NFS is a bigger risk as data that needs to be saved might not be saved if the system reboots. VMs are always writing data while they are running. It's a known performance vs. risk trade off.

NFS sync is safe and slow (without hardware acceleration) and async is fast with some risk.

0nelight

@Andrew Thanks! How to close this issue?