Slow Read & Okay Write Speeds with Xen Orchestra to NFS
-
Hi all,
i have my Host connected to my NAS via 10G Networking (over a different subnet).Situation
Sanity Check of my Setup via iperf3
- 10G IP: 9.05 Gbits/sec / 9.31 Gbits/sec
- 1G IP: 940 Mbits/sec / 941 Mbits/sec
Backup Speed via Xen Orchestra
- 10G Remote 47.79-56.66 MiB/s
XO Remote Speed Write/Read Test:
- 10G IP: 494.19 MiB/s / 1.3 GiB/s
- 1G IP: 642.79 MiB/s / 2.68 GiB/s
dd on the nfs mount from XO
dd if=/dev/zero of=/run/xo-server/mounts/{UUID}/test1.img bs=2G count=1 oflag=dsync739 MB/s
CPU
Host: 6vCPU each with around 25%
NAS: 4CPU total of around 5% usageXO Remote Settings
Single vs Multiple VM
The total speed is the same.
Multiple Data Blocks
I also tested the
Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)function for the nfs remote. No change.NFS Options
nfsvers=4.1,async,noac,proto=tcpNetwork Settings
MTU 1500 vs 9000
Regarding that with bigger files and dd I have no issues. I did not expected an issue here. But i even changed on both sides the MTU. No change.
TCP Checksumming
I disabled this everywhere.
Other
- I dabbled with tcp-offload-checksum-errors, but seeing that iperf3 works without that change, i didn't bother.
- This one intel-i218i219-slow-speed helped a bit on my VM with Public IP in the Data Center a bit. After i changed it in Xen Orchestra (see above)
Head Scratching
I'm currently scratching my head for where the bottle neck comes from.
If I Backup with the same Instance a Remote Host (1G Down/50M UP) Network Link to a DC with 2G/2G Sync, I'm only getting like 4MiB/s in XO.Anyone can point me in a direction or maybe help me with this Performance degradation in Xen Orchestra?
p.s. i'm not sure to post this in the backup or the Network Category, but regarding that the network works fine. I'll post it in backup
-
I see that there is a "known" limitation of the system.
See the following post.
https://xcp-ng.org/forum/topic/9389/backup-migration-performance/5But also my question is, how can it drop down to 2MiB/s over the network. Even when I should have the possibility (according to iperf3) to pass through nearly 1GiB/s.
I once got 250MiB/s in the Logs of my Backups. I was on a 8.3 Beta but can't for the life of me see, where the speed as evaporated into.
-
@Houbsi
what NAS are you using? -
Your backup is a stream, doing backupressure (or not). As explained in the linked topic, it depends on dozens of different parameters.
You might try to use NBD with 4 concurrent download, that might be better though, because I'm betting your bottleneck is likely XAPI export speed.
