XOCE limit ?
-
As info.
After several tests I have the following conclusions:- with SSL connection between server and XOCE does not exceed a maximum transfer of 33Mb/s
- without SSL connection (http://) between the server and XOCE, the maximum speed reached, in test, is 290Mb/s
The limit of maximum 1000mbs is given by the encrypted connection, it probably cannot encrypt on a 10Gbs band.
-
Are you sure you only compare SSL vs no SSL or also no SSL vs NBD?
-
@Gheppy If you go over a VLAN and/or a priviate, non-routed network, why even introduce the overhead of SSL unless you are super paranoid about security?
-
@olivierlambert
The only configuration I made for the final test was to pass http:// in front of the IP to connect to XCP-ng servers and the transer is the one shown above. -
Okay so you should try to enable NBD and bench the diff (secure and unsecure). That would be interesting to get a comparison on your side
See https://xen-orchestra.com/blog/xen-orchestra-5-76/#
-faster-backups-preview for more details -
Also, what is your CPU brand/model? Also, how many vCPU do you have in your XO VM?
-
@olivierlambert
I'll read to see how it's done and I'll start testing with NBD.
This server has the following configuration, it is only used for backup:LENOVO System x3650 M5
- 64Gb Ram
- 24x CPU, Xeon CPU E5-2620 v3 @ 2.40GHz
- 2 x 10Gb LAN, 4 x 1Gb LAN
XOCE
- 16x CPU,
- 12Gb RAM
- 3 x LAN: 1 x 1Gb, 2 x 10Gb
-
Okjay so rather old CPU which is relatively inefficient (compared to modern EPYCs) explaining the huge gap in SSL vs plain.
This is something we can investigate on our side, but if NBD provides a good boost even in SSL, I'm very interested
-
@Gheppy Run "top" ans well as "iostat" during your backup to see if any saturation is taking place -- CPU or memory on dom0, queue and I/O throughput on the storage. I agree with @olivierlambert that a 2.4 GHz CPU is marginal in this day and age.
-
Yes, but I'd like to be more precise: it's not an "excuse" or asking you to purchase better hardware. Just a fact: there's a bottleneck in SSL decode when doing disk export/import in XO. The gap is wider on less efficient CPUs, but also (a bit less) visible on modern ones.
I'd like to see if we can "workaround" this by using NBD in SSL, since in the future, nothing will be left in plain but full SSL.
-
I don't understand the following.
- After I set/configure NBD on server 2 on LAN 2 of 10Gb (I have 4x1Gb and 2x10Gb LAN)
- Then I set/configure NBD on server 3 on LAN 2 of 10Gb (I have 4x1Gb and 2x10Gb LAN)
- I connect XOCE on each server on LAN 2
This should be ok for an transfer with NBD?
What happens with server 1 that does not have a connection with NBD, will do the backup without NBD as it does now?
Can I make a backup of a VM with or without NBD and have it stored on the same HDD (as in photo)?
The HDD on the NAS is connected as a local iSCSI disk on XCP-ng level, it's not like a normal SR, it's like a /dev/sdX on XCP-ng OS level (to store VHDs on ext4 format to save space). -
I don't remember the code path, if there's a fallback. Adding @florent in the loop
-
@Gheppy
I am not an expert on the network part but I can help on the NBD partWe added some info in this PR https://github.com/vatesfr/xen-orchestra/pull/6596, it should be merged in this form or another in
masterin a few days. It will give the user more info on the NBD usage and the cause (misconfiguration, or network error)Then if XO ( or the proxy) can't connect to the host through the NBD (for example with a network filtering/ routing), it will fall back to the reliable (but non parallelizable) api call
Finally , the backup done with NBD or the legacy api call will produce exactly the same data so they are compatibles, but using the same target (as in the same directory , same remote and same VMs) for multiple job can lead to complex support tickets with interesting race conditions.
I would advise to only use the same remote/directory only if the backups jobs does not concern the same VMs
