10GB backup with XOA (XCP to windows NFS)

andrea.mazzoni

Hi,
i have:
Host-1 (XCP 8.1 installed)
Host-2 (Windows erver 2016 with NFS)
Both Host have 10Gb nic configured and i can see that both XCP and Windows server see the Nic without issue

The 2 Host are connected with a direct 10GB cable (no switch)
There are no other cable or nic configured (on both host) all the VM on XCP and the management network on XCP are on the 10GB nic.

I tried to run a backup from XOA on XCP on Host-1 to SHare NFS on Host-2, but the speed is around 25MiB/s.

Then i disable offloading on both Hosts but nothing chnage in the tests.

olivierlambert

In XO, Settings/server, try to connect the host IP with http://<IP> instead of just the IP.

DeOccultist

@olivierlambert Will connecting with http://<IP> authenticate using plain text l/p over the network?

olivierlambert

Yes, it's just for testing if it's related XAPI bottleneck (stunnel)

andrea.mazzoni

After the test with Http://ipserver
the speed written in the backup report is 33,8 MiB/s.

On XCP could i use iperf?

olivierlambert

So it's a bit better indeed (25% better perfs!).

You can use iperf indeed, but I already know the result: it won't be a bandwidth issue. It's all on the export speed of the host itself.

andrea.mazzoni

What do you mean with export speed of the host?
10Gb nic or 1Gb nic are the same for XCP-ng or XOA? As i can see if I use 1Gb or 10Gb nothing change and the performance are bad in both cases.

olivierlambert

When you do a backup, this will happen:

dom0 will access the VM disk (snapshot disk but whatever)
it will create a HTTP handler where the disk data can be fetch
it will give the URL of this HTTP handler to XO
then XO will fetch from this HTTP(S) handler
XO will stream the content of this handler to the NFS share

We know that a major bottleneck is the speed at which the HTTP handler can be fetch.

andrea.mazzoni

We checked the network interface status and we found that the network work as the attached picture.
New Bitmap Image.jpg

Is that normal? it goes up and then to 0 and then up again, etc...
(this test is new and we used a 1Gb nic, to find if XOA use the 1Gb nic to its full)

We will try the 10Gb soon.

olivierlambert

Please be more explicit. What generated this output? iperf?

andrea.mazzoni

XOA backup.
We start a backup of a turned off Windows server 2016.
When we try to copy from a NAS to the same NFS share we don't have the same issue.

andrea.mazzoni

With the 10Gb nic the issue is worse. Now we have the backup with 4-5 packets around 700MB and then 10-15 second with few KB, and then again 4-5 packets around 700MB.
Before we didn't have that much time without high transfer speed, i think because 10Gb nic can send more with less packets.

olivierlambert

Again, you need to think globally. The transfer speed is depending on the slowest element. If the HTTP handled can't export faster than let's say 30MiB/s, then it won't magically be transferred faster than that, despite having a 100G network.

You have export speed, SR speed, stream speed, write speed, protocol speed, read seek speed etc.

andrea.mazzoni

What do you mean with HTTP? can you be more specific with HTTP limitations?

olivierlambert

I explained that here: https://xcp-ng.org/forum/post/28832

andrea.mazzoni

Do you know where i can find some data about HTTP handler on XCP/Xenserver? How much GB can be fetch in the handler?

andrea.mazzoni

Hi,
i opened this ticket because our datacenter has 67 VM and right now we take 20 Hours to make a full backup.
We need to fast it up a lot.
The differential backup need 5 hours to backup 350GB.

olivierlambert

I think it's likely the bottleneck, and except disabling HTTPS, there's no magical rule. You can play with concurrency in XO to see if it's better.

Also, it's not a "ticket" you opened, but a thread on a community forum, where people can answer on their free time.

andrea.mazzoni

Sorry, i know this isn't a ticket, i meant the question here on the forum.
and we write this thread because we cannot find a solution anywhere for our issue/question to speed up the backup.
We are making a lot of test right now, but we are not going better in anyway and we found a lot of limits in XOA/XCP-ng and Xen server 7.1 CU2 to, we are a bit worried about this issue with the backup.

olivierlambert

Note that you can scale up with concurrency (see https://xen-orchestra.com/docs/backups.html#backup-concurrency).

Also, if you have different pools, you could use XOA Proxies (alternatively, multiple XO from sources but it's less elegant) so you can accumulate all pool speed to get to a decent global speed.