xoa slow export

akurzawa

@akurzawa said in xoa slow export:

I've endless export task

Should not cause any issues and it will automatically be destroyed by the XenAPI after 24 hours.

wow just 24 hours? Do you know that I cannot shotdown the host right now nor my UPS in case of power failure?

julien-f

@akurzawa I'm not sure but restarting xo-server might do the trick as well.

olivierlambert

Toolstack restart will cancel the task.

cg

Did you do any tests with XCP-ng Center? Just to get any numbers.
I have one VM on XS that needs to be exportet regulary, which hits about 800 GB in ~22 hours. It always starts reasonably quick and then becomes slower by time - but 800 vs. 8...

akurzawa

@cg said in xoa slow export:

Did you do any tests with XCP-ng Center? Just to get any numbers.
I have one VM on XS that needs to be exportet regulary, which hits about 800 GB in ~22 hours. It always starts reasonably quick and then becomes slower by time - but 800 vs. 8...

In 90% cases I'm obsrving the sam thing - export either in xoa or xcp-ng center is getting slowe over time... idk whats the reason of that.

cg

As Citrix barely ever gave any insights... maybe @olivierlambert can bring light into that.

olivierlambert

We aren't doing any test with XCP-ng Center because we don't use it at all You can use xe to compare (xe CLI is basically another XAPI client -like XO or XCP-ng Center- which can run on the same host or even elsewhere)

akurzawa

Regardin export from xoa I the highest speed that I've seen was 40MB/s on raid10 setup, xoa and vm on the same host. What could be the reason of that? Host is almost always idle, just very basic workload.

olivierlambert

Export from XOA to where? Your browser, a remote?

akurzawa

Export from xoa, Gb LAN, latest Chrome, Windows

cg

@olivierlambert It's basically the same. I almost only export via scripts/xe.

olivierlambert

@akurzawa in this case, data is going from the SR to the host, then from the host to XOA, and finally from XOA to your PC with your browser. As you can see there is a lot of "hops" before being stored in your hard drive. And because it's a stream (there isn't any intermediate storage), it means it's going to the speed of the slower element (back pressure).

In general, Xen Orchestra isn't the bottleneck itself because it just "passes" the stream. The most common bottleneck is in general the host itself. Using VM export with zstd can yield good results.

@cg so the bottleneck can be from your SR to your host, the host itself, or your host to the storage destination.

cg

@olivierlambert: I doubt. It wouldn't explain why it's getting slower by time and while still using 100 % of once CPU core, the dataflow becomes constantly slower. It might be an IRQ/IO problem on hardware side, but that should be constant.
I monitored network and storage: Both is mostly idling.
It has 10/15k SAS RAID on both sides and 10G Ethernet and running only one export in the end.

olivierlambert

I said "the host itself", which could be a hardware issue. Obviously the bottleneck is somewhere

ruskofd

I agree with both of you, the problem is not on XO neither on hardware. I encountered this problem too, I tried many things (encryption off/on, no compression), same results. Like @cg said, the disks are idling also (99% idle)

I tried mostly tried on the hosts by itself, the templating system seems to work in the same way as export, and this was terribly slow for local storage in terms of throughput (25/30 Mbytes/s max on 15k SAS RAID 10). I put a bet on a XenServer core limitations/bottleneck (?), it could be some kind of a software limitation : https://discussions.citrix.com/topic/374896-exporting-vms-there-seems-to-be-a-speed-difference-between-xencenter-and-vm-export/

olivierlambert

Depends on what are we talking about:

yes, export speed is often "software limited" because how a XVA is made
I'm not aware about an issue regarding speed being reduced progressively during the transfer

akurzawa

@olivierlambert maybe it's "generally" slow, but on the beginning the speed is higher due some buffering during file download

akurzawa

could it be truth?

"XenServer is notorious for their low export/import-speed, some claim that it is due to a "feature" that protects the mgmt-interface from being out of bandwidth"

olivierlambert

There is no known code backing this claim.

cg

What you found is a pretty old and outdated thread, and conclusion is: It's not limited - and there's no real difference between 1 and 10GE. In case it's limited to protect mgmt-int, it would scale with 10GE, but in fact it doesn't.