Exports OVA Timeout...
-
Background:
I’m using the XOA appliance (5.70.2 XOA Build: 20210823) on top of a physical Citrix Hypervisor8 system set to be the “latest” channel. This exposes the UI the ability to export to OVA format. I’ve done 1 successful export in the last couple of days, but this feels like a “fluke” because I’ve not been able to repeat this successful outcome.I’m not a customer of either Citrix Hypervisor or XOA. I’m helping a customer back out of Citrix onto another platform and using virt-v2v to do the conversion. I found Citrix XenCenter native OVA export creates an OVA backed by OVF file that is mixed UT-8 and UT-16 which neither Virtualbox or virt-v2v can handle.
I can export using OVF and use other tools to convert the VHD disk - but this lengthy, tedious and unautomated process - and XOA with OVA export were recommended as an approach to get a successful & useable OVA file.
So sadly, XOA OVA export is compatible – with other systems but I can’t just get the XOA export to work consistently.
Symptoms:
- The exports starts – but the browser only received a 4KB file
- The export is still run as a task in the background until reaches 100%
- I then start to see a transfer that is exceptionally slow in the kb/sec range when I’m 1gps network end-to-end
- After a while the connection times out and the web-fronted of XOA is inaccessible but still responds to pings. I suspect that's a just a generic HTTP timeout due to perceived inactivity
- As then I have no web-based management capability I’m forced to restart the XOA
- The connection to Citrix Hypervisor8 is still good and functional from the Citrix XenCenter
Causes:
This feels like a web-server issue on the XOA especially as it becomes unresponsive. It doesn’t feel like a client/browser issue because I’ve tried multiple systems and multiple web browsers with the same result. I'm 1gps network on the same physical switch - so I can't see how this issue of connectivity. It feels more like the web service doesn't have enough resources or allows the right amount of HTTP sessions to successfully complete the transfer. -
Adding @florent in the conversation (he made the OVA export feature)
-
@chelle_belle I have (and reported) similar issues like this. See post #5871
I'm on XO source (current master) and using XCP 8.2.1 and I have most of the same issues, except I do get a valid download that finishes.
How large is your VM that you are trying to export?
For me:
- the export starts and I get a small download (4-5K) and it pauses
- export task continues to run until 100%
- the rest of the download continues slowly
- I get a complete download after a while
- there is a task stuck and I have to restart the tool stack (on the master host) to make it go away.
-
@Andrew - Thanks for your response - although the outcome is slightly different - the symptoms look very similar. Perhaps your experience - was like my "fluke". Citrix XenCenter reports the disk is 32GB in size, which I assumed was thinly provisioned. However, an OVF export indicates the corresponding VHD is 32GB so it looks like the default is not thinly provisioned disks in Citrix Hypervisor8.x.x
-
32GiB isn't that big to export
@florent any idea what could go wrong?
edit: at some point, that would be interesting to get your VM in XVA format so we can import it in our lab, and try to export it into OVA and see if we can reproduce.
-
@olivierlambert - okay - one thing haven't tried is XVA export via XOA. If that falls, I can export to some other format - and zip up and ping across...
-
@chelle_belle - So i tried using an XVA export from XOA and that works like a dream - immediately see the file grow and the transfer starts immediately...
-
@florent will come here to ask some question
-
@chelle_belle @florent I have the same result. XVA export works great.
-
@chelle_belle so the ova is a tar containing many files, (one 4KB ovf file containing the metadata and one vmdk file per disk containing the data)
since it's a tar we need to know the size of each file to be able to stream it
but the most supported subformat of vmdk is the stream optimized , which mandates to compress each 64KB part of the disk. Since we can't compute the compressed size of the vhd transformed to vmdk we need to transfert the disk, compress it to vmdk , keep track of the compressed size, then start again, this time sending the data
Since we can't store the vmdk on xo-server disks, we need to transfer and transform the disk to vmdk twice. And the transform process is quite slow.
As I write this, I am wondering if I can create a less efficient vmdk with empty spaces at the end so that I can do it in one pass.
I will test it and let you know the result
-
@chelle_belle can you test this branch : fix_ova_speed
there will still have a pause after the 5kb and after each disk, but it should speed up things . I will look for other performance optimization during this sprint.
-
@florent That branch is a LOT faster for OVA export but does not compress the data as much.
- XVA/NoComp 40 seconds to complete. 1760MB
- XVA/Zstd 30 seconds to complete. 560MB
- XVA/gzip 90 seconds to complete. 586MB
- New OVA 40 seconds to start, 60 seconds to download. 1570MB
- Old OVA 6 minutes to start, 6 minutes to download. 597MB
It still leaves a zombie task....
[XO] VM OVA export (TestSmall7-64 on xcp1) 0% -
Thanks for your feedback @Andrew as usual
-
Ah I see, @florent just disabled compression entirely to see the diff, so maybe using compression level to 1 will be better while keeping that speed
-
@olivierlambert I see that now...
I guess the best choice would to have an option during export like with XVA. Quick-NoComp OR Slow-Compression. It could also be applied to export disk function.
I tried 1 but it did not seem to change the size of time. Maybe I did not change it correctly.
I guess technically the no-compression option could stream the data without the pre-calculation work for compression that takes time but is no longer needed.
-
Maybe try 6? Don't forget to rebuild after that
-
@olivierlambert @florent My mistake.... I did not rebuild XO correctly.
Using level 1 is about 1 minute to start and about 1 minute to download and was 637MB. Seems to be a reasonable tradeoff for a significant increase in speed (over level 9) and a minimal decrease in compression.
So level 0 (no compression) actually does not help unless the code is modified to just stream the data without the pre-calculation and would cut the export time in half (no scanning/calculation needed).
Level 1 is reasonable if it the current procedure is maintained. Level 9 should be optional but not forced. Level 9 is very slow.... If compression is left in place then it would be nice to have some options.... 1..3..6..9... Or some names (min=1, fast=3, standard=6, max=9).
The quickest change is to just make it level 1. That still offers compression and still takes time but is MUCH faster and I think that's what most people want. speed...speed....speed....
-
Sounds a good modification to do in the short term until make it editable indeed
@chelle_belle if you need assistance so we can configure it this way on your XOA, let me know
-
@olivierlambert - I'm struggling to switch to the channel and do the update. When I got to set it I get this error message:
06/06/2022, 17:07:33: Start updating...
06/06/2022, 17:07:33: No manifest found for fix_ova_speed channel.I assumed I had to switch to "Unlisted Channel" and then input fix_ova_speed as the Unlisted channel name...
-
@chelle_belle a release channel is not the branch on Github, which can be used only if you installed XO from the sources (https://xen-orchestra.com/docs/installation.html#from-the-sources).
In theory, we could build packages to match such a release channel, but it's not trivial on our side. It might be easier to just install XO from the sources and not using
masterbranch butfix_ova_speed. If you need assistance on doing that, I'm here
