Cannot export OVAs

sluflyer06

@florent how big was the VM? Smaller VMs always worked, the larger ones have been the problem. I've had some make it pretty far and get 10s of GB expected before failing but 0 have completed. I've tried XO from sources and also XOA deployed from the website.

I'm happy to try to provide any kind of information that would help

florent

@sluflyer06 It was a small VM, I will test bigger ones

florent

@florent a 10GB export worked
How many disks have you got in this VM ?

sluflyer06

@florent All my VM's only have 1 disk each, my XO vm (10GB) will export but the others range from 40GB up to 100GB and those have never been able to complete.

Danp

I just tried exporting a 20GB VM, and it stopped in the middle of the download. Here is what appeared in the logs --

Apr 24 09:32:42 ubuntuxo xo-server[6111]: 2023-04-24T14:32:42.348Z xo:xo ERROR HTTP request error {
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   data: { filter: undefined, limit: undefined },
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   error: NodeError: Premature close
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.onclose (/opt/xen-orchestra/packages/xo-server/node_modules/readable-stream/lib/internal/streams/end-of-stream.js:116:76)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.emit (node:events:525:35)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at emitCloseNT (node:_http_server:984:10)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.onServerResponseClose (node:_http_server:277:5)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.emit (node:events:525:35)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at TCP.<anonymous> (node:net:322:12)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17),
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   fn: 'handleGetAllObjects'
Apr 24 09:32:42 ubuntuxo xo-server[6111]: }

The export task is still active / progressing in XO.

Danp

Some additional details --

OVA export on XOA (latest) worked without any issue.
Updated XO from d48ef1f81 to 226f9ad96, and successfully exported the OVA.

Andrew

@florent Looks like there is a memory leak in XO for the OVA export.

I'm running current XO source master (226f9ad96). I gave XO 4G (as normal) and it started the OVA export. Then node ran out of memory and started to thrash swap and locked up. I increased the memory to 16G and the same thing happened..... out of memory.

FYI: I'm trying to export a 120G VM (single disk).

olivierlambert

And with XOA there's the same behavior?

Andrew

@florent @olivierlambert Yes. It's a little different, but the end result is the same. XOA VM out of memory... Swap thrash... transfer slows to almost zero.

- node: 18.12.1
- npm: 8.19.2
- xen-orchestra-upload-ova: 0.1.4
- xo-server: 5.111.1
- xo-server-telemetry: 0.5.0
- xo-server-xoa: 0.19.0
- xo-web-free: 5.114.0
- xoa-cli: 0.35.0
- xoa-updater: 0.45.1

Also, curl seems to be out of date and does not deploy XOA. Also, the XOA it deploys is old...

This worked for me: bash -c "$(wget -qO - https://xoa.io/deploy)" and then I updated to the latest XOA.

olivierlambert

Not sure to get it, what do you mean by "curl seems to be out of date"?

Also, yes, XOA is not re-generated often, eg now we are waiting the next Debian 12 release. So it's normal to update it after deploy

Andrew

@olivierlambert On current XCP 8.2.1:

# bash -c "$(curl -sS https://xoa.io/deploy)"
curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).

olivierlambert

That's our official doc: https://xen-orchestra.com/docs/xoa.html#via-a-bash-script

Maybe it's not updated somewhere else?

Andrew

@olivierlambert Curl is from my old notes, so my mistake... It's also still in the older blog posts:

https://xen-orchestra.com/blog/one-line-to-deploy-xoa/

https://xen-orchestra.com/blog/how-to-start-with-xen-orchestra/

olivierlambert

Thanks, fixed in both links

florent

@Andrew @sluflyer06 I am exporting VM since yesterday without problems

Did you make a yarn or npm install after updating code ? we rollbacked a dependency

Andrew

@florent I still see the problem.

I did a cache clean:

yarn cache clean
npm cache clean --force

and then did a XO source code update and rebuild. OVA export starts downloading at 2MBytes/sec and then fails.

I also see the problem on XOA (standard deploy):

- node: 18.12.1
- npm: 8.19.2
- xen-orchestra-upload-ova: 0.1.4
- xo-server: 5.111.1
- xo-server-telemetry: 0.5.0
- xo-server-xoa: 0.19.0
- xo-web-free: 5.114.0
- xoa-cli: 0.35.0
- xoa-updater: 0.45.1

Download starts at 10MBytes/second and runs for a while before it fails.

florent

@Andrew if this is in a XOA, could you open a support tunnel and allow me to try to download the same ova ?

Andrew

@florent Open.

So, I did not see the failure on a 60G VM, but I do see it on the 120G VM. Both use up memory but it does not hit critical on the smaller VM.

florent

@Andrew thanks you for the tunnel I found something

@sluflyer06 can you tests this branch ?
https://github.com/vatesfr/xen-orchestra/pull/6800

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-vmdk-to-vhd): wait for OVA stream to be written before reading more data #6800

Andrew

@florent That works much better for me! I was able to quickly OVA export VMs that failed before. Memory usage on XO was normal and did not grow. Download speed was good most of the time at 30-35MBytes/sec. It would pause occasionally but toward the end it picked up to 50-60MBytes/sec.