Cannot export OVAs

Danp

I just tried exporting a 20GB VM, and it stopped in the middle of the download. Here is what appeared in the logs --

Apr 24 09:32:42 ubuntuxo xo-server[6111]: 2023-04-24T14:32:42.348Z xo:xo ERROR HTTP request error {
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   data: { filter: undefined, limit: undefined },
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   error: NodeError: Premature close
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.onclose (/opt/xen-orchestra/packages/xo-server/node_modules/readable-stream/lib/internal/streams/end-of-stream.js:116:76)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.emit (node:events:525:35)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at emitCloseNT (node:_http_server:984:10)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.onServerResponseClose (node:_http_server:277:5)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.emit (node:events:525:35)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at TCP.<anonymous> (node:net:322:12)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17),
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   fn: 'handleGetAllObjects'
Apr 24 09:32:42 ubuntuxo xo-server[6111]: }

The export task is still active / progressing in XO.

Danp

Some additional details --

OVA export on XOA (latest) worked without any issue.
Updated XO from d48ef1f81 to 226f9ad96, and successfully exported the OVA.

Andrew

@florent Looks like there is a memory leak in XO for the OVA export.

I'm running current XO source master (226f9ad96). I gave XO 4G (as normal) and it started the OVA export. Then node ran out of memory and started to thrash swap and locked up. I increased the memory to 16G and the same thing happened..... out of memory.

FYI: I'm trying to export a 120G VM (single disk).

olivierlambert

And with XOA there's the same behavior?

Andrew

@florent @olivierlambert Yes. It's a little different, but the end result is the same. XOA VM out of memory... Swap thrash... transfer slows to almost zero.

- node: 18.12.1
- npm: 8.19.2
- xen-orchestra-upload-ova: 0.1.4
- xo-server: 5.111.1
- xo-server-telemetry: 0.5.0
- xo-server-xoa: 0.19.0
- xo-web-free: 5.114.0
- xoa-cli: 0.35.0
- xoa-updater: 0.45.1

Also, curl seems to be out of date and does not deploy XOA. Also, the XOA it deploys is old...

This worked for me: bash -c "$(wget -qO - https://xoa.io/deploy)" and then I updated to the latest XOA.

olivierlambert

Not sure to get it, what do you mean by "curl seems to be out of date"?

Also, yes, XOA is not re-generated often, eg now we are waiting the next Debian 12 release. So it's normal to update it after deploy

Andrew

@olivierlambert On current XCP 8.2.1:

# bash -c "$(curl -sS https://xoa.io/deploy)"
curl: (35) Cannot communicate securely with peer: no common encryption algorithm(s).

olivierlambert

That's our official doc: https://xen-orchestra.com/docs/xoa.html#via-a-bash-script

Maybe it's not updated somewhere else?

Andrew

@olivierlambert Curl is from my old notes, so my mistake... It's also still in the older blog posts:

https://xen-orchestra.com/blog/one-line-to-deploy-xoa/

https://xen-orchestra.com/blog/how-to-start-with-xen-orchestra/

olivierlambert

Thanks, fixed in both links

florent

@Andrew @sluflyer06 I am exporting VM since yesterday without problems

Did you make a yarn or npm install after updating code ? we rollbacked a dependency

Andrew

@florent I still see the problem.

I did a cache clean:

yarn cache clean
npm cache clean --force

and then did a XO source code update and rebuild. OVA export starts downloading at 2MBytes/sec and then fails.

I also see the problem on XOA (standard deploy):

- node: 18.12.1
- npm: 8.19.2
- xen-orchestra-upload-ova: 0.1.4
- xo-server: 5.111.1
- xo-server-telemetry: 0.5.0
- xo-server-xoa: 0.19.0
- xo-web-free: 5.114.0
- xoa-cli: 0.35.0
- xoa-updater: 0.45.1

Download starts at 10MBytes/second and runs for a while before it fails.

florent

@Andrew if this is in a XOA, could you open a support tunnel and allow me to try to download the same ova ?

Andrew

@florent Open.

So, I did not see the failure on a 60G VM, but I do see it on the 120G VM. Both use up memory but it does not hit critical on the smaller VM.

florent

@Andrew thanks you for the tunnel I found something

@sluflyer06 can you tests this branch ?
https://github.com/vatesfr/xen-orchestra/pull/6800

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-vmdk-to-vhd): wait for OVA stream to be written before reading more data #6800

Andrew

@florent That works much better for me! I was able to quickly OVA export VMs that failed before. Memory usage on XO was normal and did not grow. Download speed was good most of the time at 30-35MBytes/sec. It would pause occasionally but toward the end it picked up to 50-60MBytes/sec.

sluflyer06

@Andrew Exporting all my large VM's worked on that branch.

Andrew

@sluflyer06 Today, XO source master has been updated with that fix.

sluflyer06

@Andrew Awesome.

olivierlambert

Thanks for your feedback lads!