Cannot export OVAs

sluflyer06

Dec  8 09:43:34 xen-730xd xapi: [error||1163154 HTTP 10.10.10.40->:::80|[XO] Exporting content of VDI KASM Disk R:1716c671a4c2|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
Dec  8 09:43:34 xen-730xd xapi: [error||1163154 HTTP 10.10.10.40->:::80|[XO] Exporting content of VDI KASM Disk R:1716c671a4c2|vhd_tool_wrapper] vhd-tool output: vhd-tool: internal error, uncaught exception:\x0A          Unix.Unix_error(Unix.ECONNRESET, "write", "")\x0A          \x0A

Dec  8 09:43:35 xen-730xd xapi: [debug||1163154 HTTP 10.10.10.40->:::80|[XO] Exporting content of VDI KASM Disk R:1716c671a4c2|taskhelper] the status of R:1716c671a4c2 is failure; cannot set it to `failure
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] [XO] Exporting content of VDI KASM Disk R:1716c671a4c2 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 1/13 xapi Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 77
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 2/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 3/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 4/13 xapi Called from file ocaml/xapi/export_raw_vdi.ml, line 50
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 5/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 6/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 7/13 xapi Called from file ocaml/xapi/export_raw_vdi.ml, line 90
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 8/13 xapi Called from file ocaml/xapi/export_raw_vdi.ml, line 116
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 101
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 122
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 11/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 12/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 13/13 xapi Called from file lib/backtrace.ml, line 177
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace]
Dec  8 09:43:35 xen-730xd xapi: [debug||1164861 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:29e6026e7e06 created by task D:d1b3995b0263
Dec  8 09:43:35 xen-730xd xapi: [ info||1164861 /var/lib/xcp/xapi|session.logout D:7523bcca94fa|xapi_session] Session.destroy trackid=c968b6dc084a017a6452cb2381301d9d
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] VDI.export_raw_vdi D:d1b3995b0263 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 1/13 xapi Raised at file lib/debug.ml, line 227
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 2/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 3/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 4/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 5/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 201
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 7/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 8/13 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace]
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] VDI.export_raw_vdi D:14ad56338491 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace] 1/1 xapi Raised at file (Thread 1163154 has no backtrace table. Was with_backtraces called?, line 0
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||backtrace]
Dec  8 09:43:35 xen-730xd xapi: [error||1163154 :::80||xapi_http] Unhandled Api_errors.Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Dec  8 09:43:35 xen-730xd xapi: [debug||515 ||xenops] Event on VM bb987a49-84c2-40f5-b979-bd1c84d733c6; resident_here = true
Dec  8 09:43:35 xen-730xd xapi: [debug||515 ||dummytaskhelper] task timeboxed_rpc D:fb413298ca81 created by task D:8f9e0052bcc3
Dec  8 09:43:35 xen-730xd xapi: [debug||1164862 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:event.from D:9e1155c8ead5 created by task D:8f9e0052bcc3
Dec  8 09:43:39 xen-730xd xapi: [debug||1164863 HTTP 10.10.10.40->:::80|session.login_with_password D:f8a89f148638|xapi_session] Success: local auth, user root from HTTP request from Internet with User-Agent: unknown
Dec  8 09:43:39 xen-730xd xapi: [ info||1164863 HTTP 10.10.10.40->:::80|session.login_with_password D:f8a89f148638|xapi_session] Session.create trackid=ee78e2f6b58e76414d999de124ea2a35 pool=false uname=root originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec  8 09:43:39 xen-730xd xapi: [debug||1164864 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:cd6ae8df6525 created by task D:f8a89f148638
Dec  8 09:43:39 xen-730xd xapi: [debug||1164868 :::80||api_effect] pool.add_to_other_config
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 HTTP 10.10.10.40->:::80|pool.add_to_other_config D:096f5b4f5c05|sql] Duplicate key in set or map: table pool; field other_config; ref OpaqueRef:95f6dc30-a79e-44c0-8aaa-bbde3a47f2f4; key xo:clientInfo:a1fa4c05-e866-50dd-1254-c1edfe2615ec
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] pool.add_to_other_config D:096f5b4f5c05 failed with exception Db_exn.Duplicate_key("pool", "other_config", "OpaqueRef:95f6dc30-a79e-44c0-8aaa-bbde3a47f2f4", "xo:clientInfo:a1fa4c05-e866-50dd-1254-c1edfe2615ec")
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] Raised Db_exn.Duplicate_key("pool", "other_config", "OpaqueRef:95f6dc30-a79e-44c0-8aaa-bbde3a47f2f4", "xo:clientInfo:a1fa4c05-e866-50dd-1254-c1edfe2615ec")
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 316
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 233
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 101
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 122
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
Dec  8 09:43:39 xen-730xd xapi: [error||1164868 :::80||backtrace]

olivierlambert

Hmm might be unrelated to XO.

Can you export the VM in XVA?

Darkbeldin

Apparently the download failed on my side I think recreating the file after download.
I will retry tomorrow.

sluflyer06

@sluflyer06 XVA worked

Darkbeldin

It's seems I'm able to reproduce the issue with OVA i'm unable to export without an error.

sluflyer06

@Darkbeldin Checking in on if any progress has been made here, i think I tried either on the latest release or the one before and still could not export OVA. Just seeing where things stand with this, there's seems to be a huge amount of focus on being able to bring people INTO xcp-ng with frequent new features and warm migrations and everything but then once we're in we're trapped.

Thanks for all the great stuff.

olivierlambert

"trapped"

I think we fixed a problem on master recently, please try again.

sluflyer06

@olivierlambert I mean, it's a good kind of trapped right, XCP-NG is nice. It's still nice to know you can test other options easily without rebuilding from scratch though.

Will that be in the next official monthly drop?

olivierlambert

If you are using XOA, yes, it will be available in latest in 2 weeks.

If you are using the sources, just git pull from master, yarn && yarn build and you are set

sluflyer06

@olivierlambert Is this recent enough to capture it?
Xen Orchestra, commit a2451

If so, it still fails to export for me.

-Anthony

olivierlambert

It's a recent commit, however the simple answer is be always up to date to HEAD (the latest commit) before reporting a problem

As per https://xen-orchestra.com/docs/community.html#report-a-bug

Otherwise, we lose valuable time to check your commit and understand if something changed before.

sluflyer06

@olivierlambert I think it was latest when I ran it, but i just retried on f012d12 with same effect. I hope I did that right, it is not my intention to waste anyones time.

Edit: Also wanted to add I tried from sources, and also the XOA appliance.

olivierlambert

No problem. Just being sure we are at least talking about the same code when you tried it.

@florent do you also have an error when you try to export a VM in OVA format?

florent

@olivierlambert it seems to work on my side (exporting backup QA ) , but the CPU usage is over the top

I will dig a litlle more

sluflyer06

@florent how big was the VM? Smaller VMs always worked, the larger ones have been the problem. I've had some make it pretty far and get 10s of GB expected before failing but 0 have completed. I've tried XO from sources and also XOA deployed from the website.

I'm happy to try to provide any kind of information that would help

florent

@sluflyer06 It was a small VM, I will test bigger ones

florent

@florent a 10GB export worked
How many disks have you got in this VM ?

sluflyer06

@florent All my VM's only have 1 disk each, my XO vm (10GB) will export but the others range from 40GB up to 100GB and those have never been able to complete.

Danp

I just tried exporting a 20GB VM, and it stopped in the middle of the download. Here is what appeared in the logs --

Apr 24 09:32:42 ubuntuxo xo-server[6111]: 2023-04-24T14:32:42.348Z xo:xo ERROR HTTP request error {
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   data: { filter: undefined, limit: undefined },
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   error: NodeError: Premature close
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.onclose (/opt/xen-orchestra/packages/xo-server/node_modules/readable-stream/lib/internal/streams/end-of-stream.js:116:76)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.emit (node:events:525:35)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at ServerResponse.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at emitCloseNT (node:_http_server:984:10)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.onServerResponseClose (node:_http_server:277:5)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.emit (node:events:525:35)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at Socket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:52:17)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at TCP.<anonymous> (node:net:322:12)
Apr 24 09:32:42 ubuntuxo xo-server[6111]:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17),
Apr 24 09:32:42 ubuntuxo xo-server[6111]:   fn: 'handleGetAllObjects'
Apr 24 09:32:42 ubuntuxo xo-server[6111]: }

The export task is still active / progressing in XO.

Danp

Some additional details --

OVA export on XOA (latest) worked without any issue.
Updated XO from d48ef1f81 to 226f9ad96, and successfully exported the OVA.