Import from VMware fails after upgrade to XOA 5.91
-
So I was testing importing from vsphere and it had been going pretty well (a little slow), and then I saw the news blast with the improvements in 5.91 so naturally I excitedly upgraded. Now it's failing with the 'no opague ref found'. I've seen this in the forums but couldn't find any solution to it. Nothing in the environment really changed except for the XOA update.
vm.importMultipleFromEsxi { "concurrency": 2, "host": "vsphere.nest.local", "network": "7f7d2fcc-c78b-b1c9-101a-0ca9570e3462", "password": "* obfuscated *", "sr": "50d8f945-8ae4-dd87-0149-e6054a10d51f", "sslVerify": false, "stopOnError": true, "stopSource": true, "user": "administrator@vsphere.local", "vms": [ "vm-2427" ] } { "succeeded": {}, "message": "no opaque ref found", "name": "Error", "stack": "Error: no opaque ref found at importVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVm.mjs:28:19) at processTicksAndRejections (node:internal/process/task_queues:95:5) at importVdi (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVdi.mjs:6:17) at file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/migrate-vm.mjs:260:21 at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:158:22) at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:141:20)" }Any ideas on what might be the problem?
vSphere uses iSCSI datastores and XCP-ng is connecting to the SR with NFS. -
Let me ping @florent
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
importVdi
is it a running VM ? does it have snapshots ?
-
@florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
@florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.
Thanksdo you have access to the xo-server logs ? (depending on your install and os , it can be in a specific log file or in journalctl ? If yes, is there any additional information
Was the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?
-
I'm running the default XOA image that was auto-deployed from my XCP-ng node on 'latest' (obv) - so it's debian.
Here's the error block from journalctl:
Feb 01 11:50:18 xoa xo-server[107155]: 2024-02-01T16:50:18.569Z xo:xo-server WARN possibly unhandled rejection { Feb 01 11:50:18 xoa xo-server[107155]: error: Error: already finalized or destroyed Feb 01 11:50:18 xoa xo-server[107155]: at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51) Feb 01 11:50:18 xoa xo-server[107155]: at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6) Feb 01 11:50:18 xoa xo-server[107155]: at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9) Feb 01 11:50:18 xoa xo-server[107155]: at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18) Feb 01 11:50:18 xoa xo-server[107155]: at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10) Feb 01 11:50:18 xoa xo-server[107155]: at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10) Feb 01 11:50:18 xoa xo-server[107155]: at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22) Feb 01 11:50:18 xoa xo-server[107155]: } Feb 01 11:50:20 xoa xo-server[107155]: root@10.96.22.111 Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]) Feb 01 11:50:20 xoa xo-server[107155]: at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12) Feb 01 11:50:20 xoa xo-server[107155]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29) Feb 01 11:50:20 xoa xo-server[107155]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24) Feb 01 11:50:20 xoa xo-server[107155]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14 Feb 01 11:50:20 xoa xo-server[107155]: at Array.forEach (<anonymous>) Feb 01 11:50:20 xoa xo-server[107155]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12) Feb 01 11:50:20 xoa xo-server[107155]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14) { Feb 01 11:50:20 xoa xo-server[107155]: code: 'IMPORT_ERROR', Feb 01 11:50:20 xoa xo-server[107155]: params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ], Feb 01 11:50:20 xoa xo-server[107155]: call: undefined, Feb 01 11:50:20 xoa xo-server[107155]: url: undefined, Feb 01 11:50:20 xoa xo-server[107155]: task: task { Feb 01 11:50:20 xoa xo-server[107155]: uuid: 'd4267c54-3480-a7fe-4d6d-6cc18eef1dc9', Feb 01 11:50:20 xoa xo-server[107155]: name_label: '[XO] VM import', Feb 01 11:50:20 xoa xo-server[107155]: name_description: '', Feb 01 11:50:20 xoa xo-server[107155]: allowed_operations: [], Feb 01 11:50:20 xoa xo-server[107155]: current_operations: {}, Feb 01 11:50:20 xoa xo-server[107155]: created: '20240201T16:44:17Z', Feb 01 11:50:20 xoa xo-server[107155]: finished: '20240201T16:50:23Z', Feb 01 11:50:20 xoa xo-server[107155]: status: 'failure', Feb 01 11:50:20 xoa xo-server[107155]: resident_on: 'OpaqueRef:85a049dc-296e-4ef0-bdbc-82e2845ecd68', Feb 01 11:50:20 xoa xo-server[107155]: progress: 1, Feb 01 11:50:20 xoa xo-server[107155]: type: '<none/>', Feb 01 11:50:20 xoa xo-server[107155]: result: '', Feb 01 11:50:20 xoa xo-server[107155]: error_info: [ Feb 01 11:50:20 xoa xo-server[107155]: 'IMPORT_ERROR', Feb 01 11:50:20 xoa xo-server[107155]: 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' Feb 01 11:50:20 xoa xo-server[107155]: ], Feb 01 11:50:20 xoa xo-server[107155]: other_config: { object_creation: 'complete' }, Feb 01 11:50:20 xoa xo-server[107155]: subtask_of: 'OpaqueRef:NULL', Feb 01 11:50:20 xoa xo-server[107155]: subtasks: [], Feb 01 11:50:20 xoa xo-server[107155]: backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2021))((proces> Feb 01 11:50:20 xoa xo-server[107155]: } Feb 01 11:50:20 xoa xo-server[107155]: } Feb 01 11:50:20 xoa xo-server[107155]: 2024-02-01T16:50:20.571Z xo:api WARN admin | vm.importMultipleFromEsxi(...) [6m] =!> Error: no opaque ref foundWas the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?
No? The VM doesn't show up in XOA if that's what you're asking.
I might need a little hand-holding on doing this part as I'm pretty new to the platform.
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
This means you likely have a disk space problem on your Dom0. Can you do a
df -hin your XCP-ng host? -
This means you likely have a disk space problem on your Dom0. Can you do a
df -hin your XCP-ng host?Sure, it seems ok to me:
[12:21 roentgenium ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.0G 28K 2.0G 1% /dev tmpfs 2.1G 344K 2.1G 1% /dev/shm tmpfs 2.1G 9.4M 2.0G 1% /run tmpfs 2.1G 0 2.1G 0% /sys/fs/cgroup /dev/sda1 18G 2.9G 14G 18% / xenstore 2.1G 0 2.1G 0% /var/lib/xenstored /dev/sda5 3.9G 302M 3.4G 9% /var/log 10.96.25.20:/mnt/seaborgium/XCP/41b662ff-93bd-c386-9dd0-5d08daa3f4e3 39T 1.0M 39T 1% /run/sr-mount/41b662ff-93bd-c386-9dd0-5d08daa3f4e3 10.96.22.19:/mnt/array1/Isos 29T 14T 16T 47% /run/sr-mount/f60530ef-04e2-daf4-36df-c24527814e97 10.96.35.31:/mnt/strontium/XCP/50d8f945-8ae4-dd87-0149-e6054a10d51f 52T 53G 52T 1% /run/sr-mount/50d8f945-8ae4-dd87-0149-e6054a10d51f tmpfs 412M 0 412M 0% /run/user/0 -
And in your XOA?
-
[12:25 01] xoa:log$ df -h Filesystem Size Used Avail Use% Mounted on udev 954M 0 954M 0% /dev tmpfs 195M 528K 195M 1% /run /dev/xvda1 19G 3.1G 15G 18% / tmpfs 973M 0 973M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 195M 0 195M 0% /run/user/1000 -
Weird, it's like the tar stream with ended because of not enough space
But it doesn't make sense reading your available space. -
I started experiencing this exact same error after upgrading to 5.91 yesterday. It seems to affect one of my VMs that has 2 VMDK disks, the first (OS) disk copies over successfully, the 2nd (data) disk fails.
I've tried making a new VMDK and copying my data over, then destroying the old one in case there was a problem with that file but it still hangs around 50% then the task fails.
-
Might worth some investigation on our side, yes.
-
My VM doesn't have multiple vmdks, just the primary 16GB thin provisioned OS drive. Its actual size is only 4.77GB.
1vCPU
2GB RAM
16GB Disk
1 NIC
No dangling snapshots
powered offtruly nothing out of the ordinary.
-
Yeah it seems simple, I wonder what's going on, @florent will take a look. Ideally, if you can open a support ticket so we can take a look remotely, that would be helpful
-
ok, I am testing it with more disks , but the ENOSPC puzzle me for now
-
@jasonmap , @rmaclachlan , can you test this branch : fix_xva_import_thin ?
from this PR : https://github.com/vatesfr/xen-orchestra/pull/7354 -
I would be happy to test - but this is where that "hand holding" would come into play.
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
I would be happy to test - but this is where that "hand holding" would come into play.
no problem . If you have a XOA , please open a support tunnel, I can patch your install
If not:- go to your XO install folder, inside, you should have a lot of files, and also a
@vates, a@xen-orchestraand apackagesfolder git checkout fix_xva_import_thinyarn && yarn build
4 restart yourxo-server
- go to your XO install folder, inside, you should have a lot of files, and also a
-
@florent
In the FWIW department, I get the same errors when trying to import with 5.91:result succeeded {} message "no opaque ref found" name "Error" stack "Error: no opaque ref found\n at importVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVm.mjs:28:19)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at importVdi (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVdi.mjs:6:17)\n at file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/migrate-vm.mjs:260:21\n at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:158:22)\n at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:141:20)"The journalctl errors are as follows :
Feb 02 09:30:19 xoa xo-server[1178375]: 2024-02-02T14:30:19.498Z xo:xo-server WARN possibly unhandled rejection { Feb 02 09:30:19 xoa xo-server[1178375]: error: Error: already finalized or destroyed Feb 02 09:30:19 xoa xo-server[1178375]: at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51) Feb 02 09:30:19 xoa xo-server[1178375]: at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6) Feb 02 09:30:19 xoa xo-server[1178375]: at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9) Feb 02 09:30:19 xoa xo-server[1178375]: at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18) Feb 02 09:30:19 xoa xo-server[1178375]: at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10) Feb 02 09:30:19 xoa xo-server[1178375]: at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10) Feb 02 09:30:19 xoa xo-server[1178375]: at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22) Feb 02 09:30:19 xoa xo-server[1178375]: } Feb 02 09:30:21 xoa xo-server[1178375]: root@xx.xx.xx.xx Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]) Feb 02 09:30:21 xoa xo-server[1178375]: at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12) Feb 02 09:30:21 xoa xo-server[1178375]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29) Feb 02 09:30:21 xoa xo-server[1178375]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24) Feb 02 09:30:21 xoa xo-server[1178375]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14 Feb 02 09:30:21 xoa xo-server[1178375]: at Array.forEach (<anonymous>) Feb 02 09:30:21 xoa xo-server[1178375]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12) Feb 02 09:30:21 xoa xo-server[1178375]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14) Feb 02 09:30:21 xoa xo-server[1178375]: at runNextTicks (node:internal/process/task_queues:60:5) Feb 02 09:30:21 xoa xo-server[1178375]: at processImmediate (node:internal/timers:447:9) Feb 02 09:30:21 xoa xo-server[1178375]: at process.callbackTrampoline (node:internal/async_hooks:130:17) { Feb 02 09:30:21 xoa xo-server[1178375]: code: 'IMPORT_ERROR', Feb 02 09:30:21 xoa xo-server[1178375]: params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ], Feb 02 09:30:21 xoa xo-server[1178375]: call: undefined, Feb 02 09:30:21 xoa xo-server[1178375]: url: undefined, Feb 02 09:30:21 xoa xo-server[1178375]: task: task { Feb 02 09:30:21 xoa xo-server[1178375]: uuid: '37b70fee-d769-1b2a-864b-7e18bd6fcff6', Feb 02 09:30:21 xoa xo-server[1178375]: name_label: '[XO] VM import', Feb 02 09:30:21 xoa xo-server[1178375]: name_description: '', Feb 02 09:30:21 xoa xo-server[1178375]: allowed_operations: [], Feb 02 09:30:21 xoa xo-server[1178375]: current_operations: {}, Feb 02 09:30:21 xoa xo-server[1178375]: created: '20240202T13:57:21Z', Feb 02 09:30:21 xoa xo-server[1178375]: finished: '20240202T14:30:21Z', Feb 02 09:30:21 xoa xo-server[1178375]: status: 'failure', Feb 02 09:30:21 xoa xo-server[1178375]: resident_on: 'OpaqueRef:a482c281-bc5c-6aed-e5cc-91d43c02dc46', Feb 02 09:30:21 xoa xo-server[1178375]: progress: 1, Feb 02 09:30:21 xoa xo-server[1178375]: type: '<none/>', Feb 02 09:30:21 xoa xo-server[1178375]: result: '', Feb 02 09:30:21 xoa xo-server[1178375]: error_info: [ Feb 02 09:30:21 xoa xo-server[1178375]: 'IMPORT_ERROR', Feb 02 09:30:21 xoa xo-server[1178375]: 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' Feb 02 09:30:21 xoa xo-server[1178375]: ], Feb 02 09:30:21 xoa xo-server[1178375]: other_config: { object_creation: 'complete' }, Feb 02 09:30:21 xoa xo-server[1178375]: subtask_of: 'OpaqueRef:NULL', Feb 02 09:30:21 xoa xo-server[1178375]: subtasks: [], Feb 02 09:30:21 xoa xo-server[1178375]: backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2131))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 72)))' Feb 02 09:30:21 xoa xo-server[1178375]: } Feb 02 09:30:21 xoa xo-server[1178375]: }
