Import from VMware fails after upgrade to XOA 5.91
-
Let me ping @florent
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
importVdi
is it a running VM ? does it have snapshots ?
-
@florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
@florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.
Thanksdo you have access to the xo-server logs ? (depending on your install and os , it can be in a specific log file or in journalctl ? If yes, is there any additional information
Was the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?
-
I'm running the default XOA image that was auto-deployed from my XCP-ng node on 'latest' (obv) - so it's debian.
Here's the error block from journalctl:
Feb 01 11:50:18 xoa xo-server[107155]: 2024-02-01T16:50:18.569Z xo:xo-server WARN possibly unhandled rejection { Feb 01 11:50:18 xoa xo-server[107155]: error: Error: already finalized or destroyed Feb 01 11:50:18 xoa xo-server[107155]: at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51) Feb 01 11:50:18 xoa xo-server[107155]: at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6) Feb 01 11:50:18 xoa xo-server[107155]: at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9) Feb 01 11:50:18 xoa xo-server[107155]: at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18) Feb 01 11:50:18 xoa xo-server[107155]: at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10) Feb 01 11:50:18 xoa xo-server[107155]: at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10) Feb 01 11:50:18 xoa xo-server[107155]: at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22) Feb 01 11:50:18 xoa xo-server[107155]: } Feb 01 11:50:20 xoa xo-server[107155]: root@10.96.22.111 Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]) Feb 01 11:50:20 xoa xo-server[107155]: at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12) Feb 01 11:50:20 xoa xo-server[107155]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29) Feb 01 11:50:20 xoa xo-server[107155]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24) Feb 01 11:50:20 xoa xo-server[107155]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14 Feb 01 11:50:20 xoa xo-server[107155]: at Array.forEach (<anonymous>) Feb 01 11:50:20 xoa xo-server[107155]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12) Feb 01 11:50:20 xoa xo-server[107155]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14) { Feb 01 11:50:20 xoa xo-server[107155]: code: 'IMPORT_ERROR', Feb 01 11:50:20 xoa xo-server[107155]: params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ], Feb 01 11:50:20 xoa xo-server[107155]: call: undefined, Feb 01 11:50:20 xoa xo-server[107155]: url: undefined, Feb 01 11:50:20 xoa xo-server[107155]: task: task { Feb 01 11:50:20 xoa xo-server[107155]: uuid: 'd4267c54-3480-a7fe-4d6d-6cc18eef1dc9', Feb 01 11:50:20 xoa xo-server[107155]: name_label: '[XO] VM import', Feb 01 11:50:20 xoa xo-server[107155]: name_description: '', Feb 01 11:50:20 xoa xo-server[107155]: allowed_operations: [], Feb 01 11:50:20 xoa xo-server[107155]: current_operations: {}, Feb 01 11:50:20 xoa xo-server[107155]: created: '20240201T16:44:17Z', Feb 01 11:50:20 xoa xo-server[107155]: finished: '20240201T16:50:23Z', Feb 01 11:50:20 xoa xo-server[107155]: status: 'failure', Feb 01 11:50:20 xoa xo-server[107155]: resident_on: 'OpaqueRef:85a049dc-296e-4ef0-bdbc-82e2845ecd68', Feb 01 11:50:20 xoa xo-server[107155]: progress: 1, Feb 01 11:50:20 xoa xo-server[107155]: type: '<none/>', Feb 01 11:50:20 xoa xo-server[107155]: result: '', Feb 01 11:50:20 xoa xo-server[107155]: error_info: [ Feb 01 11:50:20 xoa xo-server[107155]: 'IMPORT_ERROR', Feb 01 11:50:20 xoa xo-server[107155]: 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' Feb 01 11:50:20 xoa xo-server[107155]: ], Feb 01 11:50:20 xoa xo-server[107155]: other_config: { object_creation: 'complete' }, Feb 01 11:50:20 xoa xo-server[107155]: subtask_of: 'OpaqueRef:NULL', Feb 01 11:50:20 xoa xo-server[107155]: subtasks: [], Feb 01 11:50:20 xoa xo-server[107155]: backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2021))((proces> Feb 01 11:50:20 xoa xo-server[107155]: } Feb 01 11:50:20 xoa xo-server[107155]: } Feb 01 11:50:20 xoa xo-server[107155]: 2024-02-01T16:50:20.571Z xo:api WARN admin | vm.importMultipleFromEsxi(...) [6m] =!> Error: no opaque ref foundWas the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?
No? The VM doesn't show up in XOA if that's what you're asking.
I might need a little hand-holding on doing this part as I'm pretty new to the platform.
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
This means you likely have a disk space problem on your Dom0. Can you do a
df -hin your XCP-ng host? -
This means you likely have a disk space problem on your Dom0. Can you do a
df -hin your XCP-ng host?Sure, it seems ok to me:
[12:21 roentgenium ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 2.0G 28K 2.0G 1% /dev tmpfs 2.1G 344K 2.1G 1% /dev/shm tmpfs 2.1G 9.4M 2.0G 1% /run tmpfs 2.1G 0 2.1G 0% /sys/fs/cgroup /dev/sda1 18G 2.9G 14G 18% / xenstore 2.1G 0 2.1G 0% /var/lib/xenstored /dev/sda5 3.9G 302M 3.4G 9% /var/log 10.96.25.20:/mnt/seaborgium/XCP/41b662ff-93bd-c386-9dd0-5d08daa3f4e3 39T 1.0M 39T 1% /run/sr-mount/41b662ff-93bd-c386-9dd0-5d08daa3f4e3 10.96.22.19:/mnt/array1/Isos 29T 14T 16T 47% /run/sr-mount/f60530ef-04e2-daf4-36df-c24527814e97 10.96.35.31:/mnt/strontium/XCP/50d8f945-8ae4-dd87-0149-e6054a10d51f 52T 53G 52T 1% /run/sr-mount/50d8f945-8ae4-dd87-0149-e6054a10d51f tmpfs 412M 0 412M 0% /run/user/0 -
And in your XOA?
-
[12:25 01] xoa:log$ df -h Filesystem Size Used Avail Use% Mounted on udev 954M 0 954M 0% /dev tmpfs 195M 528K 195M 1% /run /dev/xvda1 19G 3.1G 15G 18% / tmpfs 973M 0 973M 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 195M 0 195M 0% /run/user/1000 -
Weird, it's like the tar stream with ended because of not enough space
But it doesn't make sense reading your available space. -
I started experiencing this exact same error after upgrading to 5.91 yesterday. It seems to affect one of my VMs that has 2 VMDK disks, the first (OS) disk copies over successfully, the 2nd (data) disk fails.
I've tried making a new VMDK and copying my data over, then destroying the old one in case there was a problem with that file but it still hangs around 50% then the task fails.
-
Might worth some investigation on our side, yes.
-
My VM doesn't have multiple vmdks, just the primary 16GB thin provisioned OS drive. Its actual size is only 4.77GB.
1vCPU
2GB RAM
16GB Disk
1 NIC
No dangling snapshots
powered offtruly nothing out of the ordinary.
-
Yeah it seems simple, I wonder what's going on, @florent will take a look. Ideally, if you can open a support ticket so we can take a look remotely, that would be helpful
-
ok, I am testing it with more disks , but the ENOSPC puzzle me for now
-
@jasonmap , @rmaclachlan , can you test this branch : fix_xva_import_thin ?
from this PR : https://github.com/vatesfr/xen-orchestra/pull/7354 -
I would be happy to test - but this is where that "hand holding" would come into play.
-
@jasonmap said in Import from VMware fails after upgrade to XOA 5.91:
I would be happy to test - but this is where that "hand holding" would come into play.
no problem . If you have a XOA , please open a support tunnel, I can patch your install
If not:- go to your XO install folder, inside, you should have a lot of files, and also a
@vates, a@xen-orchestraand apackagesfolder git checkout fix_xva_import_thinyarn && yarn build
4 restart yourxo-server
- go to your XO install folder, inside, you should have a lot of files, and also a
-
@florent
In the FWIW department, I get the same errors when trying to import with 5.91:result succeeded {} message "no opaque ref found" name "Error" stack "Error: no opaque ref found\n at importVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVm.mjs:28:19)\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at importVdi (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVdi.mjs:6:17)\n at file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/migrate-vm.mjs:260:21\n at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:158:22)\n at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:141:20)"The journalctl errors are as follows :
Feb 02 09:30:19 xoa xo-server[1178375]: 2024-02-02T14:30:19.498Z xo:xo-server WARN possibly unhandled rejection { Feb 02 09:30:19 xoa xo-server[1178375]: error: Error: already finalized or destroyed Feb 02 09:30:19 xoa xo-server[1178375]: at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51) Feb 02 09:30:19 xoa xo-server[1178375]: at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6) Feb 02 09:30:19 xoa xo-server[1178375]: at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9) Feb 02 09:30:19 xoa xo-server[1178375]: at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18) Feb 02 09:30:19 xoa xo-server[1178375]: at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10) Feb 02 09:30:19 xoa xo-server[1178375]: at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10) Feb 02 09:30:19 xoa xo-server[1178375]: at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22) Feb 02 09:30:19 xoa xo-server[1178375]: } Feb 02 09:30:21 xoa xo-server[1178375]: root@xx.xx.xx.xx Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]) Feb 02 09:30:21 xoa xo-server[1178375]: at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12) Feb 02 09:30:21 xoa xo-server[1178375]: at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29) Feb 02 09:30:21 xoa xo-server[1178375]: at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24) Feb 02 09:30:21 xoa xo-server[1178375]: at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14 Feb 02 09:30:21 xoa xo-server[1178375]: at Array.forEach (<anonymous>) Feb 02 09:30:21 xoa xo-server[1178375]: at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12) Feb 02 09:30:21 xoa xo-server[1178375]: at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14) Feb 02 09:30:21 xoa xo-server[1178375]: at runNextTicks (node:internal/process/task_queues:60:5) Feb 02 09:30:21 xoa xo-server[1178375]: at processImmediate (node:internal/timers:447:9) Feb 02 09:30:21 xoa xo-server[1178375]: at process.callbackTrampoline (node:internal/async_hooks:130:17) { Feb 02 09:30:21 xoa xo-server[1178375]: code: 'IMPORT_ERROR', Feb 02 09:30:21 xoa xo-server[1178375]: params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ], Feb 02 09:30:21 xoa xo-server[1178375]: call: undefined, Feb 02 09:30:21 xoa xo-server[1178375]: url: undefined, Feb 02 09:30:21 xoa xo-server[1178375]: task: task { Feb 02 09:30:21 xoa xo-server[1178375]: uuid: '37b70fee-d769-1b2a-864b-7e18bd6fcff6', Feb 02 09:30:21 xoa xo-server[1178375]: name_label: '[XO] VM import', Feb 02 09:30:21 xoa xo-server[1178375]: name_description: '', Feb 02 09:30:21 xoa xo-server[1178375]: allowed_operations: [], Feb 02 09:30:21 xoa xo-server[1178375]: current_operations: {}, Feb 02 09:30:21 xoa xo-server[1178375]: created: '20240202T13:57:21Z', Feb 02 09:30:21 xoa xo-server[1178375]: finished: '20240202T14:30:21Z', Feb 02 09:30:21 xoa xo-server[1178375]: status: 'failure', Feb 02 09:30:21 xoa xo-server[1178375]: resident_on: 'OpaqueRef:a482c281-bc5c-6aed-e5cc-91d43c02dc46', Feb 02 09:30:21 xoa xo-server[1178375]: progress: 1, Feb 02 09:30:21 xoa xo-server[1178375]: type: '<none/>', Feb 02 09:30:21 xoa xo-server[1178375]: result: '', Feb 02 09:30:21 xoa xo-server[1178375]: error_info: [ Feb 02 09:30:21 xoa xo-server[1178375]: 'IMPORT_ERROR', Feb 02 09:30:21 xoa xo-server[1178375]: 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' Feb 02 09:30:21 xoa xo-server[1178375]: ], Feb 02 09:30:21 xoa xo-server[1178375]: other_config: { object_creation: 'complete' }, Feb 02 09:30:21 xoa xo-server[1178375]: subtask_of: 'OpaqueRef:NULL', Feb 02 09:30:21 xoa xo-server[1178375]: subtasks: [], Feb 02 09:30:21 xoa xo-server[1178375]: backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2131))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 72)))' Feb 02 09:30:21 xoa xo-server[1178375]: } Feb 02 09:30:21 xoa xo-server[1178375]: } -
@florent said in Import from VMware fails after upgrade to XOA 5.91:
XO install folder
Where is the "XO install folder"? I did an ssh into the xoa vm and I don't see what you refer to?
I tried running "git checkout fix_xva_import_thin" but it says "fatal: not a git repository (or any of the parent directories): .git"
