XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Import from VMware fails after upgrade to XOA 5.91

    Scheduled Pinned Locked Moved Xen Orchestra
    64 Posts 8 Posters 12.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jasonmap
      last edited by

      So I was testing importing from vsphere and it had been going pretty well (a little slow), and then I saw the news blast with the improvements in 5.91 so naturally I excitedly upgraded. Now it's failing with the 'no opague ref found'. I've seen this in the forums but couldn't find any solution to it. Nothing in the environment really changed except for the XOA update.

      vm.importMultipleFromEsxi
{
  "concurrency": 2,
  "host": "vsphere.nest.local",
  "network": "7f7d2fcc-c78b-b1c9-101a-0ca9570e3462",
  "password": "* obfuscated *",
  "sr": "50d8f945-8ae4-dd87-0149-e6054a10d51f",
  "sslVerify": false,
  "stopOnError": true,
  "stopSource": true,
  "user": "administrator@vsphere.local",
  "vms": [
    "vm-2427"
  ]
}
{
  "succeeded": {},
  "message": "no opaque ref found",
  "name": "Error",
  "stack": "Error: no opaque ref found
    at importVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVm.mjs:28:19)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at importVdi (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVdi.mjs:6:17)
    at file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/migrate-vm.mjs:260:21
    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:158:22)
    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:141:20)"
}

      Any ideas on what might be the problem?
      vSphere uses iSCSI datastores and XCP-ng is connecting to the SR with NFS.

      florentF 1 Reply Last reply Reply Quote 0
      • olivierlambertO
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Let me ping @florent 🙂

        1 Reply Last reply Reply Quote 0
        • florentF
          florent Vates 🪐 XO Team @jasonmap
          last edited by

          @jasonmap said in Import from VMware fails after upgrade to XOA 5.91:

          importVdi

          is it a running VM ? does it have snapshots ?

          J 1 Reply Last reply Reply Quote 0
          • J
            jasonmap @florent
            last edited by

            @florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.

            florentF 1 Reply Last reply Reply Quote 0
            • florentF
              florent Vates 🪐 XO Team @jasonmap
              last edited by

              @jasonmap said in Import from VMware fails after upgrade to XOA 5.91:

              @florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.
              Thanks

              do you have access to the xo-server logs ? (depending on your install and os , it can be in a specific log file or in journalctl ? If yes, is there any additional information

              Was the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?

              A 1 Reply Last reply Reply Quote 0
              • J
                jasonmap
                last edited by

                I'm running the default XOA image that was auto-deployed from my XCP-ng node on 'latest' (obv) - so it's debian.

                Here's the error block from journalctl:

                Feb 01 11:50:18 xoa xo-server[107155]: 2024-02-01T16:50:18.569Z xo:xo-server WARN possibly unhandled rejection {
Feb 01 11:50:18 xoa xo-server[107155]:   error: Error: already finalized or destroyed
Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51)
Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6)
Feb 01 11:50:18 xoa xo-server[107155]:       at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9)
Feb 01 11:50:18 xoa xo-server[107155]:       at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18)
Feb 01 11:50:18 xoa xo-server[107155]:       at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10)
Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10)
Feb 01 11:50:18 xoa xo-server[107155]:       at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22)
Feb 01 11:50:18 xoa xo-server[107155]: }
Feb 01 11:50:20 xoa xo-server[107155]: root@10.96.22.111 Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ])
Feb 01 11:50:20 xoa xo-server[107155]:     at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Feb 01 11:50:20 xoa xo-server[107155]:     at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29)
Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24)
Feb 01 11:50:20 xoa xo-server[107155]:     at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14
Feb 01 11:50:20 xoa xo-server[107155]:     at Array.forEach (<anonymous>)
Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12)
Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14) {
Feb 01 11:50:20 xoa xo-server[107155]:   code: 'IMPORT_ERROR',
Feb 01 11:50:20 xoa xo-server[107155]:   params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ],
Feb 01 11:50:20 xoa xo-server[107155]:   call: undefined,
Feb 01 11:50:20 xoa xo-server[107155]:   url: undefined,
Feb 01 11:50:20 xoa xo-server[107155]:   task: task {
Feb 01 11:50:20 xoa xo-server[107155]:     uuid: 'd4267c54-3480-a7fe-4d6d-6cc18eef1dc9',
Feb 01 11:50:20 xoa xo-server[107155]:     name_label: '[XO] VM import',
Feb 01 11:50:20 xoa xo-server[107155]:     name_description: '',
Feb 01 11:50:20 xoa xo-server[107155]:     allowed_operations: [],
Feb 01 11:50:20 xoa xo-server[107155]:     current_operations: {},
Feb 01 11:50:20 xoa xo-server[107155]:     created: '20240201T16:44:17Z',
Feb 01 11:50:20 xoa xo-server[107155]:     finished: '20240201T16:50:23Z',
Feb 01 11:50:20 xoa xo-server[107155]:     status: 'failure',
Feb 01 11:50:20 xoa xo-server[107155]:     resident_on: 'OpaqueRef:85a049dc-296e-4ef0-bdbc-82e2845ecd68',
Feb 01 11:50:20 xoa xo-server[107155]:     progress: 1,
Feb 01 11:50:20 xoa xo-server[107155]:     type: '<none/>',
Feb 01 11:50:20 xoa xo-server[107155]:     result: '',
Feb 01 11:50:20 xoa xo-server[107155]:     error_info: [
Feb 01 11:50:20 xoa xo-server[107155]:       'IMPORT_ERROR',
Feb 01 11:50:20 xoa xo-server[107155]:       'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
Feb 01 11:50:20 xoa xo-server[107155]:     ],
Feb 01 11:50:20 xoa xo-server[107155]:     other_config: { object_creation: 'complete' },
Feb 01 11:50:20 xoa xo-server[107155]:     subtask_of: 'OpaqueRef:NULL',
Feb 01 11:50:20 xoa xo-server[107155]:     subtasks: [],
Feb 01 11:50:20 xoa xo-server[107155]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2021))((proces>
Feb 01 11:50:20 xoa xo-server[107155]:   }
Feb 01 11:50:20 xoa xo-server[107155]: }
Feb 01 11:50:20 xoa xo-server[107155]: 2024-02-01T16:50:20.571Z xo:api WARN admin | vm.importMultipleFromEsxi(...) [6m] =!> Error: no opaque ref found

                Was the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?

                No? The VM doesn't show up in XOA if that's what you're asking.

                I might need a little hand-holding on doing this part as I'm pretty new to the platform.

                1 Reply Last reply Reply Quote 1
                • olivierlambertO
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  @jasonmap said in Import from VMware fails after upgrade to XOA 5.91:

                  'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'

                  This means you likely have a disk space problem on your Dom0. Can you do a df -h in your XCP-ng host?

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    jasonmap @olivierlambert
                    last edited by

                    This means you likely have a disk space problem on your Dom0. Can you do a df -h in your XCP-ng host?

                    Sure, it seems ok to me:

                    [12:21 roentgenium ~]# df -h
Filesystem                                                            Size  Used Avail Use% Mounted on
devtmpfs                                                              2.0G   28K  2.0G   1% /dev
tmpfs                                                                 2.1G  344K  2.1G   1% /dev/shm
tmpfs                                                                 2.1G  9.4M  2.0G   1% /run
tmpfs                                                                 2.1G     0  2.1G   0% /sys/fs/cgroup
/dev/sda1                                                              18G  2.9G   14G  18% /
xenstore                                                              2.1G     0  2.1G   0% /var/lib/xenstored
/dev/sda5                                                             3.9G  302M  3.4G   9% /var/log
10.96.25.20:/mnt/seaborgium/XCP/41b662ff-93bd-c386-9dd0-5d08daa3f4e3   39T  1.0M   39T   1% /run/sr-mount/41b662ff-93bd-c386-9dd0-5d08daa3f4e3
10.96.22.19:/mnt/array1/Isos                                           29T   14T   16T  47% /run/sr-mount/f60530ef-04e2-daf4-36df-c24527814e97
10.96.35.31:/mnt/strontium/XCP/50d8f945-8ae4-dd87-0149-e6054a10d51f    52T   53G   52T   1% /run/sr-mount/50d8f945-8ae4-dd87-0149-e6054a10d51f
tmpfs                                                                 412M     0  412M   0% /run/user/0
                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      And in your XOA?

                      J 1 Reply Last reply Reply Quote 0
                      • J
                        jasonmap @olivierlambert
                        last edited by

                        @olivierlambert

                        [12:25 01] xoa:log$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            954M     0  954M   0% /dev
tmpfs           195M  528K  195M   1% /run
/dev/xvda1       19G  3.1G   15G  18% /
tmpfs           973M     0  973M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           195M     0  195M   0% /run/user/1000
                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          Weird, it's like the tar stream with ended because of not enough space 🤔 But it doesn't make sense reading your available space.

                          1 Reply Last reply Reply Quote 0
                          • R
                            rmaclachlan
                            last edited by

                            I started experiencing this exact same error after upgrading to 5.91 yesterday. It seems to affect one of my VMs that has 2 VMDK disks, the first (OS) disk copies over successfully, the 2nd (data) disk fails.

                            I've tried making a new VMDK and copying my data over, then destroying the old one in case there was a problem with that file but it still hangs around 50% then the task fails.

                            1 Reply Last reply Reply Quote 1
                            • olivierlambertO
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Might worth some investigation on our side, yes.

                              J 1 Reply Last reply Reply Quote 0
                              • J
                                jasonmap @olivierlambert
                                last edited by

                                My VM doesn't have multiple vmdks, just the primary 16GB thin provisioned OS drive. Its actual size is only 4.77GB.

                                1vCPU
                                2GB RAM
                                16GB Disk
                                1 NIC
                                No dangling snapshots
                                powered off

                                truly nothing out of the ordinary.

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  Yeah it seems simple, I wonder what's going on, @florent will take a look. Ideally, if you can open a support ticket so we can take a look remotely, that would be helpful 🙂

                                  1 Reply Last reply Reply Quote 0
                                  • florentF
                                    florent Vates 🪐 XO Team
                                    last edited by

                                    ok, I am testing it with more disks , but the ENOSPC puzzle me for now

                                    1 Reply Last reply Reply Quote 0
                                    • florentF
                                      florent Vates 🪐 XO Team
                                      last edited by

                                      @jasonmap , @rmaclachlan , can you test this branch : fix_xva_import_thin ?
                                      from this PR : https://github.com/vatesfr/xen-orchestra/pull/7354

                                      fbeauchamp opened this pull request in vatesfr/xen-orchestra

                                      closed fix(xo-server/migrate-vms): XVA import - fixes omnibus #7354

                                      J 1 Reply Last reply Reply Quote 0
                                      • J
                                        jasonmap @florent
                                        last edited by

                                        @florent

                                        I would be happy to test - but this is where that "hand holding" would come into play.

                                        florentF 1 Reply Last reply Reply Quote 0
                                        • florentF
                                          florent Vates 🪐 XO Team @jasonmap
                                          last edited by julien-f

                                          @jasonmap said in Import from VMware fails after upgrade to XOA 5.91:

                                          @florent

                                          I would be happy to test - but this is where that "hand holding" would come into play.

                                          no problem . If you have a XOA , please open a support tunnel, I can patch your install
                                          If not:

                                          1. go to your XO install folder, inside, you should have a lot of files, and also a @vates, a @xen-orchestra and a packages folder
                                          2. git checkout fix_xva_import_thin
                                          3. yarn && yarn build
                                            4 restart your xo-server
                                          A J 2 Replies Last reply Reply Quote 0
                                          • A
                                            archw @florent
                                            last edited by julien-f

                                            @florent
                                            In the FWIW department, I get the same errors when trying to import with 5.91:

                                            result	
succeeded	{}
message	"no opaque ref found"
name	"Error"
stack	"Error: no opaque ref found\n    at importVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVm.mjs:28:19)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at importVdi (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVdi.mjs:6:17)\n    at file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/migrate-vm.mjs:260:21\n    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:158:22)\n    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:141:20)"

                                            The journalctl errors are as follows :

                                            Feb 02 09:30:19 xoa xo-server[1178375]: 2024-02-02T14:30:19.498Z xo:xo-server WARN possibly unhandled rejection {
Feb 02 09:30:19 xoa xo-server[1178375]:   error: Error: already finalized or destroyed
Feb 02 09:30:19 xoa xo-server[1178375]:       at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51)
Feb 02 09:30:19 xoa xo-server[1178375]:       at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6)
Feb 02 09:30:19 xoa xo-server[1178375]:       at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9)
Feb 02 09:30:19 xoa xo-server[1178375]:       at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18)
Feb 02 09:30:19 xoa xo-server[1178375]:       at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10)
Feb 02 09:30:19 xoa xo-server[1178375]:       at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10)
Feb 02 09:30:19 xoa xo-server[1178375]:       at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22)
Feb 02 09:30:19 xoa xo-server[1178375]: }
Feb 02 09:30:21 xoa xo-server[1178375]: root@xx.xx.xx.xx Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ])
Feb 02 09:30:21 xoa xo-server[1178375]:     at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
Feb 02 09:30:21 xoa xo-server[1178375]:     at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29)
Feb 02 09:30:21 xoa xo-server[1178375]:     at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24)
Feb 02 09:30:21 xoa xo-server[1178375]:     at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14
Feb 02 09:30:21 xoa xo-server[1178375]:     at Array.forEach (<anonymous>)
Feb 02 09:30:21 xoa xo-server[1178375]:     at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12)
Feb 02 09:30:21 xoa xo-server[1178375]:     at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14)
Feb 02 09:30:21 xoa xo-server[1178375]:     at runNextTicks (node:internal/process/task_queues:60:5)
Feb 02 09:30:21 xoa xo-server[1178375]:     at processImmediate (node:internal/timers:447:9)
Feb 02 09:30:21 xoa xo-server[1178375]:     at process.callbackTrampoline (node:internal/async_hooks:130:17) {
Feb 02 09:30:21 xoa xo-server[1178375]:   code: 'IMPORT_ERROR',
Feb 02 09:30:21 xoa xo-server[1178375]:   params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ],
Feb 02 09:30:21 xoa xo-server[1178375]:   call: undefined,
Feb 02 09:30:21 xoa xo-server[1178375]:   url: undefined,
Feb 02 09:30:21 xoa xo-server[1178375]:   task: task {
Feb 02 09:30:21 xoa xo-server[1178375]:     uuid: '37b70fee-d769-1b2a-864b-7e18bd6fcff6',
Feb 02 09:30:21 xoa xo-server[1178375]:     name_label: '[XO] VM import',
Feb 02 09:30:21 xoa xo-server[1178375]:     name_description: '',
Feb 02 09:30:21 xoa xo-server[1178375]:     allowed_operations: [],
Feb 02 09:30:21 xoa xo-server[1178375]:     current_operations: {},
Feb 02 09:30:21 xoa xo-server[1178375]:     created: '20240202T13:57:21Z',
Feb 02 09:30:21 xoa xo-server[1178375]:     finished: '20240202T14:30:21Z',
Feb 02 09:30:21 xoa xo-server[1178375]:     status: 'failure',
Feb 02 09:30:21 xoa xo-server[1178375]:     resident_on: 'OpaqueRef:a482c281-bc5c-6aed-e5cc-91d43c02dc46',
Feb 02 09:30:21 xoa xo-server[1178375]:     progress: 1,
Feb 02 09:30:21 xoa xo-server[1178375]:     type: '<none/>',
Feb 02 09:30:21 xoa xo-server[1178375]:     result: '',
Feb 02 09:30:21 xoa xo-server[1178375]:     error_info: [
Feb 02 09:30:21 xoa xo-server[1178375]:       'IMPORT_ERROR',
Feb 02 09:30:21 xoa xo-server[1178375]:       'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
Feb 02 09:30:21 xoa xo-server[1178375]:     ],
Feb 02 09:30:21 xoa xo-server[1178375]:     other_config: { object_creation: 'complete' },
Feb 02 09:30:21 xoa xo-server[1178375]:     subtask_of: 'OpaqueRef:NULL',
Feb 02 09:30:21 xoa xo-server[1178375]:     subtasks: [],
Feb 02 09:30:21 xoa xo-server[1178375]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2131))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 72)))'
Feb 02 09:30:21 xoa xo-server[1178375]:   }
Feb 02 09:30:21 xoa xo-server[1178375]: }
                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post