XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Import from VMware fails after upgrade to XOA 5.91

    Scheduled Pinned Locked Moved Xen Orchestra
    64 Posts 8 Posters 14.9k Views 9 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • florentF Offline
      florent Vates 🪐 XO Team @jasonmap
      last edited by

      @jasonmap said in Import from VMware fails after upgrade to XOA 5.91:

      @florent No to both questions. It was running when I first attempted it, but naturally it was shutdown. Subsequent retries, I just left it powered off.
      Thanks

      do you have access to the xo-server logs ? (depending on your install and os , it can be in a specific log file or in journalctl ? If yes, is there any additional information

      Was the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?

      A 1 Reply Last reply Reply Quote 0
      • J Offline
        jasonmap
        last edited by

        I'm running the default XOA image that was auto-deployed from my XCP-ng node on 'latest' (obv) - so it's debian.

        Here's the error block from journalctl:

        Feb 01 11:50:18 xoa xo-server[107155]: 2024-02-01T16:50:18.569Z xo:xo-server WARN possibly unhandled rejection {
        Feb 01 11:50:18 xoa xo-server[107155]:   error: Error: already finalized or destroyed
        Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51)
        Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6)
        Feb 01 11:50:18 xoa xo-server[107155]:       at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9)
        Feb 01 11:50:18 xoa xo-server[107155]:       at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18)
        Feb 01 11:50:18 xoa xo-server[107155]:       at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10)
        Feb 01 11:50:18 xoa xo-server[107155]:       at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10)
        Feb 01 11:50:18 xoa xo-server[107155]:       at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22)
        Feb 01 11:50:18 xoa xo-server[107155]: }
        Feb 01 11:50:20 xoa xo-server[107155]: root@10.96.22.111 Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ])
        Feb 01 11:50:20 xoa xo-server[107155]:     at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
        Feb 01 11:50:20 xoa xo-server[107155]:     at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29)
        Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24)
        Feb 01 11:50:20 xoa xo-server[107155]:     at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14
        Feb 01 11:50:20 xoa xo-server[107155]:     at Array.forEach (<anonymous>)
        Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12)
        Feb 01 11:50:20 xoa xo-server[107155]:     at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14) {
        Feb 01 11:50:20 xoa xo-server[107155]:   code: 'IMPORT_ERROR',
        Feb 01 11:50:20 xoa xo-server[107155]:   params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ],
        Feb 01 11:50:20 xoa xo-server[107155]:   call: undefined,
        Feb 01 11:50:20 xoa xo-server[107155]:   url: undefined,
        Feb 01 11:50:20 xoa xo-server[107155]:   task: task {
        Feb 01 11:50:20 xoa xo-server[107155]:     uuid: 'd4267c54-3480-a7fe-4d6d-6cc18eef1dc9',
        Feb 01 11:50:20 xoa xo-server[107155]:     name_label: '[XO] VM import',
        Feb 01 11:50:20 xoa xo-server[107155]:     name_description: '',
        Feb 01 11:50:20 xoa xo-server[107155]:     allowed_operations: [],
        Feb 01 11:50:20 xoa xo-server[107155]:     current_operations: {},
        Feb 01 11:50:20 xoa xo-server[107155]:     created: '20240201T16:44:17Z',
        Feb 01 11:50:20 xoa xo-server[107155]:     finished: '20240201T16:50:23Z',
        Feb 01 11:50:20 xoa xo-server[107155]:     status: 'failure',
        Feb 01 11:50:20 xoa xo-server[107155]:     resident_on: 'OpaqueRef:85a049dc-296e-4ef0-bdbc-82e2845ecd68',
        Feb 01 11:50:20 xoa xo-server[107155]:     progress: 1,
        Feb 01 11:50:20 xoa xo-server[107155]:     type: '<none/>',
        Feb 01 11:50:20 xoa xo-server[107155]:     result: '',
        Feb 01 11:50:20 xoa xo-server[107155]:     error_info: [
        Feb 01 11:50:20 xoa xo-server[107155]:       'IMPORT_ERROR',
        Feb 01 11:50:20 xoa xo-server[107155]:       'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
        Feb 01 11:50:20 xoa xo-server[107155]:     ],
        Feb 01 11:50:20 xoa xo-server[107155]:     other_config: { object_creation: 'complete' },
        Feb 01 11:50:20 xoa xo-server[107155]:     subtask_of: 'OpaqueRef:NULL',
        Feb 01 11:50:20 xoa xo-server[107155]:     subtasks: [],
        Feb 01 11:50:20 xoa xo-server[107155]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2021))((proces>
        Feb 01 11:50:20 xoa xo-server[107155]:   }
        Feb 01 11:50:20 xoa xo-server[107155]: }
        Feb 01 11:50:20 xoa xo-server[107155]: 2024-02-01T16:50:20.571Z xo:api WARN admin | vm.importMultipleFromEsxi(...) [6m] =!> Error: no opaque ref found
        

        Was the VM created ? Can I prepare a specific branch ( with more debug) for you to test ? You'll need to do a git checkout <branch_name> , and then restart xo-server ?

        No? The VM doesn't show up in XOA if that's what you're asking.

        I might need a little hand-holding on doing this part as I'm pretty new to the platform.

        1 Reply Last reply Reply Quote 1
        • olivierlambertO Online
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          @jasonmap said in Import from VMware fails after upgrade to XOA 5.91:

          'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'

          This means you likely have a disk space problem on your Dom0. Can you do a df -h in your XCP-ng host?

          J 1 Reply Last reply Reply Quote 0
          • J Offline
            jasonmap @olivierlambert
            last edited by

            This means you likely have a disk space problem on your Dom0. Can you do a df -h in your XCP-ng host?

            Sure, it seems ok to me:

            [12:21 roentgenium ~]# df -h
            Filesystem                                                            Size  Used Avail Use% Mounted on
            devtmpfs                                                              2.0G   28K  2.0G   1% /dev
            tmpfs                                                                 2.1G  344K  2.1G   1% /dev/shm
            tmpfs                                                                 2.1G  9.4M  2.0G   1% /run
            tmpfs                                                                 2.1G     0  2.1G   0% /sys/fs/cgroup
            /dev/sda1                                                              18G  2.9G   14G  18% /
            xenstore                                                              2.1G     0  2.1G   0% /var/lib/xenstored
            /dev/sda5                                                             3.9G  302M  3.4G   9% /var/log
            10.96.25.20:/mnt/seaborgium/XCP/41b662ff-93bd-c386-9dd0-5d08daa3f4e3   39T  1.0M   39T   1% /run/sr-mount/41b662ff-93bd-c386-9dd0-5d08daa3f4e3
            10.96.22.19:/mnt/array1/Isos                                           29T   14T   16T  47% /run/sr-mount/f60530ef-04e2-daf4-36df-c24527814e97
            10.96.35.31:/mnt/strontium/XCP/50d8f945-8ae4-dd87-0149-e6054a10d51f    52T   53G   52T   1% /run/sr-mount/50d8f945-8ae4-dd87-0149-e6054a10d51f
            tmpfs                                                                 412M     0  412M   0% /run/user/0
            
            
            1 Reply Last reply Reply Quote 0
            • olivierlambertO Online
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              And in your XOA?

              J 1 Reply Last reply Reply Quote 0
              • J Offline
                jasonmap @olivierlambert
                last edited by

                @olivierlambert

                [12:25 01] xoa:log$ df -h
                Filesystem      Size  Used Avail Use% Mounted on
                udev            954M     0  954M   0% /dev
                tmpfs           195M  528K  195M   1% /run
                /dev/xvda1       19G  3.1G   15G  18% /
                tmpfs           973M     0  973M   0% /dev/shm
                tmpfs           5.0M     0  5.0M   0% /run/lock
                tmpfs           195M     0  195M   0% /run/user/1000
                
                
                1 Reply Last reply Reply Quote 0
                • olivierlambertO Online
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  Weird, it's like the tar stream with ended because of not enough space 🤔 But it doesn't make sense reading your available space.

                  1 Reply Last reply Reply Quote 0
                  • R Offline
                    rmaclachlan
                    last edited by

                    I started experiencing this exact same error after upgrading to 5.91 yesterday. It seems to affect one of my VMs that has 2 VMDK disks, the first (OS) disk copies over successfully, the 2nd (data) disk fails.

                    I've tried making a new VMDK and copying my data over, then destroying the old one in case there was a problem with that file but it still hangs around 50% then the task fails.

                    1 Reply Last reply Reply Quote 1
                    • olivierlambertO Online
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      Might worth some investigation on our side, yes.

                      J 1 Reply Last reply Reply Quote 0
                      • J Offline
                        jasonmap @olivierlambert
                        last edited by

                        My VM doesn't have multiple vmdks, just the primary 16GB thin provisioned OS drive. Its actual size is only 4.77GB.

                        1vCPU
                        2GB RAM
                        16GB Disk
                        1 NIC
                        No dangling snapshots
                        powered off

                        truly nothing out of the ordinary.

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Online
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          Yeah it seems simple, I wonder what's going on, @florent will take a look. Ideally, if you can open a support ticket so we can take a look remotely, that would be helpful 🙂

                          1 Reply Last reply Reply Quote 0
                          • florentF Offline
                            florent Vates 🪐 XO Team
                            last edited by

                            ok, I am testing it with more disks , but the ENOSPC puzzle me for now

                            1 Reply Last reply Reply Quote 0
                            • florentF Offline
                              florent Vates 🪐 XO Team
                              last edited by

                              @jasonmap , @rmaclachlan , can you test this branch : fix_xva_import_thin ?
                              from this PR : https://github.com/vatesfr/xen-orchestra/pull/7354

                              fbeauchamp opened this pull request in vatesfr/xen-orchestra

                              closed fix(xo-server/migrate-vms): XVA import - fixes omnibus #7354

                              J 1 Reply Last reply Reply Quote 0
                              • J Offline
                                jasonmap @florent
                                last edited by

                                @florent

                                I would be happy to test - but this is where that "hand holding" would come into play.

                                florentF 1 Reply Last reply Reply Quote 0
                                • florentF Offline
                                  florent Vates 🪐 XO Team @jasonmap
                                  last edited by julien-f

                                  @jasonmap said in Import from VMware fails after upgrade to XOA 5.91:

                                  @florent

                                  I would be happy to test - but this is where that "hand holding" would come into play.

                                  no problem . If you have a XOA , please open a support tunnel, I can patch your install
                                  If not:

                                  1. go to your XO install folder, inside, you should have a lot of files, and also a @vates, a @xen-orchestra and a packages folder
                                  2. git checkout fix_xva_import_thin
                                  3. yarn && yarn build
                                    4 restart your xo-server
                                  A J 2 Replies Last reply Reply Quote 0
                                  • A Offline
                                    archw @florent
                                    last edited by julien-f

                                    @florent
                                    In the FWIW department, I get the same errors when trying to import with 5.91:

                                    result	
                                    succeeded	{}
                                    message	"no opaque ref found"
                                    name	"Error"
                                    stack	"Error: no opaque ref found\n    at importVm (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVm.mjs:28:19)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at importVdi (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/importVdi.mjs:6:17)\n    at file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/migrate-vm.mjs:260:21\n    at Task.runInside (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:158:22)\n    at Task.run (/usr/local/lib/node_modules/xo-server/node_modules/@vates/task/index.js:141:20)"
                                    

                                    The journalctl errors are as follows :

                                    Feb 02 09:30:19 xoa xo-server[1178375]: 2024-02-02T14:30:19.498Z xo:xo-server WARN possibly unhandled rejection {
                                    Feb 02 09:30:19 xoa xo-server[1178375]:   error: Error: already finalized or destroyed
                                    Feb 02 09:30:19 xoa xo-server[1178375]:       at Pack.entry (/usr/local/lib/node_modules/xo-server/node_modules/tar-stream/pack.js:138:51)
                                    Feb 02 09:30:19 xoa xo-server[1178375]:       at Pack.resolver (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:5:6)
                                    Feb 02 09:30:19 xoa xo-server[1178375]:       at Promise._execute (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/debuggability.js:384:9)
                                    Feb 02 09:30:19 xoa xo-server[1178375]:       at Promise._resolveFromExecutor (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:518:18)
                                    Feb 02 09:30:19 xoa xo-server[1178375]:       at new Promise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:103:10)
                                    Feb 02 09:30:19 xoa xo-server[1178375]:       at Pack.fromCallback (/usr/local/lib/node_modules/xo-server/node_modules/promise-toolbox/fromCallback.js:9:10)
                                    Feb 02 09:30:19 xoa xo-server[1178375]:       at writeBlock (file:///usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/xva/_writeDisk.mjs:11:22)
                                    Feb 02 09:30:19 xoa xo-server[1178375]: }
                                    Feb 02 09:30:21 xoa xo-server[1178375]: root@xx.xx.xx.xx Xapi#putResource /import/ XapiError: IMPORT_ERROR(INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ])
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:11:29)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1006:24)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1040:14
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at Array.forEach (<anonymous>)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1030:12)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1203:14)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at runNextTicks (node:internal/process/task_queues:60:5)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at processImmediate (node:internal/timers:447:9)
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     at process.callbackTrampoline (node:internal/async_hooks:130:17) {
                                    Feb 02 09:30:21 xoa xo-server[1178375]:   code: 'IMPORT_ERROR',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:   params: [ 'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]' ],
                                    Feb 02 09:30:21 xoa xo-server[1178375]:   call: undefined,
                                    Feb 02 09:30:21 xoa xo-server[1178375]:   url: undefined,
                                    Feb 02 09:30:21 xoa xo-server[1178375]:   task: task {
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     uuid: '37b70fee-d769-1b2a-864b-7e18bd6fcff6',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     name_label: '[XO] VM import',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     name_description: '',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     allowed_operations: [],
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     current_operations: {},
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     created: '20240202T13:57:21Z',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     finished: '20240202T14:30:21Z',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     status: 'failure',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     resident_on: 'OpaqueRef:a482c281-bc5c-6aed-e5cc-91d43c02dc46',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     progress: 1,
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     type: '<none/>',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     result: '',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     error_info: [
                                    Feb 02 09:30:21 xoa xo-server[1178375]:       'IMPORT_ERROR',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:       'INTERNAL_ERROR: [ Unix.Unix_error(Unix.ENOSPC, "write", "") ]'
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     ],
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     other_config: { object_creation: 'complete' },
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     subtask_of: 'OpaqueRef:NULL',
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     subtasks: [],
                                    Feb 02 09:30:21 xoa xo-server[1178375]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/import.ml)(line 2131))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 72)))'
                                    Feb 02 09:30:21 xoa xo-server[1178375]:   }
                                    Feb 02 09:30:21 xoa xo-server[1178375]: }
                                    
                                    1 Reply Last reply Reply Quote 1
                                    • A Offline
                                      archw @florent
                                      last edited by archw

                                      @florent said in Import from VMware fails after upgrade to XOA 5.91:

                                      XO install folder

                                      Where is the "XO install folder"? I did an ssh into the xoa vm and I don't see what you refer to?
                                      I tried running "git checkout fix_xva_import_thin" but it says "fatal: not a git repository (or any of the parent directories): .git"

                                      florentF 1 Reply Last reply Reply Quote 0
                                      • florentF Offline
                                        florent Vates 🪐 XO Team @archw
                                        last edited by

                                        @archw said in Import from VMware fails after upgrade to XOA 5.91:

                                        @florent said in Import from VMware fails after upgrade to XOA 5.91:

                                        XO install folder

                                        Where is the "XO install folder"? I did an ssh into the xoa vm and I don't see what you refer to?
                                        I tried running "git checkout fix_xva_import_thin" but it says "fatal: not a git repository (or any of the parent directories): .git"

                                        if it's an XOA, even a starter/free one, open a support tunnel, I will take care of it
                                        If not, how did you install your XO from source ?

                                        A 2 Replies Last reply Reply Quote 0
                                        • A Offline
                                          archw @florent
                                          last edited by

                                          @florent
                                          The one I showed above what the starter version. Since I made that post, I also tried running it on the one I built from source and it is version "Master, commit 09247". It has the same errors.

                                          I just opened a support tunnel.

                                          1 Reply Last reply Reply Quote 0
                                          • A Offline
                                            archw @florent
                                            last edited by

                                            @florent I just sent you tunnell id

                                            florentF 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post