XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    VM Migration Failed & Now Can't Access VM

    Scheduled Pinned Locked Moved Xen Orchestra
    12 Posts 2 Posters 5.2k Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      stevewest15
      last edited by

      Hi @olivierlambert , Thank you for your assistance. The XCP-ng servers are not set in a pool so not sure what you mean by "disconnect the source pool"?

      Thanks,

      SW

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        A server is always in a pool, regardless there's one or more.

        So go into settings/server, disconnect one of the two there.

        1 Reply Last reply Reply Quote 0
        • S Offline
          stevewest15
          last edited by

          @olivierlambert I don't see an option to "Disconnect" in XO > Settings > Servers. I did try change the status to "Disabled" as you can see below but now I don't see the VM listed under Home > VMs.
          83b13f83-f7ed-45ff-af62-6079c270782a-image.png

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Yes, it's Disabled/Enabled, sorry.

            Try to do the opposite (first server connected and not the other one).

            Anyway, the VM can't disappear, it's somewhere

            1 Reply Last reply Reply Quote 0
            • S Offline
              stevewest15
              last edited by

              @olivierlambert Tried different combinations of disabling source, target and both. The only way I can see the VM in XO is by enabling the source server.

              I even tried removing the VM from the source server so I can restore from backup and I'm getting error:

              vm.delete
              {
                "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c"
              }
              {
                "code": "INTERNAL_ERROR",
                "params": [
                  "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
                ],
                "task": {
                  "uuid": "66366c1a-e0fa-cc12-0ecf-aab05f558613",
                  "name_label": "Async.VM.hard_shutdown",
                  "name_description": "",
                  "allowed_operations": [],
                  "current_operations": {},
                  "created": "20201030T18:23:39Z",
                  "finished": "20201030T18:23:39Z",
                  "status": "failure",
                  "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
                  "progress": 1,
                  "type": "<none/>",
                  "result": "",
                  "error_info": [
                    "INTERNAL_ERROR",
                    "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
                  ],
                  "other_config": {},
                  "subtask_of": "OpaqueRef:NULL",
                  "subtasks": [],
                  "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
                },
                "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
                "name": "XapiError",
                "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
                  at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
                  at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
                  at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
                  at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
                  at Array.forEach (<anonymous>)
                  at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
                  at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
              }
              
              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Do a reset VM power state first.

                It means the VM is seen by XAPI as running but it's not.

                1 Reply Last reply Reply Quote 0
                • S Offline
                  stevewest15
                  last edited by

                  @olivierlambert Tried Force shutdown, Force reboot and both return similar error about "VM does not exist in xenopsd":

                  vm.stop
                  {
                    "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c",
                    "force": true
                  }
                  {
                    "code": "INTERNAL_ERROR",
                    "params": [
                      "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
                    ],
                    "task": {
                      "uuid": "d085e780-7fc9-1e2d-60f1-ec8c144b4cfa",
                      "name_label": "Async.VM.hard_shutdown",
                      "name_description": "",
                      "allowed_operations": [],
                      "current_operations": {},
                      "created": "20201030T18:30:41Z",
                      "finished": "20201030T18:30:41Z",
                      "status": "failure",
                      "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
                      "progress": 1,
                      "type": "<none/>",
                      "result": "",
                      "error_info": [
                        "INTERNAL_ERROR",
                        "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
                      ],
                      "other_config": {},
                      "subtask_of": "OpaqueRef:NULL",
                      "subtasks": [],
                      "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
                    },
                    "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
                    "name": "XapiError",
                    "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
                      at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
                      at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
                      at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
                      at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
                      at Array.forEach (<anonymous>)
                      at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
                      at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
                  }
                  
                  1 Reply Last reply Reply Quote 0
                  • S Offline
                    stevewest15
                    last edited by

                    When I login to xcp-ng cli, I see this when I run list_domains where the UUID of this VM looks different cli vs what xo shows:

                    c705bf2f-fc57-492e-baac-2afff2e8fa23-image.png

                    1 Reply Last reply Reply Quote 1
                    • S Offline
                      stevewest15
                      last edited by

                      Ok, was able to force shutdown via CLI and now I'm able to start VM on source server:

                      # xe vm-reset-powerstate vm=<vm name> --force
                      

                      I'm not going to perform live migrations anytime soon again. I was trying to move VMs so we can do an upgrade which requires wiping out all storage because we used experimental ext4.

                      1 Reply Last reply Reply Quote 3
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        If migration failed, there's a reason. It's usually stable.

                        Obviously, because your hosts weren't in the pool, a different CPU model can be the problem (they are automatically adapted if they are in the same pool).

                        1 Reply Last reply Reply Quote 1
                        • First post
                          Last post