VM Migration Failed & Now Can't Access VM

stevewest15

Hi,

I tried to migrate a running VM using XO from across different servers and XO reported 100% complete and then returned the following error:

Migrating Internal error: Xenops_interface.Xenopsd_error([S(Internal_error);S(Xenops_migrate.Remote_failed("unmarshalling error message from remote"))])
Time: 00:03:47	XenServer05	Oct 30, 2020 11:54 AM	Dismiss

I found this citrix support doc.

My question is why can't I access the VM on the source server if the migration failed? XO gives the following errors when I try to restart/shutdown, etc:

INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)

Is my VM lost? I would have thought that with a migration, XO would have kept a copy on the source server and set it in a shutdown state in case of an issue like this.

Thx,

SW

olivierlambert

XO doesn't handle the migration internals, it just orders XAPI to do it.

You might have your VM on the destination with a duplicate UUID, that's why you can only see one. Disconnect the source pool, it should appear on destination pool.

stevewest15

Hi olivierlambert , Thank you for your assistance. The XCP-ng servers are not set in a pool so not sure what you mean by "disconnect the source pool"?

Thanks,

SW

olivierlambert

A server is always in a pool, regardless there's one or more.

So go into settings/server, disconnect one of the two there.

stevewest15

olivierlambert I don't see an option to "Disconnect" in XO > Settings > Servers. I did try change the status to "Disabled" as you can see below but now I don't see the VM listed under Home > VMs.

olivierlambert

Yes, it's Disabled/Enabled, sorry.

Try to do the opposite (first server connected and not the other one).

Anyway, the VM can't disappear, it's somewhere

stevewest15

olivierlambert Tried different combinations of disabling source, target and both. The only way I can see the VM in XO is by enabling the source server.

I even tried removing the VM from the source server so I can restore from backup and I'm getting error:

vm.delete
{
  "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "66366c1a-e0fa-cc12-0ecf-aab05f558613",
    "name_label": "Async.VM.hard_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20201030T18:23:39Z",
    "finished": "20201030T18:23:39Z",
    "status": "failure",
    "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
}

olivierlambert

Do a reset VM power state first.

It means the VM is seen by XAPI as running but it's not.

stevewest15

olivierlambert Tried Force shutdown, Force reboot and both return similar error about "VM does not exist in xenopsd":

vm.stop
{
  "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c",
  "force": true
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "d085e780-7fc9-1e2d-60f1-ec8c144b4cfa",
    "name_label": "Async.VM.hard_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20201030T18:30:41Z",
    "finished": "20201030T18:30:41Z",
    "status": "failure",
    "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
}

stevewest15

When I login to xcp-ng cli, I see this when I run list_domains where the UUID of this VM looks different cli vs what xo shows:

stevewest15

Ok, was able to force shutdown via CLI and now I'm able to start VM on source server:

# xe vm-reset-powerstate vm=<vm name> --force

I'm not going to perform live migrations anytime soon again. I was trying to move VMs so we can do an upgrade which requires wiping out all storage because we used experimental ext4.

olivierlambert

If migration failed, there's a reason. It's usually stable.

Obviously, because your hosts weren't in the pool, a different CPU model can be the problem (they are automatically adapted if they are in the same pool).