VM Migration Failed & Now Can't Access VM

[stevewest15]

Hi,

I tried to migrate a running VM using XO from across different servers and XO reported 100% complete and then returned the following error:

Migrating Internal error: Xenops_interface.Xenopsd_error([S(Internal_error);S(Xenops_migrate.Remote_failed("unmarshalling error message from remote"))])
Time: 00:03:47	XenServer05	Oct 30, 2020 11:54 AM	Dismiss

I found this citrix support doc.

My question is why can't I access the VM on the source server if the migration failed? XO gives the following errors when I try to restart/shutdown, etc:

INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)

Is my VM lost? I would have thought that with a migration, XO would have kept a copy on the source server and set it in a shutdown state in case of an issue like this.

Thx,

SW

[olivierlambert]

XO doesn't handle the migration internals, it just orders XAPI to do it.

You might have your VM on the destination with a duplicate UUID, that's why you can only see one. Disconnect the source pool, it should appear on destination pool.

[stevewest15]

Hi @olivierlambert , Thank you for your assistance. The XCP-ng servers are not set in a pool so not sure what you mean by "disconnect the source pool"?

Thanks,

SW

[olivierlambert]

A server is always in a pool, regardless there's one or more.

So go into settings/server, disconnect one of the two there.

[stevewest15]

@olivierlambert I don't see an option to "Disconnect" in XO > Settings > Servers. I did try change the status to "Disabled" as you can see below but now I don't see the VM listed under Home > VMs.

[olivierlambert]

Yes, it's Disabled/Enabled, sorry.

Try to do the opposite (first server connected and not the other one).

Anyway, the VM can't disappear, it's somewhere

[stevewest15]

@olivierlambert Tried different combinations of disabling source, target and both. The only way I can see the VM in XO is by enabling the source server.

I even tried removing the VM from the source server so I can restore from backup and I'm getting error:

vm.delete
{
  "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "66366c1a-e0fa-cc12-0ecf-aab05f558613",
    "name_label": "Async.VM.hard_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20201030T18:23:39Z",
    "finished": "20201030T18:23:39Z",
    "status": "failure",
    "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
}

[olivierlambert]

Do a reset VM power state first.

It means the VM is seen by XAPI as running but it's not.

[stevewest15]

@olivierlambert Tried Force shutdown, Force reboot and both return similar error about "VM does not exist in xenopsd":

vm.stop
{
  "id": "8acd2276-f779-4cff-5dd3-0664ef98a08c",
  "force": true
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "d085e780-7fc9-1e2d-60f1-ec8c144b4cfa",
    "name_label": "Async.VM.hard_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20201030T18:30:41Z",
    "finished": "20201030T18:30:41Z",
    "status": "failure",
    "resident_on": "OpaqueRef:cd5c6aa1-75fc-4d2d-9c47-4709f7ba8cf2",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_utils.ml)(line 576))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1376))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1481))((process\"xenopsd-xc @ XenServer05\")(filename list.ml)(line 117))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1476))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1619))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1975))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 1986))((process\"xenopsd-xc @ XenServer05\")(filename lib/xenops_server.ml)(line 2000))((process\"xenopsd-xc @ XenServer05\")(filename lib/task_server.ml)(line 159))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2495))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2618))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2869))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2867))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2503))((process\"xapi @ XenServer05\")(filename lib/backtrace.ml)(line 210))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_xenops.ml)(line 2509))((process\"xapi @ XenServer05\")(filename ocaml/xapi/xapi_vm.ml)(line 339))((process\"xapi @ XenServer05\")(filename ocaml/xapi/message_forwarding.ml)(line 100))((process\"xapi @ XenServer05\")(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process\"xapi @ XenServer05\")(filename ocaml/xapi/rbac.ml)(line 236))((process\"xapi @ XenServer05\")(filename ocaml/xapi/server_helpers.ml)(line 83)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 8acd2276-f779-4cff-5dd3-0664ef98a08c/config does not exist in xenopsd)
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:866:24)
    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:887:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:877:12)
    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:1038:14)"
}

[stevewest15]

When I login to xcp-ng cli, I see this when I run list_domains where the UUID of this VM looks different cli vs what xo shows:

[stevewest15]

Ok, was able to force shutdown via CLI and now I'm able to start VM on source server:

# xe vm-reset-powerstate vm=<vm name> --force

I'm not going to perform live migrations anytime soon again. I was trying to move VMs so we can do an upgrade which requires wiping out all storage because we used experimental ext4.

[olivierlambert]

If migration failed, there's a reason. It's usually stable.

Obviously, because your hosts weren't in the pool, a different CPU model can be the problem (they are automatically adapted if they are in the same pool).