VM stuck on old host after migration, non-responsive

mauzilla

The worst result, I migrated a VM from 1 standalone host to another. After some time I went back to see progress and see there are no pending tasks. The VM is not on the new host, and it seems to still "be" on the old host but it's non responsive. Console does nothing, cannot shut it down with an error indicating "Config" for the VM is missing:

Here is the migration log:

vm.migrate
{
  "vm": "429077f8-2582-5932-71f2-27cc30123754",
  "force": true,
  "mapVifsNetworks": {
    "05bc9d14-24b2-42e5-a7df-3bf6b46720dd": "dd8d028b-71a5-b9b8-7efc-4e03261dc90a"
  },
  "migrationNetwork": "48e4aeaf-af9c-dd97-8af6-93e48df0353e",
  "sr": "0d5c70cc-57d4-e770-1b06-11e052cde02c",
  "targetHost": "551aa992-eeab-4ce6-9d9e-87dd3e7551dc"
}
{
  "code": 21,
  "data": {
    "objectId": "429077f8-2582-5932-71f2-27cc30123754",
    "code": "INTERNAL_ERROR"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at operationFailed (/home/xo5/xen-orchestra/packages/xo-common/api-errors.js:26:11)
    at file:///home/xo5/xen-orchestra/packages/xo-server/src/api/vm.mjs:497:15
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)
    at Xo.migrate (file:///home/xo5/xen-orchestra/packages/xo-server/src/api/vm.mjs:483:3)
    at Api.#callApiMethod (file:///home/xo5/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:394:20)"
}

When I try to shutdown the VM on the old host:

vm.stop
{
  "id": "429077f8-2582-5932-71f2-27cc30123754",
  "force": true,
  "bypassBlockedOperation": false
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Object with type VM and id 429077f8-2582-5932-71f2-27cc30123754/config does not exist in xenopsd"
  ],
  "task": {
    "uuid": "3c0f9353-d694-6ccf-8d5c-bf68a508d74a",
    "name_label": "Async.VM.hard_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20230324T13:57:32Z",
    "finished": "20230324T13:57:32Z",
    "status": "failure",
    "resident_on": "OpaqueRef:4649e134-1941-4655-b731-7e96455226f5",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "Object with type VM and id 429077f8-2582-5932-71f2-27cc30123754/config does not exist in xenopsd"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xenopsd-xc)(filename lib/xenops_utils.ml)(line 391))((process xenopsd-xc)(filename lib/xenops_server.ml)(line 1922))((process xenopsd-xc)(filename lib/xenops_server.ml)(line 2064))((process xenopsd-xc)(filename list.ml)(line 121))((process xenopsd-xc)(filename lib/xenops_server.ml)(line 2057))((process xenopsd-xc)(filename lib/xenops_server.ml)(line 2213))((process xenopsd-xc)(filename lib/xenops_server.ml)(line 2746))((process xenopsd-xc)(filename lib/xenops_server.ml)(line 2756))((process xenopsd-xc)(filename lib/xenops_server.ml)(line 2775))((process xenopsd-xc)(filename lib/task_server.ml)(line 177))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3302))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3470))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3742))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3740))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3311))((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3317))((process xapi)(filename ocaml/xapi/xapi_vm.ml)(line 438))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 233))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 104)))"
  },
  "message": "INTERNAL_ERROR(Object with type VM and id 429077f8-2582-5932-71f2-27cc30123754/config does not exist in xenopsd)",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Object with type VM and id 429077f8-2582-5932-71f2-27cc30123754/config does not exist in xenopsd)
    at Function.wrap (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/_XapiError.js:16:12)
    at _default (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:954:24)
    at forEach (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:988:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:978:12)
    at Xapi._watchEvents (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:1144:14)"
}

Any ideas?

Danp

Is the associated disk present on either SR? If yes, then you may be able to create a new VM and attach the disk to it.

Otherwise, you may need to restore from backup to regain a working VM.

mauzilla

2 points

Was entirely my fault, I was warned that the new host did not have a compatible CPU but I continued anyway. I assume that it tried to "continue" on the old host and could not and within this an issue occurred deleting the VM on the new host
I am also still initializing the underlying storage on the "new host" which could have also caused a disk issue, dont know but we are where we are

Good news is this was a non-critical VM, losing it wouldnt be a big issue, however, I want to learn from this experience on how to potentially fix it.

First issue I see is that the VM is now stuck on the origin host as the config files are missing. How can I "remove" this VM so that it no longer shows as a running VM?

Then, I have the VHD file on the original host, it fortunately was not deleted. How do I create a new VM with this disk? I assume the disk is still "mounted" to the broken VM, so where do I start with this process?

dredge999

@mauzilla I have had this same issue. I was forced to restore from backup as Danp suggested. My case was just like yours, destination cpu doesn't support features.

I am pretty sure that I would have needed to shut down the machine prior to the move. In that case it would have moved just fine.

Rick

mauzilla

@dredge999 yeah in hindsight thats what I should have done I wonder if I can restore the config files and have the VM operational again. It's more getting it to stop now without a host restart