XCP-NG 8.3 + rust tools - Unable to shutdown VM

nikade

Im running a virtual XCP-NG 8.3, on that host i've got a Debian 12 VM running with the rust tools and im unable to shutdown the VM from XO with the following error:

vm.stop
{
  "id": "bea5ac37-4198-6db8-20d8-181f82c9f3b1",
  "force": false,
  "bypassBlockedOperation": false
}
{
  "code": "XEN_INCOMPATIBLE",
  "params": [],
  "task": {
    "uuid": "0a9efb1b-7420-b735-900e-a7bafdfdd411",
    "name_label": "Async.VM.clean_shutdown",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20240428T13:40:58Z",
    "finished": "20240428T13:40:58Z",
    "status": "failure",
    "resident_on": "OpaqueRef:bfac3264-adeb-7c37-17d2-d82d5748893f",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "XEN_INCOMPATIBLE"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host_helpers.ml)(line 346))((process xapi)(filename ocaml/xapi/xapi_xenops_queue.ml)(line 29))((process xapi)(filename src/lib/idl.ml)(line 550))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3825))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3823))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3394))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3503))((process xapi)(filename ocaml/xapi/xapi_vm.ml)(line 429))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 143))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2003))((process xapi)(filename ocaml/xapi/rbac.ml)(line 189))((process xapi)(filename ocaml/xapi/rbac.ml)(line 198))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))"
  },
  "message": "XEN_INCOMPATIBLE()",
  "name": "XapiError",
  "stack": "XapiError: XEN_INCOMPATIBLE()
    at Function.wrap (file:///opt/xen-orchestra/xo-builds/xen-orchestra-202403291445/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xen-orchestra/xo-builds/xen-orchestra-202403291445/packages/xen-api/_getTaskResult.mjs:11:29)
    at Xapi._addRecordToCache (file:///opt/xen-orchestra/xo-builds/xen-orchestra-202403291445/packages/xen-api/index.mjs:1029:24)
    at file:///opt/xen-orchestra/xo-builds/xen-orchestra-202403291445/packages/xen-api/index.mjs:1063:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xen-orchestra/xo-builds/xen-orchestra-202403291445/packages/xen-api/index.mjs:1053:12)
    at Xapi._watchEvents (file:///opt/xen-orchestra/xo-builds/xen-orchestra-202403291445/packages/xen-api/index.mjs:1226:14)"
}

VM tools:

Is this expected behaviour? XO version:

Xen Orchestra, commit 8b7e1
Master, commit 7b7a4

olivierlambert

Your issue was unrelated to tools but a Xen upgrade from 4.13 to 4.17 without rebooting it (ie the libs were on Xen 4.17 while the hypervisor still running in memory was 4.13). A reboot solved that.

Danp

@nikade said in XCP-NG 8.3 + rust tools - Unable to shutdown VM:

XEN_INCOMPATIBLE

There have been several threads related to the recent 8.3 updates where xen was updated from v4.13 to v4.17, and the user encountering problems until the host was rebooted. Could this be your issue?

nikade

@Danp said in XCP-NG 8.3 + rust tools - Unable to shutdown VM:

@nikade said in XCP-NG 8.3 + rust tools - Unable to shutdown VM:

XEN_INCOMPATIBLE

There have been several threads related to the recent 8.3 updates where xen was updated from v4.13 to v4.17, and the user encountering problems until the host was rebooted. Could this be your issue?

Yea maybe, but im unable to shutdown the VM.
Console is blank so I cant jump in there and shut it down:

Tried doing it the "oldschool" xenserver-way but it doesnt work:

[15:54 sto-xcp2 ~]# list_domains
Fatal error: exception Xenctrl.Error("13: Permission denied")
[15:54 sto-xcp2 ~]# xe vm-list
uuid ( RO)           : c0bd5156-9fe4-486d-9525-554adb290aef
     name-label ( RW): Control domain on host: sto-xcp2
    power-state ( RO): running


uuid ( RO)           : bea5ac37-4198-6db8-20d8-181f82c9f3b1
     name-label ( RW): DEBIAN 12 test-vm5
    power-state ( RO): running


[15:54 sto-xcp2 ~]# xe vm-shutdown uuid=bea5ac37-4198-6db8-20d8-181f82c9f3b1
The current version of Xen or its control libraries is incompatible with the Toolstack.
[15:54 sto-xcp2 ~]# xe vm-shutdown uuid=bea5ac37-4198-6db8-20d8-181f82c9f3b1 force=true
The current version of Xen or its control libraries is incompatible with the Toolstack.
[15:54 sto-xcp2 ~]# xe vm-reset-powerstate uuid=bea5ac37-4198-6db8-20d8-181f82c9f3b1 force=true
[15:55 sto-xcp2 ~]# xe vm-list
uuid ( RO)           : c0bd5156-9fe4-486d-9525-554adb290aef
     name-label ( RW): Control domain on host: sto-xcp2
    power-state ( RO): running


uuid ( RO)           : bea5ac37-4198-6db8-20d8-181f82c9f3b1
     name-label ( RW): DEBIAN 12 test-vm5
    power-state ( RO): halted

Edit: It actually shutdown after ~1min, so im able to reboot now!

nikade

Works after rebooting!

olivierlambert

Your issue was unrelated to tools but a Xen upgrade from 4.13 to 4.17 without rebooting it (ie the libs were on Xen 4.17 while the hypervisor still running in memory was 4.13). A reboot solved that.