Try to shutdown host results in NO_HOSTS_AVAILABLE

ChuckNorrison

For maintenance reasons, i wanted to shutdown my single host via xo webinterface. It results in NO_HOSTS_AVAILABLE. So i used the shutdown -h now command via ssh. After power on everything works. The host has logged a PBD_PLUG_FAILED_ON_SERVER_START.

host.restart
{
  "id": "9b239e09-8b49-4925-96bc-55135b1cfe94",
  "force": false
}
{
  "code": "NO_HOSTS_AVAILABLE",
  "params": [
    "OpaqueRef:8c31fcb1-c3ee-4747-8235-7ace5b9cc7b1"
  ],
  "task": {
    "uuid": "c8278134-cdf1-61e3-e593-3fdfc26b6061",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20210516T15:15:04Z",
    "finished": "20210516T15:15:04Z",
    "status": "failure",
    "resident_on": "OpaqueRef:573d21ff-edcd-4a6a-a2f6-1fd88d7c3537",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "NO_HOSTS_AVAILABLE",
      "OpaqueRef:8c31fcb1-c3ee-4747-8235-7ace5b9cc7b1"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 560))((process xapi)(filename hashtbl.ml)(line 266))((process xapi)(filename hashtbl.ml)(line 272))((process xapi)(filename hashtbl.ml)(line 277))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 556))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
  },
  "message": "NO_HOSTS_AVAILABLE(OpaqueRef:8c31fcb1-c3ee-4747-8235-7ace5b9cc7b1)",
  "name": "XapiError",
  "stack": "XapiError: NO_HOSTS_AVAILABLE(OpaqueRef:8c31fcb1-c3ee-4747-8235-7ace5b9cc7b1)
    at Function.wrap (/opt/xen-orchestra/packages/xen-api/src/_XapiError.js:16:12)
    at _default (/opt/xen-orchestra/packages/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/opt/xen-orchestra/packages/xen-api/src/index.js:806:24)
    at forEach (/opt/xen-orchestra/packages/xen-api/src/index.js:827:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:817:12)
    at Xapi._watchEvents (/opt/xen-orchestra/packages/xen-api/src/index.js:975:14)"
}

Can someone help to understand and maybe improve?

olivierlambert

If you don't force reboot/shutdown (so doing a "soft" reboot/shutdown), XAPI will try to live migrate VMs to another host.

And because you don't have another host, XAPI tells your "there's no hosts available to send those VMs".

So in XO, host/advanced, use the force reboot button (I don't remember if we got a force shutdown but at worse shutdown your VMs or shutdown from the host itself as you did with shutdown -h now)

ChuckNorrison

@olivierlambert said in Try to shutdown host results in NO_HOSTS_AVAILABLE:

If you don't force reboot/shutdown (so doing a "soft" reboot/shutdown), XAPI will try to live migrate VMs to another host.

And because you don't have another host, XAPI tells your "there's no hosts available to send those VMs".

ok thanks.

So best practice to shutdown a single host is: Stop VMs and manually shutdown host from console.

Should i stop something else from xenserver prior shutdown host?

olivierlambert

shutdown -h is enough.

hrv3e

@olivierlambert

What happens when I do it like this

ssh to DOM
run xsconsole
- select Reboot or Shutdown
  - select Reboot Server

Will XAPI try something like migration or is this like shutdown -h on DOM?

Note: I have only one host.

olivierlambert

Migrating to where if you have only one host?

gskger

@hrv3e In case I have to shut down my two host playlab, I usually

shutdown all VMs except XO from source from XO
shutdown secondary host from XO
SSH into primary host
run xsconsole
stop XO VM from xsconsole
shutdown primary host from xsconsole

which than terminates the SSH session and I continue to shutdown the storage and backup server.

Not sure if step 5 and/or 6 is required, because I would expect a clean shutdown of XO when the primary host is shutting down after a shutdown -h now. Currently lacking the time to automate this...

hrv3e

@olivierlambert Ok. I have not made my point there.

My question was how to "safe reboot" host?
Does going through xsconsole make it safer for running VMs or is this the same as plain shutdown?

olivierlambert

Plain shutdown should trigger the same consequence because there's a systemd unit that will do actions (like halting the VMs before the reboot)