Health Check - No Hosts Available
-
Not sure where to start to diagnose this issue.
The backup works however fails on health check. My backup is configured to use the DR host for the health check:
The VM has the following resource limits which I understand means start with 8 vCPUs and 16GB RAM.
Here are the specs of my DR host:
-
Hi,
To help you we need:
- the XO version (or XOA)
- the detailed error log
-
This is using XO.
There are three VMs in the backup however to isolate the issue I only have the health check being performed on one.
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1730359077342", "jobId": "73df5884-3e18-4ce9-9833-53be5a6e0a34", "jobName": "Production", "message": "backup", "scheduleId": "dae62e27-1434-45d5-930b-489e58fd7909", "start": 1730359077342, "status": "failure", "infos": [ { "data": { "vms": [ "ae29cc56-8db5-560a-7694-ba74c66f9b21", "14de66b1-4010-083c-a6f1-36718eec5c71", "af2b36be-d1cf-4e67-e1a7-37f0c94300d5" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "ae29cc56-8db5-560a-7694-ba74c66f9b21", "name_label": "SERVER3" }, "id": "1730359078760", "message": "backup VM", "start": 1730359078760, "status": "success", "tasks": [ { "id": "1730359079179", "message": "snapshot", "start": 1730359079179, "status": "success", "end": 1730359080449, "result": "ba1ea463-f4df-189e-1215-2cf200da84eb" }, { "id": "1730359089180", "message": "health check", "start": 1730359089180, "status": "success", "infos": [ { "message": "This VM doesn't match the health check's tags for this schedule" } ], "end": 1730359089181 }, { "data": { "id": "eb9076c3-13a5-f9a7-43e3-ff1146e5599d", "isFull": false, "name_label": "SVR10533 ZFS", "type": "SR" }, "id": "1730359080450", "message": "export", "start": 1730359080450, "status": "interrupted", "tasks": [ { "id": "1730359081779", "message": "transfer", "start": 1730359081779, "status": "success", "end": 1730359087928, "result": { "size": 205881344 } } ] } ], "end": 1730359089208 }, { "data": { "type": "VM", "id": "14de66b1-4010-083c-a6f1-36718eec5c71", "name_label": "SERVER2" }, "id": "1730359078763", "message": "backup VM", "start": 1730359078763, "status": "success", "tasks": [ { "id": "1730359079179:0", "message": "snapshot", "start": 1730359079179, "status": "success", "end": 1730359399186, "result": "8b2645a2-13ef-72f7-9b67-6c5a0d0ae055" }, { "id": "1730359704874", "message": "health check", "start": 1730359704874, "status": "success", "infos": [ { "message": "This VM doesn't match the health check's tags for this schedule" } ], "end": 1730359704875 }, { "data": { "id": "eb9076c3-13a5-f9a7-43e3-ff1146e5599d", "isFull": false, "name_label": "SVR10533 ZFS", "type": "SR" }, "id": "1730359399186:0", "message": "export", "start": 1730359399186, "status": "interrupted", "tasks": [ { "id": "1730359400614", "message": "transfer", "start": 1730359400614, "status": "success", "end": 1730359421833, "result": { "size": 570976768 } } ] } ], "end": 1730359704909 }, { "data": { "type": "VM", "id": "af2b36be-d1cf-4e67-e1a7-37f0c94300d5", "name_label": "SERVER01" }, "id": "1730359089212", "message": "backup VM", "start": 1730359089212, "status": "failure", "tasks": [ { "id": "1730359089388", "message": "snapshot", "start": 1730359089388, "status": "success", "end": 1730359091775, "result": "775b85a8-25e9-c424-5661-29b5658a32b0" }, { "data": { "id": "eb9076c3-13a5-f9a7-43e3-ff1146e5599d", "isFull": false, "name_label": "SVR10533 ZFS", "type": "SR" }, "id": "1730359091775:0", "message": "export", "start": 1730359091775, "status": "failure", "tasks": [ { "id": "1730359093218", "message": "transfer", "start": 1730359093218, "status": "success", "end": 1730359139809, "result": { "size": 2256734720 } }, { "id": "1730359703451", "message": "health check", "start": 1730359703451, "status": "failure", "tasks": [ { "id": "1730359703457", "message": "cloning-vm", "start": 1730359703457, "status": "success", "end": 1730359712724, "result": "OpaqueRef:97325b70-1993-bfc6-d976-0578cd13fafc" }, { "id": "1730359712732", "message": "vmstart", "start": 1730359712732, "status": "failure", "end": 1730359713131, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "d4383fd4-5a0a-bf4d-ee58-9dd87e5ef698", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241031T07:28:32Z", "finished": "20241031T07:28:32Z", "status": "failure", "resident_on": "OpaqueRef:1cfd3a0d-5417-0899-2351-6da1b9449f09", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 106))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1484))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1512))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1476))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1876))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1858))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1047:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1081:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1071:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1244:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1730359720906, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "d4383fd4-5a0a-bf4d-ee58-9dd87e5ef698", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241031T07:28:32Z", "finished": "20241031T07:28:32Z", "status": "failure", "resident_on": "OpaqueRef:1cfd3a0d-5417-0899-2351-6da1b9449f09", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 106))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1484))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1512))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1476))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1876))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1858))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1047:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1081:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1071:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1244:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1730359720906, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "d4383fd4-5a0a-bf4d-ee58-9dd87e5ef698", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241031T07:28:32Z", "finished": "20241031T07:28:32Z", "status": "failure", "resident_on": "OpaqueRef:1cfd3a0d-5417-0899-2351-6da1b9449f09", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 106))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1484))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1512))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1476))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1876))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1858))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1047:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1081:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1071:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1244:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1730359720956, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "d4383fd4-5a0a-bf4d-ee58-9dd87e5ef698", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20241031T07:28:32Z", "finished": "20241031T07:28:32Z", "status": "failure", "resident_on": "OpaqueRef:1cfd3a0d-5417-0899-2351-6da1b9449f09", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 106))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1484))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1512))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1476))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1876))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1858))((process xapi)(filename ocaml/xapi/rbac.ml)(line 191))((process xapi)(filename ocaml/xapi/rbac.ml)(line 200))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 75)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1047:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1081:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1071:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410311742/packages/xen-api/index.mjs:1244:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1730359720957 } -
@stephane-m-dev can you take a look when you are around?
FYI,
NO_HOSTS_AVAILABLE()in caps means that's a XAPI message, telling us, when we ask to boot the VM, that's impossible to boot it because there's no possible host where the VM can run. It could be for multiple reasons, eg not enough memory anywhere, a resource that's not available in here etc. -
I am not sure if we are booting the VM on the production host or BCDR host however there should be plenty of RAM.
Production - 41GB free
BCDR - 27GB freeHow can I further diagnose this?
-
Hello !
Not enough CPU and not enough RAM, because the RAM is 32G (dynamic max) and the CPU is 16 (8 socket 2 core). The host has max 8 CPU and 27G ram. -
What is the difference between the static and dynamic values for RAM?
My production host has 128GB and my DR host has 32GB so I want to have enough RAM on the VM in production but still allow the health check on the DR host.
-
static is the interval that cannot be changed while the VM is running, unlike dynamic.
-
So when determining the resource allocation on a VM to ensure the health check does not encounter the "NO_HOSTS_AVAILABLE" error should I set the static or dynamic memory values?
In Hyper-V there is a toggle to select between static or dynamic memory allocation
-
IIRC, it will check if it fits with the static interval. It's visible in your VM view, Advanced tab.
-
Is it PV, HVM or a PVHVM VM?
Health Check can't work on HVM because there are no tools.
