HOST_NOT_ENOUGH_FREE_MEMORY

[micahm322]

I am running a 3 node cluster on XCP-NG 8.2. Node 1 has 192GB of ram (179 free), Node 2 has 512GB (431 free), and Node 3 has 512 GB (493GB free). Node 3 has one VM running on it which is allocated 4GB of ram. When I hit restart on Node 3 through XOA, I get HOST_NOT_ENOUGH_FREE_MEMORY. I am not sure why I am getting this error when I have so much free ram. I have already rebooted all nodes and restarted the tool stacks. I have also tried restarting different nodes and get the same error.

Would appreciate any insight!

host.restart
{
  "id": "df35f1fe-ecba-4fb0-b4c6-1a9db0efac2b",
  "force": false
}
{
  "code": "HOST_NOT_ENOUGH_FREE_MEMORY",
  "params": [
    "OpaqueRef:a399f13f-ca17-44c7-a2de-4c25d0319293"
  ],
  "task": {
    "uuid": "48724829-015b-3de1-4542-5fb21b56fd12",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20210219T23:36:59Z",
    "finished": "20210219T23:36:59Z",
    "status": "failure",
    "resident_on": "OpaqueRef:f6b6a08e-f323-4883-82a4-7418a4780633",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "HOST_NOT_ENOUGH_FREE_MEMORY",
      "OpaqueRef:a399f13f-ca17-44c7-a2de-4c25d0319293"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 560))((process xapi)(filename hashtbl.ml)(line 266))((process xapi)(filename hashtbl.ml)(line 272))((process xapi)(filename hashtbl.ml)(line 277))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 556))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 231))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 103)))"
  },
  "message": "HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:a399f13f-ca17-44c7-a2de-4c25d0319293)",
  "name": "XapiError",
  "stack": "XapiError: HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:a399f13f-ca17-44c7-a2de-4c25d0319293)
    at Function.wrap (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/_XapiError.js:16:12)
    at _default (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/_getTaskResult.js:11:29)
    at Xapi._addRecordToCache (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:866:24)
    at forEach (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:887:14)
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:877:12)
    at Xapi._watchEvents (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/src/index.js:1038:14)"
}

[olivierlambert]

Host restart is trying to live migrate VMs to other hosts of the pool, with host.evacuate.

Do you have HA enabled? Is the VM protected by HA?

[micahm322]

@olivierlambert Yes I have HA enabled. The VM is set to disable for HA. Does it need to be protected to be moved? I thought it would live migrate regardless of HA status, and HA was for host failure/etc.

[micahm322]

@olivierlambert I just enabled HA for the VM and when I reboot the host it migrated the VM and then rebooted. Do I need to protect all of my VMs with HA in order to restart hosts?

[olivierlambert]

I think that's a bug of HA and XAPI. That's funny because I had a conversation just few hours before about this very issue.

We'll investigate. Thanks for confirming this bug

To recap the "how to reproduce":

1. evacuate a host
2. HA enable at pool level
3. a VM not protected with HA

Then it will fail with HOST_NOT_ENOUGH_FREE_MEMORY (which is NOT the case).

This big doesn't happen if the VM is HA protected.

[olivierlambert]

Issue reported upstream: https://github.com/xapi-project/xen-api/issues/4323

We'll work with XAPI team on investigations (since we can rebuild a XAPI easily with some modification to reproduce or not a potential fix)

olivierlambert created this issue in xapi-project/xen-api

open Mysterious failure in HA with HOST_NOT_ENOUGH_FREE_MEMORY #4323

[Danp]

@olivierlambert Seems related to this discussion from almost 5 years ago.

[olivierlambert]

This time we are also the upstream, so it's a bit different

[micahm322]

@olivierlambert Thanks for the quick feedback!