Moving vm in pool fails with "HOST_NOT_ENOUGH_FREE_MEMORY" (there is enough memory)!

rvreugde

I have an XCP-ng pool with 2 hosts, each having 384GB memory.
Moving a VM (16GB) from one host to another host with enough free memory (254GB), results in an error: "HOST_NOT_ENOUGH_FREE_MEMORY"!
The log states:

vm.migrate
{
  "targetHost": "8b56a961-3eaa-45f7-833c-bea08565e7a9",
  "vm": "10afa9ed-44bf-733e-7f89-d26e1263f27b"
}
{
  "code": "HOST_NOT_ENOUGH_FREE_MEMORY",
  "params": [
    "4334813184",
    "1790664704"
  ],
  "task": {
    "uuid": "a17234bc-2d21-57b7-c8d2-c5ba7050b9e2",
    "name_label": "Async.VM.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20241027T10:01:55Z",
    "finished": "20241027T10:01:55Z",
    "status": "failure",
    "resident_on": "OpaqueRef:0ea9e83b-d9d2-4e80-9176-634aea4c77cb",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "HOST_NOT_ENOUGH_FREE_MEMORY",
      "4334813184",
      "1790664704"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 618))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 718))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1226))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1356))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1225))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "HOST_NOT_ENOUGH_FREE_MEMORY(4334813184, 1790664704)",
  "name": "XapiError",
  "stack": "XapiError: HOST_NOT_ENOUGH_FREE_MEMORY(4334813184, 1790664704)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1041:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1075:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1065:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1238:14)"
}

Should a toolstack restart do the trick?

olivierlambert

Hi,

IIRC it's a very wrong message coming from XAPI, because there's another problem under which is not reported correctly.

Do you have HA enabled out of curiosity?

rvreugde

Good afternoon @olivierlambert ,
NO, HA is not enabled.

olivierlambert

So from the error message, it's like the host needs 4GiB or memory while there's only 1,6GiB free. Are you live migrating with a shared storage in the same pool (so just the memory to move) OR migrating with the storage? (to another pool OR from a local SR).

rvreugde

Hi @olivierlambert,

I am just trying to do a live migration with shared storage (NFS), memory only, without a storage migration.

Now, I'm freeing up xcp028 (which, btw, is pool master).

Should a toolstack restart solve the issue?
Or should I reboot the system?
Before eventually a reboot, should I change the Pool Master to the other system?

olivierlambert

There's no reason to reboot or restart the toolstack without knowing more what's going on.

rvreugde

Since it was somewhat critical (urgent migration from other pool), I had to switch the pool masters and had to restart the xcp028.

But I downloaded the system log. Maybe you find the cause of the problem there?

BTW, before restarting the host, I was not able to download these logs (failed).