HOST_NOT_ENOUGH_FREE_MEMORY during host.restart / host-evacuate despite destination having ample free RAM

michmoor0725

Environment

XCP-ng version: 8.3 (latest)
Xen Orchestra (XO): Build 202606261855
Pool topology: 2-node pool ("Olympus")
- Host A (master, Zeus): 128 GB RAM, 7 VMs running (~34 GB allocated)
- Host B (slave, Hera): 64 GB RAM, 4 VMs running (~32 GB allocated)
HA: Enabled, ha-host-failures-to-tolerate = 0, ha-overcommitted = false
Storage: Shared NFS SRs (all VMs on shared storage — no local-SR VMs)

What I'm trying to do

Test planned maintenance / HA evacuation: reboot Host B (Hera) via Xen Orchestra, expecting its running VMs to live-migrate to Host A (Zeus) automatically.

What happens

The host.restart call from XO (with suspendResidentVms: false, bypassCurrentVmCheck: false) fails immediately with:

HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:<vm-ref>)

Full task error:

  {
    "error_info": [
      "HOST_NOT_ENOUGH_FREE_MEMORY",
      "OpaqueRef:<vm-opaque-ref>"
    ],
    "name": "XapiError",
    "message": "HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:<vm-opaque-ref>)",
    "stack": "XapiError: HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:<vm-opaque-ref>)\n    at XapiError.wrap (file:///opt/xo/xo-builds/.../packages/xen-api/_XapiError.mjs:16:12)\n    at default 
  (file:///opt/xo/xo-builds/.../packages/xen-api/_getTaskResult.mjs:13:29)\n    ..."
  }

Backtrace points to ocaml/xapi/xapi_host.ml:629 inside Async.host.evacuate.

The same error is produced when running xe host-evacuate directly on the pool master:

Not enough server memory is available to perform this operation.
needed: <vm-uuid> (<VM name>)
available: <unknown>

Note the available: <unknown> — XCP-ng cannot determine the available memory on the destination host.

Actual memory figures from xe host-list params=memory-free,memory-total,memory-overhead:

┌────────────────────────┬───────────┬──────────┬──────────┐
│          Host          │ Total RAM │ Free RAM │ Overhead │
├────────────────────────┼───────────┼──────────┼──────────┤
│ Zeus (destination)     │ 128 GB    │ 84.7 GB  │ 1.58 GB  │
├────────────────────────┼───────────┼──────────┼──────────┤
│ Hera (being evacuated) │ 64 GB     │ 26.5 GB  │ 0.93 GB  │
└────────────────────────┴───────────┴──────────┴──────────┘

VMs on Hera that need to migrate:

┌───────┬──────────────────┬─────────────────┐
│  VM   │ RAM (static-max) │ memory-overhead │
├───────┼──────────────────┼─────────────────┤
│ VM-1  │ 12 GB            │ 106 MB          │
├───────┼──────────────────┼─────────────────┤
│ VM-2  │ 8 GB             │ 74 MB           │
├───────┼──────────────────┼─────────────────┤
│ VM-3  │ 8 GB             │ 74 MB           │
├───────┼──────────────────┼─────────────────┤
│ VM-4  │ 4 GB             │ 36 MB           │
├───────┼──────────────────┼─────────────────┤
│ Total │ 32 GB            │ 290 MB          │
└───────┴──────────────────┴─────────────────┘

Zeus has 84.7 GB free. The VMs need ~32.3 GB (RAM + overhead). There is 52 GB of headroom. The error should not occur.

Any help appreciated — this makes planned host maintenance require manual CLI steps every time, which defeats the purpose of the evacuate-on-reboot workflow.

olivierlambert

Hi,

It's related to an existing bug: https://github.com/xapi-project/xen-api/issues/4323

Quick answer so you can move forward:

Set the VMs' HA restart priority to "Restart" (ha_restart_priority = restart), then they're "protected" and get a real evacuation plan. In a 2-node pool with ha-host-failures-to-tolerate=0 this is somewhat cosmetic but it satisfies the partition check.
Or temporarily disable HA, evacuate / restart the host, re-enable HA. Cleanest for maintenance on a 2-node pool, since failures-to-tolerate is forced to 0 anyway.

I will see to accelerate for a fix upstream.

olivierlambert created this issue in xapi-project/xen-api

open Mysterious failure in HA with HOST_NOT_ENOUGH_FREE_MEMORY #4323

olivierlambert

I made 2 PRs upstream, let me ping @Team-XAPI-Network about it.

olivierlambert opened this pull request in xapi-project/xen-api

open xapi: report a clear error when host evacuation is blocked by unprotected VMs #7145

olivierlambert opened this pull request in xapi-project/xen-api

draft xapi: evacuate non-HA-protected VMs when no host failures are tolerated #7146

michmoor0725

@olivierlambert
Thanks for being so transparent about this. I suspect just updating that error message should be helpful enough. I was really struggling yesterday as I couldn’t figure out why my test plan was failing.
Your suggestions did indeed work. In my case just disabling HA briefly did the trick