Rolling Pool Update fails with HOST_NOT_ENOUGH_FREE_MEMORY, when it really ought to be fine

pkgw

I've had problems on and off with the RPU (Rolling Pool Update) feature for a while now. Usually, it mostly works for me, but runs into a problem figuring out how to move my VMs around and ends up quitting in the middle. Often this has to do with available memory.

With one of my pools right now, if I launch the RPU it instantly fails with an error like this:

CANNOT_EVACUATE_HOST(HOST_NOT_ENOUGH_FREE_MEMORY,OpaqueRef:...)

Here's a screenshot of the pool host summary, where the horizontal bars show the RAM usage:

The first 7 hosts have 128 GB RAM but the last host is a big box with 1 TB RAM total, and 650 GB of RAM free. This pool hosts a set of VMs that use 64 GB of memory, so they are hard to fit on the smaller hosts, but they could all easily fit on the big host at once. We're not using dynamic memory or any host affinities.

So it really seems to me like the RPU algorithm should be able to figure out how to make this work. Are there any settings I can adjust that might help here? Are there surprising constraints that the algorithm applies that might explain why it's failing here?

(Semi-side note: often when an RPU fails, I do updates manually with xe host-evacuate, and that command seems to work fine when the RPU algorithm fails.)

pkgw

OK, I've spent some time looking into this and have learned a few things worth mentioning.

First, there is pretty ample logging of the RPU planning step in the /var/log/xensource.log file. In my particular case it didn't end up being directly helpful, but it's a good resource to be aware of.

Second, it looks like the Xen Orchestra half of the RPU implementation lives here on GitHub. In my case, the error was reported from inside the host.assert_can_evacuate API call that's implemented here inside the xen-api repo.

I don't know OCaml and haven't really grappled with the source code there in detail, but I believe what's happening is that the server is checking that each individual host can be evacuated, but only treating them individually — not taking into account the fact that VMs will be moving from one host to another as the evacuation proceeds in practice.

In my particular case, the logging seems to be saying that this analysis was failing because, as things currently stood, when it would come time to reboot my big-memory machine (mpcxenbackup9), there wouldn't be room to hold its VMs on the other hosts, treating the current VM layout as a given. It has two 64 GB VMs, and only one of the other hosts (the one with >50% memory free) can fit one.

This would be consistent, I think, with other failure modes I've experienced, where the RPU starts and gets partway done, but fails with a not-enough-memory error. In those cases, I think that the assert_can_evacuate checks are passing, but as the VMs reshuffle during the migration, it becomes impossible to migrate a large-memory VM.

I think the solution would be that the planning and execution algorithm needs to be smarter and understand how VMs will move around over the course of the migration; plus I think it has weaknesses in deciding where to send VMs during the process. My system has five 64 GB VMs, which a smart enough packing algorithm should be able to fit at all times no matter which host is offline. But if the algorithm just assigns each VM to the first host that will fit it (in some broad sense), it can get itself wedged later in the process.

(FWIW, my intuition is that, if you only think worry about memory, there will always be a successful plan that migrates VMs a minimal number of times, but that might not be correct. Factoring in other constraints like SRs and networks, you probably need to try a bunch of different solutions with randomization to have higher confidence in finding a fully working migration plan.)

Danp

It's possible that you hit this known issue.

pkgw

@Danp No, it's not quite that. We don't use HA anywhere. But reading the source code I did notice exactly what's mentioned in that bug: in some HA cases, the xen-api code will throw a NOT_ENOUGH_FREE_MEMORY that is just absolutely misleading.

At the macro scale, I think that you could say that the issue I'm running into is that the RPU migrations aren't planned out across the entire process of the RPU. If you have VMs that take up a significant fraction of some hosts' memory, failure to plan the whole process at once can lead the system to get itself "wedged" where it's unable to evacuate one of the hosts.

More narrowly, you could say that the issue is that during an RPU the host evacuation code gives up more easily than it could. Say that you're trying to evacuate host A with a 64 GB VM, and you have two other hosts, B and C, with 60 GB free. It will give up, even if there are VMs that you could move from B to C (or vice versa) that would open up one of them to having 64 GB free. If the evacuation during the RPU was more resilient, then the macro-scale issue would be less of a problem. (Although with a more resilient evacuation command but no large-scale planning, you might end up with an RPU that succeeds but is wildly inefficient.)

Danp

@pkgw Thanks for the feedback. The good news is that there is already an effort under way rework and enhance the RPU process. Let me ping @Team-XAPI-Network and @Team-XO-Backend in case they want to request additional information from you.

andriy.sultanov

@pkgw Planning an RPU as a whole is a nice idea in theory, but it's inherently designed to co-exist with normal pool management operations (you should still be able to migrate VMs between hosts B and C during it, or start new ones), we are not "pre-booking" memory and other resources for groups of VMs, otherwise it can quickly make the rest of the pool hard to manage while some VM migrations could still be far away.

Regarding the narrower case - I'll note this down and see if we can improve things, thank you!

acebmxer

@danp

Think i am facing this issue now. Just tried to put master host into maintance mode to apply latest 8.3 updates and getting the error.

host.setMaintenanceMode
{
  "id": "c1372ec6-3651-4481-808b-34293e53e144",
  "maintenance": true
}
{
  "code": "HOST_NOT_ENOUGH_FREE_MEMORY",
  "params": [
    "OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0"
  ],
  "task": {
    "uuid": "631dc2d9-7598-8ce6-5dd0-558342eb7d1f",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260710T15:20:03Z",
    "finished": "20260710T15:20:03Z",
    "status": "failure",
    "resident_on": "OpaqueRef:119546e4-e0fd-adc7-d42a-b17c4efe789d",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "HOST_NOT_ENOUGH_FREE_MEMORY",
      "OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 629))((process xapi)(filename hashtbl.ml)(line 159))((process xapi)(filename hashtbl.ml)(line 165))((process xapi)(filename hashtbl.ml)(line 170))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 625))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0)",
  "name": "XapiError",
  "stack": "XapiError: HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0)
    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1078:24)
    at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1112:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1102:12)
    at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1275:14)"
}

Screenshot 2026-07-10 112216.png

acebmxer

after manually migrate vms off host i was able to put host into maintenance mode to apply updates.

Edit after both host updated still not able to put host into maintenance mode...

host.setMaintenanceMode
{
  "id": "c1372ec6-3651-4481-808b-34293e53e144",
  "maintenance": true
}
{
  "code": "HOST_NOT_ENOUGH_FREE_MEMORY",
  "params": [
    "OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca"
  ],
  "task": {
    "uuid": "ad526091-5706-3e6b-a544-b7f84a5cfdb0",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260710T15:43:17Z",
    "finished": "20260710T15:43:17Z",
    "status": "failure",
    "resident_on": "OpaqueRef:119546e4-e0fd-adc7-d42a-b17c4efe789d",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "HOST_NOT_ENOUGH_FREE_MEMORY",
      "OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 629))((process xapi)(filename hashtbl.ml)(line 159))((process xapi)(filename hashtbl.ml)(line 165))((process xapi)(filename hashtbl.ml)(line 170))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 625))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca)",
  "name": "XapiError",
  "stack": "XapiError: HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca)
    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1078:24)
    at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1112:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1102:12)
    at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1275:14)"
}

Possible error with Twinstore enabled HA?

Danp

@acebmxer Do you have VMs that aren't configured for HA? Does it work if you temporarily disable HA?

acebmxer

@Danp said:

@acebmxer Do you have VMs that aren't configured for HA? Does it work if you temporarily disable HA?

yes disabling HA did work. Note HA was setup via Twinstore. I currently do not have a vm on that SR.

MajorP93

@acebmxer @pkgw Do / did you guys have any VMs set to "best-effort" under VM's HA-settings?

I encountered the exact same issue (HOST_NOT_ENOUGH_FREE_MEMORY) and was able to completely solve it by changing all VMs from "best effort" to "restart".

Best regards

acebmxer

@MajorP93

No VM had HA set. All were set to disabled. Just set all running vms to best effort and still error.

host.setMaintenanceMode
{
  "id": "35233210-4e37-4703-9bf6-9e8a9c24df9f",
  "maintenance": true
}
{
  "code": "HOST_NOT_ENOUGH_FREE_MEMORY",
  "params": [
    "OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca"
  ],
  "task": {
    "uuid": "a73ffab2-ef34-7daa-8f26-0bb0c5769559",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260710T16:42:09Z",
    "finished": "20260710T16:42:09Z",
    "status": "failure",
    "resident_on": "OpaqueRef:119546e4-e0fd-adc7-d42a-b17c4efe789d",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "HOST_NOT_ENOUGH_FREE_MEMORY",
      "OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 629))((process xapi)(filename hashtbl.ml)(line 159))((process xapi)(filename hashtbl.ml)(line 165))((process xapi)(filename hashtbl.ml)(line 170))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 625))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca)",
  "name": "XapiError",
  "stack": "XapiError: HOST_NOT_ENOUGH_FREE_MEMORY(OpaqueRef:9ada2e57-1c8a-acb8-3637-df6819b64aca)
    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1078:24)
    at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1112:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1102:12)
    at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1275:14)"
}

pkgw

@andriy.sultanov Yes, I completely agree, which is why I brought up the narrower case. I'd put it like this:

You can't just plan the process and blindly execute that plan, because the ground might shift underneath your feet.
This implies to me that the individual steps of the RPUs (individual host evacuations, mainly) would benefit from being made more resilient
But there's still value in trying to make a plan. First, there are different sequences of moves that might be more or less efficient, and you might as well try to be more efficient, even if you might have to change course midway through.
And second, if you can't even come up with a plan that works in theory, you know that you have a real problem.

I'm pretty sure the gold-plated approach would be start with a plan, evacuate resiliently, reboot, re-plan based on the current ground truth, evacuate, reboot, etc. If you have the code to plan things out once, rerunning it after every step isn't too much extra (though it does need to learn to keep track of the dwindling number of hosts that need reboots).