XCP-ng 8.3 updates announcements and testing

MajorP93

Hello,

thanks for publishing these updates!

However did you guys change anything regarding rolling pool update?

I am using latest XO from sources.

It's the third time I am getting this:

I am using dynamic memory control.

During last few rounds of updates (last few months) rolling pool update was working just fine.
This is the third VM today for which RPU is complaining about ballooning / DMC.

For the first two VMs I disabled DMC and re-tried RPU.
It progressed to the next VM but then failed again with the error shown on the screenshot.

In the past RPU was working fine with DMC hence the question regarding RPU changes?

Thanks and best regards

MajorP93

Aaaaand it happened for the next VM.
Apart from updating XO I did not change anything...
It looks like RPU is failing on every single VM that has DMC enabled...
To me it looks like a regression

//EDIT:
the weird thing is that it fails with the same error even with DMC disabled...
It also fails when manually trying to live migrate the VM.
This particular VM is a Windows Server VM, XCP-ng tools version: Management agent 9.2.385-0
What leaves me baffled is that all of this has been working just fine for several months...

vm.migrate
{
  "vm": "25cf2537-c8f9-a3ef-ca68-6621351544c1",
  "migrationNetwork": "1ee81d68-6614-f134-f4b4-6bb5399bf66a",
  "targetHost": "d984660e-c9a9-4981-aac8-de3df67833a9"
}
{
  "code": "BALLOONING_TIMEOUT_BEFORE_MIGRATION",
  "params": [
    "OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813"
  ],
  "task": {
    "uuid": "dd29dc7b-62a6-1820-79d7-f447b7a5dcf6",
    "name_label": "Async.VM.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20261103T16:17:54Z",
    "finished": "20261103T16:20:44Z",
    "status": "failure",
    "resident_on": "OpaqueRef:73f0c67e-d88c-6ed0-9070-ca96f899319f",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "BALLOONING_TIMEOUT_BEFORE_MIGRATION",
      "OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2802))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2859))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2122))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2121))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 239))((process xenopsd-xc)(filename ocaml/xenopsd/xc/emu_manager.ml)(line 244))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2257))((process xenopsd-xc)(filename result.ml)(line 23))((process xenopsd-xc)(filename ocaml/xenopsd/xc/domain.ml)(line 2250))((process xenopsd-xc)(filename ocaml/xenopsd/xc/xenops_server_xen.ml)(line 2853))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2589))((process xenopsd-xc)(filename list.ml)(line 121))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2582))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2973))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 439))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2986))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 439))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2885))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3432))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3442))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3463))((process xenopsd-xc)(filename ocaml/xapi-idl/lib/task_server.ml)(line 192))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3483))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3654))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 355))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 361))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 386))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 527))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3492))((process xapi)(filename lib/backtrace.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 551))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2605))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "BALLOONING_TIMEOUT_BEFORE_MIGRATION(OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813)",
  "name": "XapiError",
  "stack": "XapiError: BALLOONING_TIMEOUT_BEFORE_MIGRATION(OpaqueRef:0b74d118-6c0d-aed2-ca21-99f0f7d5c813)
    at Function.wrap (file:///etc/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///etc/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1374:24)
    at file:///etc/xen-orchestra/packages/xen-api/index.mjs:1408:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1398:12)
    at Xapi._watchEvents (file:///etc/xen-orchestra/packages/xen-api/index.mjs:1605:14)"
}

acebmxer

@lmolina

I applied updates and nothing broken for me to report. I have done delta backup and a full backup both were successful.

JeffBerntsen

Updates applied to my systems, test and live. All is working well so far.

Andrew

@lmolina All updated. RPU successful. Systems running normally.

gduperrey

@MajorP93

We discussed your feedback yesterday, but we don't have an explanation at the moment. Our Windows expert tried to reproduce the behavior but was unable to trigger this error.

Furthermore, we haven't encountered any issues on our end with the RPU (Rolling Pool Update) operations performed during our tests, whether done manually or via XO. It is possible, however, that we are missing the specific scenario (VMs or configuration) required to trigger this one.

Have you been able to find any further information in the host logs?

If not, could you export and send us a status report using xen-bugtool? Perhaps one of our developers will be able to spot something in it that helps explain the issue.

jr-m4

3 pools updated. RPU went fine, where one is HA. A couple of test-backups went fine as well.

robertblissitt

Greetings, @majorp93 . I have always heard that Dynamic Memory should not be used in XenServer/XCP-ng as it is not 100% stable, and that has been my experience as well.

lmolina

We have just released a security update for a critical Linux kernel vulnerability CVE-2026-98432 / VSA-2026-056.

Full details are available on the blog: https://xcp-ng.org/blog/2026/11/04/november-2026-security-updates-2-for-xcp-ng-8-3-lts/

manilx

@lmolina And you couldn't have waited yesterday's update for ONE DAY!

Give me a break, another 1hr updating.....

It's GREAT that you address security updates.... BUT come on.

gduperrey

@manilx

If we could have waited, we would have. However, the xen security update released yesterday, given its critical nature, did not allow for that.

Please rest assured that we are aware of this issue and are doing our utmost to keep the number of releases to a minimum. We are also discussing how to communicate regarding the most critical updates and how to limit the need for reboots.

acebmxer

applying the updates on hosts at work. So far 2 out of 3 pools updated succesfully. however when i go to attempt to apply updates on the last pool i see this...

Screenshot 2026-11-04 105107.png

Screenshot 2026-11-04 104921.png

ThierryC01

@lmolina When will we be able to update or patch everything from XO6? It's kind of a bummer to switch from a slick UI to the legacy, yet functional one, just to apply some patches.

john205

Hi,
There is a bug in the blog post:

https://xcp-ng.org/blog/2026/11/04/november-2026-security-updates-2-for-xcp-ng-8-3-lts/

"apply the iptables-based mitigation described below"

but no mitigation is described!

Thanks,

John

lmolina

@john205 said:

"apply the iptables-based mitigation described below"

Thanks for reporting the mistake. The mitigation is described in the VSA.

The blog is now fixed!

lmolina

@acebmxer said:

applying the updates on hosts at work. So far 2 out of 3 pools updated succesfully. however when i go to attempt to apply updates on the last pool i see this...

@acebmxer, thanks for the feedback. We have no explanation at the moment. So far, all our tests have been successful. We may be missing a specific scenario that triggers this error.

Do you have any additional information in the host logs?

If not, could you export and send us a status report using xen-bugtool? One of our developers may be able to spot something that helps explain the issue.

mpiton

@acebmxer that banner comes from XO, not from the hosts. The previous RPU on Tilton Pool 1 stopped because Host 1 didn't come back within 20 minutes of its reboot (XO waits for the host to be enabled again with a restarted toolstack), and Host 2 is still pending. As long as an RPU is left incomplete, XO won't start a new one on that pool. That's why the button is greyed out.

What I'd do:

Check that Host 1 is up, enabled, and running the new packages.
Click Finalize. If the failed run left something it had changed (HA, WLB, load balancer, backup schedules, a disabled host, VMs moved or halted), XO refuses and lists it. Put those back, or force it if you're fine leaving them as they are. Forcing only lists them in the task, it doesn't change anything on the pool.
Run the RPU again to update Host 2. If the master is already up to date but the other host isn't, XO asks you to confirm first.

If that host is just slow to boot (long POST, RAID init, that kind of thing), you can raise restartHostTimeout in the [xapiOptions] section of the xo-server config. The default is 20 minutes.

If it never came back enabled at all, then the host logs lmolina asked for are still the way to go.

acebmxer

@lmolina, @mpiton

I was able to get the two host in the pool updated. I did finalization and it did the updates on host 1 but would not kick of the updates on the second host. I put that host into maintenance mode and did the updates and rebooted.

Do you still want to pull anything.. My guess the error was last round of updates as I never touched the pool till just then to apply the updates.