XCP-ng 8.3 updates announcements and testing

jr-m4

Updated two dev-pools successfully!

new-xcp-dev: 2x Dell Inc. PowerEdge R650xs
old-xcp-dev: 3x Dell Inc. PowerEdge R350 (HA)

I even mistankenly did a wrong updating order on old-xcp-dev. And neglected to disable HA before updating, leading to two hosts being updated and promoting the next master. And by chance the last master was the non-updated one. Here I disabled HA, but couldn't promote another master (not unexpected)... Post update, and everything seems to be good anyways. And HA re-enabled successfully

Note:
No further testing other than the hosts booting and showing good life signs have been done, yet.

bufanda

Updated my Lab Pool.
For the Test:

@lmolina said:

Reboot your VMs, starting with Linux VMs that use PCI passthrough (GPU, NVMe, NIC). The new HVM PIRQ default applies to every VM at its next boot. If you set platform:hvm-pirq on a VM yourself, that value is kept. Devices that worked before must still work, with similar performance and CPU usage. If one doesn't, check whether platform:hvm-pirq=true brings it back, and tell us.

-> No such VM on my Lab Pool, not tested

Run your usual Xen Orchestra backups (incremental with NBD, full) and a restore. Every NBD backup now goes through new code (tapdisk's NBD server, parallel requests), so they must complete, and shouldn't get slower.

-> ran a full backup job and an incremental backup job. No issues. No diffrences to previous backups.

Watch the disk performance of your busiest VMs, on VHD and QCOW2 SRs. The locking change in tapdisk applies to every VM disk, not only QCOW2.

-> Didn't notice any performance issues.

After running the snapshot_fixer.py script if necessary, take, revert to and delete snapshots, including on VMs with several disks, and check that the VMs and their disks are as expected afterwards. The fix changes how xapi handles VDIs during reverts and SR scans.

-> Performed test with 3 VMs, 1 with one disk, 2 with 2 discs. -> No issues seen.

flakpyro

Installed on my usual test hosts.

Backups seem to have major ussues for me, much less stable with these updates. On two different test pools i ran into a similar issue. These pools are different hardware, one pool uses CBT for backups, one does not.

Pool 1:
Started a backup after running the updates, 2 VMs in the job hung on first run requiring restarting XO and the hosts tool stack. When trying to clear the leftover snapshots one VM got stuck with "VBD.unplug_force", restarting the toolstack didn't help with this i had to reboot the entire host to remove it. Second attempt backups ran ok.

Third attempt, VM hung again, same story as above, had to reboot the whole host to clear the "VBD.unplug_force"

Pool 2:

Backup job completed however again two VMs got stuck at the end with "VBD.unplug_force" and "VDIs attached to Control Domain"
On one VM the backup showed "Error: operation timed out" despite it otherwise finishing.

Is there a way to revert these updates for now? They are test hosts but i do like backing up the test VMs on them.

acebmxer

@flakpyro

Same i am continuing to have backup issues.. multiple reboots, disabled NBD backup getting stuck.

Is this the correct procedure to downgrade from the updates?

yum history list | head -n 6
yum history undo <ID> --enablerepo=xcp-ng-testing
reboot
rpm -q xapi-core blktap xen-hypervisor sm

Or do you advise doing - yum downgrade blktap-3.55.5-11.3.xcpng8.3 to just downgrade the blktap?

Edit - Actually looks like diabling NBD might have worked... I thought backup was still runing.. no the full backup completed this time and a scheduled delta job is running.... NBD is enabled on the delta job.

flakpyro

@acebmxer

I was able to revert by running

 yum history list

Finding the transaction from todays updates then running

 yum history undo <id>

I did this on both pools, doing the slaves first then the master. (Reverse order of updating) and so far everything is back to normal.

flakpyro

@acebmxer I was able to run successfully one time after multiple failures. I don't think its consistently reliable. Interesting that disabling NBD helped though. I didn't try that before reverting.

rzr

@acebmxer said:
(...)
yum history undo <ID> --enablerepo=xcp-ng-testing
(...)

Your're lucky ! this is a risky operation we don't recommend to downgrade because some packages like xapi are not handling this properly, prefer other workarounds if there are any (like disabling nbd until @storage team suggest a better option)

Let me thank you again @flakpyro for this regression report it is very valuable to us,

Andrew

@lmolina Updates ran fine, but I'm having CR backup problems too... Mostly ran fine, but I now have 3 stuck with VBD.unplug_force. I can't get them to clear. I'll have to try harder before I revert updates.

flakpyro

@rzr I was worried about this because of XAPI. Luckily i have a host thats not currently running any VMs so i tried it there first hoping it was a small enough change that i could get away with it. Good to know i could have just disabled NBD for now though!

acebmxer

@rzr said:

@acebmxer said:
(...)
yum history undo <ID> --enablerepo=xcp-ng-testing
(...)

Your're lucky ! this is a risky operation we don't recommend to downgrade because some packages like xapi are not handling this properly, prefer other workarounds if there are any (like disabling nbd until @storage team suggest a better option)

Let me thank you again @flakpyro for this regression report it is very valuable to us,

I didn't run the commands I stated. As I was asking first.. I did the command @flakpryo

Of course this is was before i saw your post. As this is the home lab i am fine with if it blows up. but for others this is good to know.

rzr

We have just released security updates for ca-certificates.

Full details are available on the blog:
https://xcp-ng.org/blog/2026/10/22/october-2026-security-updates-3-for-xcp-ng-8-3-lts/

Greg_E

Ok, so what's the scoop here? Safe to roll this out or wait a few days? I just saw the notice in my XO that there are updates and I don't want to wreck my system right now after fixing a few other "issues".

[edit] I see this is just the certificate update so pushing it out and listening to servers reboot.