XCP-ng 8.3 updates announcements and testing

acebmxer

after multiple failed attempts and rebooted all three host. Host 1 did update and finally got host 2 and host 3 updated both via rolling pool update.

During the evacuation multiple vms failed to migrate / reboot. All showing similar errors in each console for the vms.

jacob.becker

@acebmxer
Had some similar Errors on 2 Arista appliances with an older Alma-Linux base (9.5).
After upgrading the image (Alma 9.6 based) the error went away - so it looks like a kernel thing

what kernel run inside your vm ?

acebmxer

@jacob.becker

VM were Ubuntu and Fedora distros. Ubuntu is 7.0.0-38-generic. The Fedora vms 7.2.7-200.fc44.x86_64 and 7.2.9-300.fc45.x86_64.

Had no issues with the beta test but those were done not via Rolling pool update.

jacob.becker

@acebmxer
Can you suspend the vm ?

If its not possible its most likely a kernel thing inside the vm.
If you can suspend the vm however, it points more to a timing/timeout issue with the migration.

gduperrey

We have just released security updates for xen and blktap. Full details are available on the blog: https://xcp-ng.org/blog/2026/10/13/october-2026-security-updates-2-for-xcp-ng-8-3-lts/

lmolina

New maintenance update candidates for XCP-ng 8.3 LTS

This batch of updates contains mostly fixes, tools version update, and some improvements.

What changed

Virtualization & System

xen:
- This release is a synchronization with XenServer release 4.17.7-4. XSAs 517, 518, and 520 patches that were included in XCP-ng previous releases are now part of this XS release.
- This release also fixes a possible microcode hang on Granite Rapids systems.

Control Plane

xapi:
- Fix the VDI.snapshot_of corruption (see the corresponding troubleshooting page). Note that the old corrupted snapshot_of references still need to be fixed up by the snapshot-fixer.py helper script.
- Fix issues for PCI passthrough of some devices (AMD and Intel GPUs, Kioxia SSDs) by disabling HVM PIRQ
- PCI passthrough: HVM PIRQ now disabled by default. Some devices passed through to VMs (AMD and Intel GPUs, Kioxia SSDs) could have problems caused by the "HVM PIRQ" feature (see the troubleshooting section). This update disables it by default on HVM VMs. The new default applies the next time a VM starts or reboots. If you set platform:hvm-pirq on a VM yourself, that value is kept.
- Improve performance of xapi-nbd (used by XenOrchestra backups)
- Fix XenOrchestra backups of VDIs on shared SRs failing when a pool has offline hosts
- Prevent SR issues from blocking XAPI startup
- Report a clear error during HA when host evacuation is blocked by unprotected VMs
xcp-ng-xapi-plugins:
- lsblk.py : Add a new output in lsblk device-id-paths that is a list of paths on the host using stable identifier.
- service.py: Restrict which services the plugin is allowed to stop/start

Storage

blktap: Several tapdisk improvements:
- Qcow2 support introduced multithread and therefore locking. One of the blkif's critical session protected by a lock was larger than necessary. By restricting locking at the blkif level, write operations on Qcow2-format VDIs can see great performance improvement if there is sufficient I/O depth.
- Fix a potential crash on NBD server handshake connection. This was observed under artificial conditions and should not happen in real scenarios.
- Storage team continue to modernize the tapdisk codebase to allow more improvements in the future. No functional change expected here but preparatory work on next features.
sm:
- Repair plugin method healthCheck of linstor-manager that was broken due to internal refactoring.
- Repair regression caused by previous release where LINSTOR HA cannot be activated.
- Reset on boot feature is now usable on ZFSSR driver using on-boot param with persist or reset values.

For XOSTOR users:

xcp-ng-linstor: Fix rolling pool updates with large pools using XOSTOR

UI

xo-lite:
- [Host] Add possibility to reboot a host (PR #10250)
- [Web-Core/TabItem] Update the component to remove uppercase for better readability (PR #10338)
- [Host] Add possibility to force reboot a host (PR #10311)
- [Icons] Update FontAwesome, clean unused libraries (PR #10365)
- [Host] Add possibility to shut down and start a host (PR #10314)
- Use MenuItem's accent prop to color menu actions instead of custom CSS classes (PR #10348)
- [Host] Add possibility to forget a host (PR #10315)
- [Pool/Network] Add the possibility to delete host internal networks (PR #10332)

Drivers and Firmware

amd-microcode: Update to 2026-09-23 drop as redistributed by XenServer. Updated CPUs
- BRH-C1 00b00f21: 2025-10-17, rev 0b002162 -> 2026-08-10, rev 0b00217b
- BRHD-B0 00b10f10: 2025-10-17, rev 0b101059 -> 2026-08-10, rev 0b101068
intel-microcode:
- Update to the 2026-09-25 drop. This is IPU 2026.3 with fixed Meteor Lake microcode.
- fix of a bugus microcode for Sierra Forest (E-Core) CPUs.
- microcode update for Granite Rapids to address a security vulnerability for which a mitigation had already been released in previous updates (https://xcp-ng.org/blog/2026/10/13/october-2026-security-updates-2-for-xcp-ng-8-3-lts/).
- Removed MTL/06-aa-04/c0 due to functional issues observed when loading the MCU in some platforms.
- Updates for functional issues.
- Security updates for: INTEL-SA-01379, INTEL-SA-01404, INTEL-SA-01423, INTEL-SA-01428, INTEL-SA-01435, INTEL-SA-01441, INTEL-SA-01442, INTEL-SA-01443, INTEL-SA-01420

Others

sudo: Update to version 1.9.17
- 3 security fixes of vulnerabilities in sudo which could allow a privilege escalation in certain condition.
- 1 security fix of a vulnerability allowing a user with temporary authorization to extend their authorization by 24 hours.

Versions

amd-microcode: 20260519-1.1.xcpng8.3 -> 20260923-1.1.xcpng8.3
blktap: 3.55.5-11.3.xcpng8.3 -> 3.55.5-12.2.xcpng8.3
gpumon: 24.1.0-98.1.xcpng8.3 -> 24.1.0-99.1.xcpng8.3
intel-microcode: 20260416-2.xcpng8.3 -> 20260925-1.xcpng8.3
sm: 3.2.12-25.1.xcpng8.3 -> 3.2.12-25.2.xcpng8.3
sudo: 1.9.15-5.1.xcpng8.3 -> 1.9.17-10.p2.1.xcpng8.3
xapi: 26.1.19-1.2.xcpng8.3 -> 26.1.21-1.1.xcpng8.3
xcp-featured: 1.2.1-5.xcpng8.3 -> 1.2.1-6.xcpng8.3
xcp-ng-xapi-plugins: 1.17.0-1.xcpng8.3 -> 1.18.0-1.xcpng8.3
xen: 4.17.7-2.5.xcpng8.3 -> 4.17.7-4.1.xcpng8.3
xo-lite: 0.25.0-1.xcpng8.3 -> 0.26.0-1.xcpng8.3

XOSTOR users: specific update procedure

Some XOSTOR packages are provided in separate repository, and should be installed along xcp-ng regular packages, using the following commands:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates,xcp-ng-linstor-testing
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates,xcp-ng-linstor-testing
reboot

The following XOSTOR-specific package received an update:

xcp-ng-linstor: 1.2-6.xcpng8.3 -> 1.2-7.xcpng8.3

Test on XCP-ng 8.3

Warning: XOSTOR users should skip this part and follow the instructions of the previous section.

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

Normal use and anything else you want to test, as usual. We mostly want to make sure nothing regressed. These checks would help us most:

Reboot your VMs, starting with Linux VMs that use PCI passthrough (GPU, NVMe, NIC). The new HVM PIRQ default applies to every VM at its next boot. If you set platform:hvm-pirq on a VM yourself, that value is kept. Devices that worked before must still work, with similar performance and CPU usage. If one doesn't, check whether platform:hvm-pirq=true brings it back, and tell us.
Run your usual Xen Orchestra backups (incremental with NBD, full) and a restore. Every NBD backup now goes through new code (tapdisk's NBD server, parallel requests), so they must complete, and shouldn't get slower.
Watch the disk performance of your busiest VMs, on VHD and QCOW2 SRs. The locking change in tapdisk applies to every VM disk, not only QCOW2.
After running the snapshot_fixer.py script if necessary, take, revert to and delete snapshots, including on VMs with several disks, and check that the VMs and their disks are as expected afterwards. The fix changes how xapi handles VDIs during reverts and SR scans.

Test window before official release of the updates

~5 days

We would like to thank users who shared feedback since our last call for testing:
@flakpyro, @acebmxer, @abudef, @bufanda, @jr-m4, @JeffBerntsen, @Andrew

acebmxer

@lmolina

installed updates.... by accident i ran the linstore-testing updates. but no issues with the udpate itself.

I did download the snapshot-fixer.py in dry run it found to vms. I ran a full backup first and was going fine till here...

Here are some of the logs - if you have a place i can send the rest as i am not able to post all of them...

tapdisk-28167-daemonlog.txt
tapdisk-probe.txt
tapdisk-28167-fd.txt
tapdisk-28167-threads.txt
versions.txt

Dug into it some more. The stuck process is tapdisk pid 28167. tap-ctl list just hangs and tap-ctl list -p 28167 times out, but the other 3 tapdisks answer fine. It’s in S state, not D, so it doesn’t look like it’s waiting on the disk.

28167 is the one doing the NBD export of the backup snapshot for my Dockerthings_anepi VM (snapshot bf2a422e-9df3-4b3d-83f0-0fa3ff191efb, VHD on SR 4991d4aa...). That’s the export that stopped at 75%. Everything after it is stuck at 0% (VBD.plug, importing raw VDI, the XO-CE-root import), and when I tried force unplug those got stuck too. The backup job said it finished though, and the VM itself is still running fine.

Ran the snapshot-fixer dry run and it found 2 VMs. I did not run the rewrite before doing the backup.

Update went through fine at 05:58, all from xcp-ng-testing, and I rebooted at 06:00, so this happened on the new packages. The linstor-testing thing didn’t install anything, rpm -qa | grep -i linstor comes back empty. Attached the yum history.

I have the full logs (xensource, SMlog, daemon, kern) if you want them, just tell me where to send them.
yum-history-11.txt

Update: kill 28167 (plain SIGTERM, not -9) cleared it, then the stuck tasks cancelled without a reboot. Will post if it happens again on the next backup.

jr-m4

Updated two dev-pools successfully!

new-xcp-dev: 2x Dell Inc. PowerEdge R650xs
old-xcp-dev: 3x Dell Inc. PowerEdge R350 (HA)

I even mistankenly did a wrong updating order on old-xcp-dev. And neglected to disable HA before updating, leading to two hosts being updated and promoting the next master. And by chance the last master was the non-updated one. Here I disabled HA, but couldn't promote another master (not unexpected)... Post update, and everything seems to be good anyways. And HA re-enabled successfully

Note:
No further testing other than the hosts booting and showing good life signs have been done, yet.

bufanda

Updated my Lab Pool.
For the Test:

@lmolina said:

Reboot your VMs, starting with Linux VMs that use PCI passthrough (GPU, NVMe, NIC). The new HVM PIRQ default applies to every VM at its next boot. If you set platform:hvm-pirq on a VM yourself, that value is kept. Devices that worked before must still work, with similar performance and CPU usage. If one doesn't, check whether platform:hvm-pirq=true brings it back, and tell us.

-> No such VM on my Lab Pool, not tested

Run your usual Xen Orchestra backups (incremental with NBD, full) and a restore. Every NBD backup now goes through new code (tapdisk's NBD server, parallel requests), so they must complete, and shouldn't get slower.

-> ran a full backup job and an incremental backup job. No issues. No diffrences to previous backups.

Watch the disk performance of your busiest VMs, on VHD and QCOW2 SRs. The locking change in tapdisk applies to every VM disk, not only QCOW2.

-> Didn't notice any performance issues.

After running the snapshot_fixer.py script if necessary, take, revert to and delete snapshots, including on VMs with several disks, and check that the VMs and their disks are as expected afterwards. The fix changes how xapi handles VDIs during reverts and SR scans.

-> Performed test with 3 VMs, 1 with one disk, 2 with 2 discs. -> No issues seen.

flakpyro

Installed on my usual test hosts.

Backups seem to have major ussues for me, much less stable with these updates. On two different test pools i ran into a similar issue. These pools are different hardware, one pool uses CBT for backups, one does not.

Pool 1:
Started a backup after running the updates, 2 VMs in the job hung on first run requiring restarting XO and the hosts tool stack. When trying to clear the leftover snapshots one VM got stuck with "VBD.unplug_force", restarting the toolstack didn't help with this i had to reboot the entire host to remove it. Second attempt backups ran ok.

Third attempt, VM hung again, same story as above, had to reboot the whole host to clear the "VBD.unplug_force"

Pool 2:

Backup job completed however again two VMs got stuck at the end with "VBD.unplug_force" and "VDIs attached to Control Domain"
On one VM the backup showed "Error: operation timed out" despite it otherwise finishing.

Is there a way to revert these updates for now? They are test hosts but i do like backing up the test VMs on them.

acebmxer

@flakpyro

Same i am continuing to have backup issues.. multiple reboots, disabled NBD backup getting stuck.

Is this the correct procedure to downgrade from the updates?

yum history list | head -n 6
yum history undo <ID> --enablerepo=xcp-ng-testing
reboot
rpm -q xapi-core blktap xen-hypervisor sm

Or do you advise doing - yum downgrade blktap-3.55.5-11.3.xcpng8.3 to just downgrade the blktap?

Edit - Actually looks like diabling NBD might have worked... I thought backup was still runing.. no the full backup completed this time and a scheduled delta job is running.... NBD is enabled on the delta job.

flakpyro

@acebmxer

I was able to revert by running

 yum history list

Finding the transaction from todays updates then running

 yum history undo <id>

I did this on both pools, doing the slaves first then the master. (Reverse order of updating) and so far everything is back to normal.

flakpyro

@acebmxer I was able to run successfully one time after multiple failures. I don't think its consistently reliable. Interesting that disabling NBD helped though. I didn't try that before reverting.

rzr

@acebmxer said:
(...)
yum history undo <ID> --enablerepo=xcp-ng-testing
(...)

Your're lucky ! this is a risky operation we don't recommend to downgrade because some packages like xapi are not handling this properly, prefer other workarounds if there are any (like disabling nbd until @storage team suggest a better option)

Let me thank you again @flakpyro for this regression report it is very valuable to us,

Andrew

@lmolina Updates ran fine, but I'm having CR backup problems too... Mostly ran fine, but I now have 3 stuck with VBD.unplug_force. I can't get them to clear. I'll have to try harder before I revert updates.

flakpyro

@rzr I was worried about this because of XAPI. Luckily i have a host thats not currently running any VMs so i tried it there first hoping it was a small enough change that i could get away with it. Good to know i could have just disabled NBD for now though!

acebmxer

@rzr said:

@acebmxer said:
(...)
yum history undo <ID> --enablerepo=xcp-ng-testing
(...)

Your're lucky ! this is a risky operation we don't recommend to downgrade because some packages like xapi are not handling this properly, prefer other workarounds if there are any (like disabling nbd until @storage team suggest a better option)

Let me thank you again @flakpyro for this regression report it is very valuable to us,

I didn't run the commands I stated. As I was asking first.. I did the command @flakpryo

Of course this is was before i saw your post. As this is the home lab i am fine with if it blows up. but for others this is good to know.

rzr

We have just released security updates for ca-certificates.

Full details are available on the blog:
https://xcp-ng.org/blog/2026/10/22/october-2026-security-updates-3-for-xcp-ng-8-3-lts/