Thank you everyone for your tests and your feedback!

The updates are live now: https://xcp-ng.org/blog/2025/12/18/december-2025-security-and-maintenance-updates-for-xcp-ng-8-3-lts/

@Andrew had the wrong link. I fixed it as soon as it was mentioned.

The name is xcp-ng-8.3.0-20250606.2.iso because it's the same update level as back then... Only with two updated drivers.

@dcskinner That was a mistake. Thanks for your vigilance, it's fixed!

Let us know if you have any issue with it. It's OK on our side, but I'll wait for some time before making is the new default download.

Hi! We have an updated installer for you in https://xcp-ng.org/blog/2025/12/18/december-2025-security-and-maintenance-updates-for-xcp-ng-8-3-lts/ !

I added a warning to my initial announcement.

@ovicz I'd also like to have a look at /var/log/daemon.log after a failed VM startup attempt.

@ovicz Is Secure Boot enabled on these VMs?

New security and maintenance update candidate for you to test!

A hardware issue was found in AMD Zen 5 CPU devices, related to how random numbers are generated. It's best fixed via a firmware update, but we also provide updated microcode to mitigate it, and Xen is updated to support loading the newer microcode. We also publish other non-urgent updates which we had in the pipe for the next update release.

Security updates:

• amd-microcode: This release fixes vulnerability CVE-2025-62626 in AMD Zen 5 CPUs microcode that may generate excessive number of zeros in random outputs, potentially compromising cryptographic security.
• xen:
  • Introduce support for the new Linux AMD microcode container format (multiple blobs per CPU),
  • Address the XSA-476 vulnerability (CVE-2025-58149), low severity on XCP-ng (affects an unsupported feature of Xen)
  • Enable passthrough of devices on non-zero PCI segments.
  • Improve performance of resumed or migrated VMs by supporting superpage restoration
  • Fix detection of the Self Snooping feature on capable Intel CPUs
• gpumon, xcp-featured: rebuilt for updated XAPI
• qemu:
  • Synchronize with XenServer's fix for the Windows Server 2025 NVMe write cache issue that we fixed previously
  • Fix device passthrough with devices in a PCI segment different from 0
• sm:
  • Upstream changes:
    • Robustify CBT enable/disable calls to prevent errors.
    • Various fixes regarding SCSI commands/functions.
    • Add tolerance in the GC during leaf coalesce.
    • Improves GC logging and corrects rare race conditions.
  • Our changes
    • Use serial instead of SCSI ID for SR on USB devices to prevent bad match.
    • Explicit error message during LVM metadata generation when VDI type is missing.
    • Correct and robustify LINSTOR deletion algorithm to manage in-use volumes.
    • Avoid throwing LINSTOR exceptions in case of impossible temporary volume deletion in order to properly terminate higher-level API calls.
    • Prevent XOSTOR operations if LINSTOR versions mismatches on a pool.
• varstored:
  • Restore and update the default dbx for new VMs. That's the main change for users: we now embed the latest UEFI certificates with XCP-ng, making pools ready for secure boot out of the box. We'll update the documentation to explain how to handle the transition for existing pools (ranging from "nothing to do" to "do something to ensure that future certificate updates become automatically the pool's default).
  • Fix the format of the default included KEK/db/dbx to ensure safe updates
  • Fix an issue with UEFI variable length limit
• xapi:
  • Support up to 16 VIFs (virtual network interfaces) per VM (previously: 7)
  • Runnable metrics:
    • runnable_any
    • runnable_vcpus
  • Various fixes, optimizations, small improvements, and foundational changes (such as getting prepared for a newer version of ocaml)
• gpumon xcp-featured: rebuild for updated XAPI.
• xcp-ng-pv-tools:
  • Properly detect Red Hat 10 and its derivatives, when installing the Linux guest agent
  • Update Windows Tools to 9.1.100
• xcp-ng-release: fix benign "unary operator expected" error, displayed when connecting from some terminal software
• xha: Nothing of note, minor changes such as logging typos...
• xo-lite: version 0.17.0
  • [VM/New] Fix the default topology by setting the platform:cores-per-socket value correctly (PR #9136)
  • [Host/HostSystemResourceManagement] Fix display when control domain memory is undefined (PR [#9197])
• xsconsole: Prepare for a future feature.

Optional packages updated:

• qlogic-netxtreme2-alt: alternate driver for NetXtreme2 updated to version 7.15.24.
• qlogic-qla2xxx-alt: alternate driver qla2xxx updated to version 10.02.14.01_k

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Do not apply these updates if you are using the QCOW2 disk format. QCOW2 testing requires specific update repositories. Updating via the normal test channels would render your disks invisible, and even once the necessary packages are restored, their metadata (which disk is attached to what VM, etc.) will be lost.

For QCOW2 testers, update with:

yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates,xcp-ng-qcow2

For others who'd like to start testing with the QCOW2 format, please head towards the dedicated thread: https://xcp-ng.org/forum/topic/10308/dedicated-thread-removing-the-2tib-limit-with-qcow2-volumes

Versions:

• amd-microcode: 20251203-1.1.xcpng8.3
• gpumon: 24.1.0-71.1.xcpng8.3
• qemu: 4.2.1-5.2.15.1.xcpng8.3
• sm: 3.2.12-16.1.xcpng8.3
• varstored: 1.2.0-3.4.xcpng8.3
• xapi: 25.33.1-2.1.xcpng8.3
• xcp-featured: 1.1.8-3.xcpng8.3
• xcp-ng-pv-tools: 8.3-15.xcpng8.3
• xcp-ng-release: 8.3.0-35
• xen: 4.17.5-23.1.xcpng8.3
• xha: 25.2.0-1.1.xcpng8.3
• xo-lite: 0.17.0-1.xcpng8.3
• xsconsole: 11.0.9.1-1.1.xcpng8.3.3

Optional packages:

• qlogic-netxtreme2-alt: 7.15.24-1.xcpng8.3
• qlogic-qla2xxx-alt: 10.02.14.01_k-1.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the updates

2 days.

The installer is built. We are testing it, then we will be able to provide it to you so you can test it in turn.

We do warn about the certificate situation in the 8.3 release notes, indeed, but it's easy to get caught by that.

There's a way to temporarily disable LTS verification on the new hosts in order to join it to the existing pool.

See https://docs.xcp-ng.org/releases/release-8-3/#certificate-verification-xs which in turns points to https://docs.xenserver.com/en-us/xenserver/8/hosts-pools/certificate-verification where you'll find that command.

Regarding your initial situation, I'm not 100% sure, but I think Warm Migration, might be a way to migrate your VMs off your slave hosts while minimizing downtime. I don't know how it plays with CBT and heterogenous pool state exactly though.

@paco I think it's the first time someone asks this, which is surprising to me, because CBT enabled + local storage may not be such a rare thing.

I wasn't aware of this blocking situation. We'll need to evaluate it, document it, and if possible find a way to avoid it.

In your situation, if all you've done is upgrading the pool master, I would advise to boot the upgrade ISO again and use it to restore the 8.2 backup that was made automatically during the upgrade. Then boot the master again, disable CBT on all your disks, and start again with the upgrade.

@thomasp I'm struggling so much to find the time to do it, and my colleagues are equally busy, but I'll try to make it happen this week!

@gduperrey said in How to Install XCP-ng Guest Tools on Rocky Linux 10?:

but I don't have a release date yet, even for testing

Actually it's already available as xcp-ng-pv-tools in the xcp-ng-incoming repository. What Gaël means is that we haven't run CI on it yet, so we haven't moved the package to the testing repository yet, which is when we usually invite users to test.

However here I'm able to say that there's no risk in installing it now for testing, with:

yum update xcp-ng-pv-tools --enablerepo=xcp-ng-incoming,xcp-ng-ci,xcp-ng-testing,xcp-ng-candidates

(the testing repos will only be enabled for the time of the command, not permanently)

@flakpyro Thanks for letting us know. I suppose there was a mirror that was not ready yet, or had a transient issue, and unfortunately XOA's rolling pool update feature is not very resilient to that at the moment.

IMPORTANT NOTICE!

After publishing the updates, we discovered a very nasty bug when using the UEFI certificates that we distribute. Long story short, they're too big, and there's only limited space (57K), and combined to a preexisting bug in varstored, this will cause the VM to stop booting after Windows or any other OS attempts to append to the DBX (revocation database).

We pulled the varstored update, but those who updated can be affected.

There are conditions for the issue:

• Existing VMs are not affected, unless you propagated the new certs to them
• New VMs are affected only if you never installed UEFI certs to the pool yourself (through XOA or secureboot-certs install), or cleared them using secureboot-certs clear in order to use our default certificates.

If you have the affected version of varstored (rpm -q varstored yields varstored-1.2.0-3.1.xcpng8.3) :

• on every host, downgrade it with yum downgrade varstored-1.2.0-2.3.xcpng8.3. No reboot or toolstack restart required.
• if you have affected UEFI VMs, that is VMs that meet the conditions above but are not broken yet, don't install updates, turn them off, and fix them by deleting their DBX database: https://docs.xcp-ng.org/guides/guest-UEFI-Secure-Boot/#remove-certificates-from-a-vm. This has to be done when the VM is off. Your OS will add its own DBX afterwards.
• If you already have broken VMs (this warning reaching you too late), revert to a snapshot or backup. Other ways to fix them will require a patched varstored currently in the making.

@acebmxer said in XCP-ng 8.3 updates announcements and testing:

@stormi
How to revert changes if needed to? and/or how to switch back to normal repo?

The command only enables the testing repositories for the time of the update, so no need to disable them afterwards.

Reverting changes can be done with yum downgrade, but it's not always doable. XAPI updates can come with an upgrade of the XAPI database. If you downgrade, then XAPI with detect that the database is too recent and will refuse to start.

So, you can technically downgrade the files, but not the state.

New update candidates for you to test! (adding to the previous batch)

New updates join the previous batch of update candidates. I also take this opportunity to call for more feedback on the previous batch of updates, in particular on the changes mentioned in its "What to test" part. Anyway, installing this batch will also install the previous one.

Main changes:

• qemu: Fix BSODs on VMs having the Windows Server 2025 September update and emulated NVMe controllers
• xcp-ng-pv-tools: FINALLY, we could embed our own, signed, Windows Guest Tools in the guest tools ISO shipped with XCP-ng! See https://xcp-ng.org/blog/2025/10/10/signed-windows-pv-drivers-now-available/
• xcp-ng-xapi-plugins:
  • Reworked sdncontroller plugin to properly support all network types:
    • Standard networks on physical devices
    • Bonded networks
    • VLAN on top of either standard networks or bonds
    • Private networks
  • Support per-VIF rules, as well as network-wide rules (no UI in XO at this time, xo-cli recommended)

Other changes:

Optional packages:

• netdata: Minor change in the systemd unit file to avoid minor log pollution. No functional change.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• qemu: qemu-4.2.1-5.2.12.2.xcpng8.3
• xcp-ng-pv-tools: xcp-ng-pv-tools-8.3-13.xcpng8.3
• xcp-ng-xapi-plugins: xcp-ng-xapi-plugins-1.15.0-1.xcpng8.3

Optional packages:

• netdata: netdata-1.47.5-4.2.xcpng8.3

What to test

Normal use and anything else you want to test.

Additional focus can be given to:

• Everything we mentioned in the previous batch
• Make sure Windows+Linux VM installation and booting works on UEFI without PV drivers (that's when the NVMe emulated disks are used)
• XCP-ng's signed Windows Guest tools that are finally available on the guest tools ISO!

Known issues

XAPI's handling of remote logging remains to be fixed before the release.

So: don't attempt to set up remote logging yet. If you set it up previously, then it should continue to work.

Test window before official release of the updates

~5 days.

@olivierlambert LVM also plays a role with such SRs, maybe that's it. Or it's another optimization. XAPI had some too.