XCP-ng 8.3 updates announcements and testing

stormi

New update candidates for you to test! (adding to the previous batch)

New updates join the previous batch of update candidates. I also take this opportunity to call for more feedback on the previous batch of updates, in particular on the changes mentioned in its "What to test" part. Anyway, installing this batch will also install the previous one.

Main changes:

qemu: Fix BSODs on VMs having the Windows Server 2025 September update and emulated NVMe controllers
xcp-ng-pv-tools: FINALLY, we could embed our own, signed, Windows Guest Tools in the guest tools ISO shipped with XCP-ng! See https://xcp-ng.org/blog/2025/10/10/signed-windows-pv-drivers-now-available/
xcp-ng-xapi-plugins:
- Reworked sdncontroller plugin to properly support all network types:
  - Standard networks on physical devices
  - Bonded networks
  - VLAN on top of either standard networks or bonds
  - Private networks
- Support per-VIF rules, as well as network-wide rules (no UI in XO at this time, xo-cli recommended)

Other changes:

Optional packages:

netdata: Minor change in the systemd unit file to avoid minor log pollution. No functional change.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

qemu: qemu-4.2.1-5.2.12.2.xcpng8.3
xcp-ng-pv-tools: xcp-ng-pv-tools-8.3-13.xcpng8.3
xcp-ng-xapi-plugins: xcp-ng-xapi-plugins-1.15.0-1.xcpng8.3

Optional packages:

netdata: netdata-1.47.5-4.2.xcpng8.3

What to test

Normal use and anything else you want to test.

Additional focus can be given to:

Everything we mentioned in the previous batch
Make sure Windows+Linux VM installation and booting works on UEFI without PV drivers (that's when the NVMe emulated disks are used)
XCP-ng's signed Windows Guest tools that are finally available on the guest tools ISO!

Known issues

XAPI's handling of remote logging remains to be fixed before the release.

So: don't attempt to set up remote logging yet. If you set it up previously, then it should continue to work.

Test window before official release of the updates

~5 days.

acebmxer

@stormi

Sorry if this has been mentioned before.

You state to run the below command to test the updates.

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

How to revert changes if needed to? and/or how to switch back to normal repo?

Andrew

@stormi Update (to the update) installed and running. Buggy Windows 2025 boots now with QEMU update.

olivierlambert

Yay \o/ Thanks for the feedback

ph7

@stormi

I also take this opportunity to call for more feedback on the previous batch of updates,

Well I updated a few days ago, tough I dont run much of the updated functions on my simple home lab, it all seems to work fine.
i7 gen 4 and NFS

Now testing the new updates......

ph7

The new template for debian 13 is working in XO-Lite

gskger

@stormi Updated the usual suspects (HP ProDesk 600 G6, Dell Optiplex 9010, Dell R720) with no problem. Host run as expected.

stormi

@acebmxer said in XCP-ng 8.3 updates announcements and testing:

@stormi
How to revert changes if needed to? and/or how to switch back to normal repo?

The command only enables the testing repositories for the time of the update, so no need to disable them afterwards.

Reverting changes can be done with yum downgrade, but it's not always doable. XAPI updates can come with an upgrade of the XAPI database. If you downgrade, then XAPI with detect that the database is too recent and will refuse to start.

So, you can technically downgrade the files, but not the state.

acebmxer

@stormi

Thanks for the reply back. Update when sucessfull. Windows Server 2025 iso now properly installs.

At work I was not able to install default certs for UEFI due to one failing to download. Run these updates and I was able to successfully install the certs to the host.

JeffBerntsen

@stormi My "test/production" system, an HP DL165 is updated and running normally with the updated updates. Not seeing any change with secure boot VMs at all, i.e. working just fine.

gduperrey

New update candidates for you to test! (adding to the previous batch again)

New updates join the previous batch of update candidates. They're the last ones.

A new XSA (Xen Security Advisory) was published on the 21th of October, and updates to Xen address the disclosed vulnerabilities. We also reverted a change in XAPI that we deemed risky.

Additionally, we also publish an updated Intel-Ice alternate driver.

xen:
- XSA-475 - Potential risks include Denial of Service (DoS) impacting the whole host, information exposure, or escalation of privileges. There are two vulnerabilities related to hypercalls in the Viridian code:
  - CVE-2025-58147: Out-of-bounds write in vpmask_set() from hypercalls using the HV_VP_SET Sparse format.
  - CVE-2025-58148: Out-of-bound read in send_ipi() from hypercalls using any format, that could lead to a wild vCPU pointer.
xapi:
- We reverted a change related to how rsyslog configuration is handled. The way XenServer handled the change seemed risky to us, we'll take the time to make it in a safer way.

Optional packages:

Alternate Driver: Updated to newer version.
- intel-ice-alt: Update driver sources to v1.17.2

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xapi: 25.27.0-2.2.xcpng8.3
xen: 4.17.5-20.2.xcpng8.3

Optional packages:

Alternate drivers:
- intel-ice-alt: 1.17.2-1.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the updates

~2 days.

gskger

@gduperrey Works on my play-/homelab (HP ProDesk 600 G6, Dell Optiplex 9010). Can't update my Dell R720s GPU cluster at the moment, though.

JeffBerntsen

@gduperrey This is working without problems on my test lab system, an HP Proliant DL165 G7

gduperrey

Updates published: https://xcp-ng.org/blog/2025/10/23/october-2025-security-and-maintenance-update-for-xcp-ng-8-3-lts/

Thank you for the tests!

manilx

@gduperrey Updated 2 pools/2 servers @home, updated 3 pools/5 servers @ office with RPU.
All fine.

flakpyro

Updates 3 pools. All fine except for the last host on the last pool ran into a yum mirror error and failed. Manually running yum update and rebooting the host worked. Sadly i then had to move VMs back around since the RPU failed and the process of migrating VMs back did not kick off as a result.

Error was

One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=<repoid> ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable <repoid>
        or
            subscription-manager repos --disable=<repoid>

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

xcp-ng-base: Check uncompressed DB failed

acebmxer

Updated AMD Ryzen pool at home and update two Intel Dell r660 and r640 pools at work. No issues to report back.

marcoi

updated my two hosts without issues.

stormi

IMPORTANT NOTICE!

After publishing the updates, we discovered a very nasty bug when using the UEFI certificates that we distribute. Long story short, they're too big, and there's only limited space (57K), and combined to a preexisting bug in varstored, this will cause the VM to stop booting after Windows or any other OS attempts to append to the DBX (revocation database).

We pulled the varstored update, but those who updated can be affected.

There are conditions for the issue:

Existing VMs are not affected, unless you propagated the new certs to them
New VMs are affected only if you never installed UEFI certs to the pool yourself (through XOA or secureboot-certs install), or cleared them using secureboot-certs clear in order to use our default certificates.

If you have the affected version of varstored (rpm -q varstored yields varstored-1.2.0-3.1.xcpng8.3) :

on every host, downgrade it with yum downgrade varstored-1.2.0-2.3.xcpng8.3. No reboot or toolstack restart required.
if you have affected UEFI VMs, that is VMs that meet the conditions above but are not broken yet, don't install updates, turn them off, and fix them by deleting their DBX database: https://docs.xcp-ng.org/guides/guest-UEFI-Secure-Boot/#remove-certificates-from-a-vm. This has to be done when the VM is off. Your OS will add its own DBX afterwards.
If you already have broken VMs (this warning reaching you too late), revert to a snapshot or backup. Other ways to fix them will require a patched varstored currently in the making.

manilx

@stormi Downgraded all hosts @home and @office. I have not done anything to running Windows Server VM's or touch anything regarding certs. So I guess I'm all good.