New update candidates for you to test! (adding to the previous batch)

stormi

@flakpyro Thanks for letting us know. I suppose there was a mirror that was not ready yet, or had a transient issue, and unfortunately XOA's rolling pool update feature is not very resilient to that at the moment.

stormi

IMPORTANT NOTICE!

After publishing the updates, we discovered a very nasty bug when using the UEFI certificates that we distribute. Long story short, they're too big, and there's only limited space (57K), and combined to a preexisting bug in varstored, this will cause the VM to stop booting after Windows or any other OS attempts to append to the DBX (revocation database).

We pulled the varstored update, but those who updated can be affected.

There are conditions for the issue:

Existing VMs are not affected, unless you propagated the new certs to them
New VMs are affected only if you never installed UEFI certs to the pool yourself (through XOA or secureboot-certs install), or cleared them using secureboot-certs clear in order to use our default certificates.

If you have the affected version of varstored (rpm -q varstored yields varstored-1.2.0-3.1.xcpng8.3) :

on every host, downgrade it with yum downgrade varstored-1.2.0-2.3.xcpng8.3. No reboot or toolstack restart required.
if you have affected UEFI VMs, that is VMs that meet the conditions above but are not broken yet, don't install updates, turn them off, and fix them by deleting their DBX database: https://docs.xcp-ng.org/guides/guest-UEFI-Secure-Boot/#remove-certificates-from-a-vm. This has to be done when the VM is off. Your OS will add its own DBX afterwards.
If you already have broken VMs (this warning reaching you too late), revert to a snapshot or backup. Other ways to fix them will require a patched varstored currently in the making.

stormi

@acebmxer said in XCP-ng 8.3 updates announcements and testing:

@stormi
How to revert changes if needed to? and/or how to switch back to normal repo?

The command only enables the testing repositories for the time of the update, so no need to disable them afterwards.

Reverting changes can be done with yum downgrade, but it's not always doable. XAPI updates can come with an upgrade of the XAPI database. If you downgrade, then XAPI with detect that the database is too recent and will refuse to start.

So, you can technically downgrade the files, but not the state.

stormi

New update candidates for you to test! (adding to the previous batch)

New updates join the previous batch of update candidates. I also take this opportunity to call for more feedback on the previous batch of updates, in particular on the changes mentioned in its "What to test" part. Anyway, installing this batch will also install the previous one.

Main changes:

qemu: Fix BSODs on VMs having the Windows Server 2025 September update and emulated NVMe controllers
xcp-ng-pv-tools: FINALLY, we could embed our own, signed, Windows Guest Tools in the guest tools ISO shipped with XCP-ng! See https://xcp-ng.org/blog/2025/10/10/signed-windows-pv-drivers-now-available/
xcp-ng-xapi-plugins:
- Reworked sdncontroller plugin to properly support all network types:
  - Standard networks on physical devices
  - Bonded networks
  - VLAN on top of either standard networks or bonds
  - Private networks
- Support per-VIF rules, as well as network-wide rules (no UI in XO at this time, xo-cli recommended)

Other changes:

Optional packages:

netdata: Minor change in the systemd unit file to avoid minor log pollution. No functional change.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

qemu: qemu-4.2.1-5.2.12.2.xcpng8.3
xcp-ng-pv-tools: xcp-ng-pv-tools-8.3-13.xcpng8.3
xcp-ng-xapi-plugins: xcp-ng-xapi-plugins-1.15.0-1.xcpng8.3

Optional packages:

netdata: netdata-1.47.5-4.2.xcpng8.3

What to test

Normal use and anything else you want to test.

Additional focus can be given to:

Everything we mentioned in the previous batch
Make sure Windows+Linux VM installation and booting works on UEFI without PV drivers (that's when the NVMe emulated disks are used)
XCP-ng's signed Windows Guest tools that are finally available on the guest tools ISO!

Known issues

XAPI's handling of remote logging remains to be fixed before the release.

So: don't attempt to set up remote logging yet. If you set it up previously, then it should continue to work.

Test window before official release of the updates

~5 days.

stormi

@olivierlambert LVM also plays a role with such SRs, maybe that's it. Or it's another optimization. XAPI had some too.

stormi

@Andrew Nice. What kind of SR?

stormi

I'm going to build an updated installer with an updated bnxt_en driver, as more and more servers require it.

stormi

@flakpyro said in XCP-ng 8.3 updates announcements and testing:

@stormi Installed on my usual test hosts (Intel Minisforum MS-01, and Supermicro running a Xeon E-2336 CPU). Also installed onto a 2 host AMD epyc pool. Updates went smooth, backups continue to function as before.

3 windows 11 VMs had secure boot enabled. In XOA i clicked "Copy pool's default UEFI certificates to the VM" after the update was complete. The VMs continued to boot without issue after.

If you want to go further with the test, you need to clear your pool's secure boot certificates (the ones you probably had installed in the past from XO to "set up the pool for Guest SB"), so that the new pool defaults become the ones we provided with the update.

Then you can try again propagating the certs to the VMs.

stormi

A new batch of non-urgent updates is ready for user tests before a future collective release. Below are the details about these.

Main changes:

edk2: In the virtual firmware for UEFI VMs (OVMF):
- Update the embedded OpenSSL to version 3.0.9 + additional fixes. This mainly addresses network boot of UEFI VMs from an HTTPS server.
- Fix VLAN tag handling, that is fix network boot of UEFI VMs on a tagged VLAN (without handling the VLAN at the pool network level, as this makes it transparent to the VM).
ethtool: Allow ethtool to enable 50G/100G/200G link modes. The dom0 kernel and the Mellanox network adapters driver have been updated accordingly.
expat: Update from 2.1.0 to 2.5.0, bringing security fixes related to XML handling.
guest-templates-json: Add templates for almalinux 10, rocky linux 10, debian 13, oracle linux 10, redhat 10
intel-microcode:
- Update to publicly released microcode-20250812
- Security updates for: INTEL-SA-01249, INTEL-SA-01308, INTEL-SA-01310, INTEL-SA-01311, INTEL-SA-01367
- Updates for multiple functional issues.
- Note: this update is provided with XCP-ng as a convenience, but this doesn't constitute a fix of a vulnerability in XCP-ng, nor does it entirely replace the action of upgrading your firmware.
kernel:
- Enable 50G/100G/200G ethtool link modes in XCP-ng 8.3 kernel and Mellanox network adapters driver. The userspace tool ethtool has been updated accordingly.
- Fix race condition regarding namespace identifier attributes in sysfs
- Fix deadlock on PCI passthrough. This is related to the following Known Issue in XenServer: "When NVIDIA T4 added in pass-through mode to a VM on some specific server hardware, that VM might not power on"
libtpms: Fix CVE-2025-49133 in libtpms - "Potential out-of-bounds access and abort in libtpms due to inconsistent HMAC signing parameters". A guest process with access to the TPM can cause the guest's TPM server to crash by sending malicious commands. This crash does not affect other VMs or the host.
lvm2: Performance improvements for LVM-based SRs on systems with a large number of VDIs.
mellanox-mlnxen: Enable 50G/100G/200G ethtool link modes (goes with kernel patches and ethtool patches). Note: this set of changes was initially made by XenServer, and we haven't had the occasion to test it.
qlogic-qla2xxx: Update to version 10.02.13.00_k. Bug fixes only.
sm:
- Adapt the LargeBlock SR driver following the change of configuration in lvm2 rebase.
- Robustify LINSTOR volume size retrievals to avoid throwing exceptions when not needed.
- Improve LINSTOR DB robustness: detect failure with a small delay, use specific DRBD options to fit for drbd-reactor.
- Limit LINSTOR logs in SMlog.
- Rewrite of the handling of DRBD/LINSTOR command calls: in particular speed gain concerning scan commands.
- Robustify LINSTOR DB umount call in case of network outage.
- Robustify the garbage collector to avoid falsely marking VDIs as hidden, subsequently preventing journal rollbacks.
- Fix error message reported when a snapshot failed.
- Robustify LinstorSR creation with thick mode.
varstored: Provide the latest Secure Boot certificates from Microsoft by default. This is a big change compared to the past situation where you had to prepare pools for Guest Secure Boot by manually running a command or clicking a button in Xen Orchestra. Now, if you haven't install any UEFI certificates to the pool, it will automatically use the latest we provide on the system to set up Secure Boot on new VMs. Existing VMs are untouched. Additionally, this will allow to support Secure Boot with future Windows media that no longer use the expired 2011 certificates. Documentation updates are on their way: https://github.com/xcp-ng/xcp-ng-org/pull/328.
xapi:
- Notable fixes
  - Consoles are now started for PVH guests => steps towards supporting PVH virtualization mode.
  - Stop ballooning down memory on localhost migration => VDI migration to another SR no longer fails because of unrelated memory configuration.
  - Allow SHA-512 in host certificates.
  - Better error reporting for other_operation_in_progress => now describing what operation was blocking another one.
  - Avoid trying to suspend a VM which doesn't support it, thus preventing a VM crash.
  - Fix an issue that disabled CBT unnecessarily on VDIs on shared SRs during VM live migration. This will also allow to live migrate such VMs during a rolling pool update.
  - Message.get_all_records_where now properly evaluates the query. This will be leveraged by Xen Orchestra to get some information from XAPI faster (by fetching smaller amounts of items).
  - Fix issues with emergency network reset on IPv6 hosts
- Notable features
  - Best effort mode for NUMA: This is not enabled by default at the moment, but when enabled this means that xapi will try to use a single NUMA node when creating VMs. It is best effort, meaning that this strategy sometimes fails and instead all nodes are used. Especially when many VMs are started or migrated at the same time. Test instructions are provided in the "What to test" section below.
  - Host evacuation was parallelized further, so that the migrating flow of VMs is maintained, avoiding bottlenecks.
  - Storage migration reworked: Allows migration from and to SMAPIv3 storage backends => a step towards SMAPIv3, but not immediately usable.
  - CLI interface: improved autocompletion (xe).
  - New HA option to avoid rebooting VMs on internal shutdown: https://docs.xcp-ng.org/management/ha/#halting-the-vm
- A lot of other fixes and internal improvements.
xen:
- Enhance support for Intel Granite Rapids systems
- Fix PCI passthrough on some systems
- Add additional CPU RRD metrics
xenserver-status-report: bug fixes + collection of additional debug data.
xo-lite: update to version 0.15.0. See Xen Orchestra's release announcements for the changelog.

Other changes

blktap: Add a log line when an operation is not supported.
gpumon: Rebuilt for updated XAPI.
libarchive: Fixed libarchive package to refresh the ldconfig cache, whose lack impacted driver disk generation
samba: remove unneeded dependencies.
swtpm: Rebuilt for updated libtpms.
xcp-featured: Rebuilt for updated XAPI.
xcp-python-libs: Various fixes.
xha: Various fixes.

Added dependency:

qcow-stream-tool: will be used by XAPI when support for the QCOW2 disk format is added.

XOSTOR
In addition to the changes in common packages, the following XOSTOR-specific packages received updates:

kmod-drbd:
- Update DRBD kernel module (improvements, fixes)
- Important update to fix memory leaks during DRBD resource synchronization: This bug could be triggered during the addition of a node, or after an LINSTOR evacuation following a HW problem and a recreation of the resources.
linstor:
- Updated the LINSTOR controller and satellite packages.
- In particular, a resizing issue that may block a resource has been fixed.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

blktap: 3.55.5-6.1.xcpng8.3
edk2: 20220801-1.7.10.1.xcpng8.3
ethtool: 4.19-3.xcpng8.3
expat: 2.5.0-3.xcpng8.3
gpumon: 24.1.0-65.1.xcpng8.3
guest-templates-json: 2.0.14-1.1.xcpng8.3
intel-microcode: 20250715-1.xcpng8.3
kernel: 4.19.19-8.0.43.1.xcpng8.3
libarchive: 3.3.3-1.1.xcpng8.3
libtpms: 0.9.6-3.xcpng8.3
lvm2: 2.02.180-18.2.1.xcpng8.3
mellanox-mlnxen: 5.9_0.5.5.0-3.1.xcpng8.3
qcow-stream-tool: 25.27.0-2.1.xcpng8.3
qlogic-qla2xxx: 10.02.13.00_k-1.xcpng8.3
samba: 4.10.16-25.2.xcpng8.3
sm: 3.2.12-10.2.xcpng8.3
swtpm: 0.7.3-9.xcpng8.3
varstored: 1.2.0-3.1.xcpng8.3
xapi: 25.27.0-2.1.xcpng8.3
xcp-featured: 1.1.8-2.xcpng8.3
xcp-python-libs: 3.0.8-1.1.xcpng8.3
xen: 4.17.5-20.1.xcpng8.3
xenserver-status-report: 2.0.15-1.xcpng8.3
xha: 25.1.0-1.1.xcpng8.3
xo-lite: 0.15.0-1.xcpng8.3

XOSTOR

kmod-drbd-9.2.14-1.0.xcpng8.3
linstor-common-1.29.2-1.el7_9
linstor-controller-1.29.2-1.el7_9
linstor-satellite-1.29.2-1.el7_9

What to test

Normal use and anything else you want to test.

Additional focus can be given to:

Network boot of a UEFI VM from HTTPS server
Network boot of a UEFI VM on a tagged VLAN (without handling the VLAN at the pool network level, as this makes it transparent to the VM)
UEFI Secure Boot: new VMs, existing VMs, ... And/or just verify that you understand the updated documentation (work in progress)
Testing on Intel Granite Rapids systems
50G/100G/200G link modes with Mellanox network adapters, using ethtool
vTPM
Hardware depending on the qla2xxx driver, that is Qlogic 2500/2600/2700/277x/2800 Series Fibre Channel Adapters
NUMA's best effort mode, for hosts with more than one NUMA node:
Turn it on, per host, by running xe host-param-set numa-affinity-policy=best_effort uuid=$HOST_UUID.
Now new VMs should be able to be allocated to single NUMA nodes. To verify that it works, (re)start a VM that can fit in a single NUMA memory node, run xl debug-keys u on dom0, then check the output by running xl dmesg. The last lines of the output will show on which numa nodes each domain (VM) has memory allocated, you should see that the domain that was restarted, usually the one with the highest domain number; has several nodes below it and that all of them except one have the number 0.

Known issues

XAPI's handling of remote logging changed. XAPI now expects a configuration file in a specific location, and we haven't applied this system change yet. We'll publish the related update candidate to complement the current batch in the next days.

So: don't attempt to set up remote logging yet. If you set it up previously, then it should continue to work.

Test window before official release of the updates

~10 days. But please test as early as possible.

dinhngtu opened this pull request in xcp-ng/xcp-ng-org

Lead Maintainer

Posts

New update candidates for you to test! (adding to the previous batch)

Test on XCP-ng 8.3

Versions:

What to test

Known issues

Test window before official release of the updates

Test on XCP-ng 8.3

Versions:

What to test

Known issues

Test window before official release of the updates

closed Announce Secure Boot changes #328

Member List