@acebmxer said in XCP-ng 8.3 updates announcements and testing:

@stormi
How to revert changes if needed to? and/or how to switch back to normal repo?

The command only enables the testing repositories for the time of the update, so no need to disable them afterwards.

Reverting changes can be done with yum downgrade, but it's not always doable. XAPI updates can come with an upgrade of the XAPI database. If you downgrade, then XAPI with detect that the database is too recent and will refuse to start.

So, you can technically downgrade the files, but not the state.

New update candidates for you to test! (adding to the previous batch)

New updates join the previous batch of update candidates. I also take this opportunity to call for more feedback on the previous batch of updates, in particular on the changes mentioned in its "What to test" part. Anyway, installing this batch will also install the previous one.

Main changes:

• qemu: Fix BSODs on VMs having the Windows Server 2025 September update and emulated NVMe controllers
• xcp-ng-pv-tools: FINALLY, we could embed our own, signed, Windows Guest Tools in the guest tools ISO shipped with XCP-ng! See https://xcp-ng.org/blog/2025/10/10/signed-windows-pv-drivers-now-available/
• xcp-ng-xapi-plugins:
  • Reworked sdncontroller plugin to properly support all network types:
    • Standard networks on physical devices
    • Bonded networks
    • VLAN on top of either standard networks or bonds
    • Private networks
  • Support per-VIF rules, as well as network-wide rules (no UI in XO at this time, xo-cli recommended)

Other changes:

Optional packages:

• netdata: Minor change in the systemd unit file to avoid minor log pollution. No functional change.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• qemu: qemu-4.2.1-5.2.12.2.xcpng8.3
• xcp-ng-pv-tools: xcp-ng-pv-tools-8.3-13.xcpng8.3
• xcp-ng-xapi-plugins: xcp-ng-xapi-plugins-1.15.0-1.xcpng8.3

Optional packages:

• netdata: netdata-1.47.5-4.2.xcpng8.3

What to test

Normal use and anything else you want to test.

Additional focus can be given to:

• Everything we mentioned in the previous batch
• Make sure Windows+Linux VM installation and booting works on UEFI without PV drivers (that's when the NVMe emulated disks are used)
• XCP-ng's signed Windows Guest tools that are finally available on the guest tools ISO!

Known issues

XAPI's handling of remote logging remains to be fixed before the release.

So: don't attempt to set up remote logging yet. If you set it up previously, then it should continue to work.

Test window before official release of the updates

~5 days.

@olivierlambert LVM also plays a role with such SRs, maybe that's it. Or it's another optimization. XAPI had some too.

@Andrew Nice. What kind of SR?

I'm going to build an updated installer with an updated bnxt_en driver, as more and more servers require it.

@flakpyro said in XCP-ng 8.3 updates announcements and testing:

@stormi Installed on my usual test hosts (Intel Minisforum MS-01, and Supermicro running a Xeon E-2336 CPU). Also installed onto a 2 host AMD epyc pool. Updates went smooth, backups continue to function as before.

3 windows 11 VMs had secure boot enabled. In XOA i clicked "Copy pool's default UEFI certificates to the VM" after the update was complete. The VMs continued to boot without issue after.

If you want to go further with the test, you need to clear your pool's secure boot certificates (the ones you probably had installed in the past from XO to "set up the pool for Guest SB"), so that the new pool defaults become the ones we provided with the update.

Then you can try again propagating the certs to the VMs.

A new batch of non-urgent updates is ready for user tests before a future collective release. Below are the details about these.

Main changes:

• edk2: In the virtual firmware for UEFI VMs (OVMF):
  • Update the embedded OpenSSL to version 3.0.9 + additional fixes. This mainly addresses network boot of UEFI VMs from an HTTPS server.
  • Fix VLAN tag handling, that is fix network boot of UEFI VMs on a tagged VLAN (without handling the VLAN at the pool network level, as this makes it transparent to the VM).
• ethtool: Allow ethtool to enable 50G/100G/200G link modes. The dom0 kernel and the Mellanox network adapters driver have been updated accordingly.
• expat: Update from 2.1.0 to 2.5.0, bringing security fixes related to XML handling.
• guest-templates-json: Add templates for almalinux 10, rocky linux 10, debian 13, oracle linux 10, redhat 10
• intel-microcode:
  • Update to publicly released microcode-20250812
  • Security updates for: INTEL-SA-01249, INTEL-SA-01308, INTEL-SA-01310, INTEL-SA-01311, INTEL-SA-01367
  • Updates for multiple functional issues.
  • Note: this update is provided with XCP-ng as a convenience, but this doesn't constitute a fix of a vulnerability in XCP-ng, nor does it entirely replace the action of upgrading your firmware.
• kernel:
  • Enable 50G/100G/200G ethtool link modes in XCP-ng 8.3 kernel and Mellanox network adapters driver. The userspace tool ethtool has been updated accordingly.
  • Fix race condition regarding namespace identifier attributes in sysfs
  • Fix deadlock on PCI passthrough. This is related to the following Known Issue in XenServer: "When NVIDIA T4 added in pass-through mode to a VM on some specific server hardware, that VM might not power on"
• libtpms: Fix CVE-2025-49133 in libtpms - "Potential out-of-bounds access and abort in libtpms due to inconsistent HMAC signing parameters". A guest process with access to the TPM can cause the guest's TPM server to crash by sending malicious commands. This crash does not affect other VMs or the host.
• lvm2: Performance improvements for LVM-based SRs on systems with a large number of VDIs.
• mellanox-mlnxen: Enable 50G/100G/200G ethtool link modes (goes with kernel patches and ethtool patches). Note: this set of changes was initially made by XenServer, and we haven't had the occasion to test it.
• qlogic-qla2xxx: Update to version 10.02.13.00_k. Bug fixes only.
• sm:
  • Adapt the LargeBlock SR driver following the change of configuration in lvm2 rebase.
  • Robustify LINSTOR volume size retrievals to avoid throwing exceptions when not needed.
  • Improve LINSTOR DB robustness: detect failure with a small delay, use specific DRBD options to fit for drbd-reactor.
  • Limit LINSTOR logs in SMlog.
  • Rewrite of the handling of DRBD/LINSTOR command calls: in particular speed gain concerning scan commands.
  • Robustify LINSTOR DB umount call in case of network outage.
  • Robustify the garbage collector to avoid falsely marking VDIs as hidden, subsequently preventing journal rollbacks.
  • Fix error message reported when a snapshot failed.
  • Robustify LinstorSR creation with thick mode.
• varstored: Provide the latest Secure Boot certificates from Microsoft by default. This is a big change compared to the past situation where you had to prepare pools for Guest Secure Boot by manually running a command or clicking a button in Xen Orchestra. Now, if you haven't install any UEFI certificates to the pool, it will automatically use the latest we provide on the system to set up Secure Boot on new VMs. Existing VMs are untouched. Additionally, this will allow to support Secure Boot with future Windows media that no longer use the expired 2011 certificates. Documentation updates are on their way: https://github.com/xcp-ng/xcp-ng-org/pull/328.
• xapi:
  • Notable fixes
    • Consoles are now started for PVH guests => steps towards supporting PVH virtualization mode.
    • Stop ballooning down memory on localhost migration => VDI migration to another SR no longer fails because of unrelated memory configuration.
    • Allow SHA-512 in host certificates.
    • Better error reporting for other_operation_in_progress => now describing what operation was blocking another one.
    • Avoid trying to suspend a VM which doesn't support it, thus preventing a VM crash.
    • Fix an issue that disabled CBT unnecessarily on VDIs on shared SRs during VM live migration. This will also allow to live migrate such VMs during a rolling pool update.
    • Message.get_all_records_where now properly evaluates the query. This will be leveraged by Xen Orchestra to get some information from XAPI faster (by fetching smaller amounts of items).
    • Fix issues with emergency network reset on IPv6 hosts
  • Notable features
    • Best effort mode for NUMA: This is not enabled by default at the moment, but when enabled this means that xapi will try to use a single NUMA node when creating VMs. It is best effort, meaning that this strategy sometimes fails and instead all nodes are used. Especially when many VMs are started or migrated at the same time. Test instructions are provided in the "What to test" section below.
    • Host evacuation was parallelized further, so that the migrating flow of VMs is maintained, avoiding bottlenecks.
    • Storage migration reworked: Allows migration from and to SMAPIv3 storage backends => a step towards SMAPIv3, but not immediately usable.
    • CLI interface: improved autocompletion (xe).
    • New HA option to avoid rebooting VMs on internal shutdown: https://docs.xcp-ng.org/management/ha/#halting-the-vm
  • A lot of other fixes and internal improvements.
• xen:
  • Enhance support for Intel Granite Rapids systems
  • Fix PCI passthrough on some systems
  • Add additional CPU RRD metrics
• xenserver-status-report: bug fixes + collection of additional debug data.
• xo-lite: update to version 0.15.0. See Xen Orchestra's release announcements for the changelog.

Other changes

• blktap: Add a log line when an operation is not supported.
• gpumon: Rebuilt for updated XAPI.
• libarchive: Fixed libarchive package to refresh the ldconfig cache, whose lack impacted driver disk generation
• samba: remove unneeded dependencies.
• swtpm: Rebuilt for updated libtpms.
• xcp-featured: Rebuilt for updated XAPI.
• xcp-python-libs: Various fixes.
• xha: Various fixes.

Added dependency:

• qcow-stream-tool: will be used by XAPI when support for the QCOW2 disk format is added.

XOSTOR
In addition to the changes in common packages, the following XOSTOR-specific packages received updates:

• kmod-drbd:
  • Update DRBD kernel module (improvements, fixes)
  • Important update to fix memory leaks during DRBD resource synchronization: This bug could be triggered during the addition of a node, or after an LINSTOR evacuation following a HW problem and a recreation of the resources.
• linstor:
  • Updated the LINSTOR controller and satellite packages.
  • In particular, a resizing issue that may block a resource has been fixed.

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

• blktap: 3.55.5-6.1.xcpng8.3
• edk2: 20220801-1.7.10.1.xcpng8.3
• ethtool: 4.19-3.xcpng8.3
• expat: 2.5.0-3.xcpng8.3
• gpumon: 24.1.0-65.1.xcpng8.3
• guest-templates-json: 2.0.14-1.1.xcpng8.3
• intel-microcode: 20250715-1.xcpng8.3
• kernel: 4.19.19-8.0.43.1.xcpng8.3
• libarchive: 3.3.3-1.1.xcpng8.3
• libtpms: 0.9.6-3.xcpng8.3
• lvm2: 2.02.180-18.2.1.xcpng8.3
• mellanox-mlnxen: 5.9_0.5.5.0-3.1.xcpng8.3
• qcow-stream-tool: 25.27.0-2.1.xcpng8.3
• qlogic-qla2xxx: 10.02.13.00_k-1.xcpng8.3
• samba: 4.10.16-25.2.xcpng8.3
• sm: 3.2.12-10.2.xcpng8.3
• swtpm: 0.7.3-9.xcpng8.3
• varstored: 1.2.0-3.1.xcpng8.3
• xapi: 25.27.0-2.1.xcpng8.3
• xcp-featured: 1.1.8-2.xcpng8.3
• xcp-python-libs: 3.0.8-1.1.xcpng8.3
• xen: 4.17.5-20.1.xcpng8.3
• xenserver-status-report: 2.0.15-1.xcpng8.3
• xha: 25.1.0-1.1.xcpng8.3
• xo-lite: 0.15.0-1.xcpng8.3

XOSTOR

• kmod-drbd-9.2.14-1.0.xcpng8.3
• linstor-common-1.29.2-1.el7_9
• linstor-controller-1.29.2-1.el7_9
• linstor-satellite-1.29.2-1.el7_9

What to test

Normal use and anything else you want to test.

Additional focus can be given to:

• Network boot of a UEFI VM from HTTPS server
• Network boot of a UEFI VM on a tagged VLAN (without handling the VLAN at the pool network level, as this makes it transparent to the VM)
• UEFI Secure Boot: new VMs, existing VMs, ... And/or just verify that you understand the updated documentation (work in progress)
• Testing on Intel Granite Rapids systems
• 50G/100G/200G link modes with Mellanox network adapters, using ethtool
• vTPM
• Hardware depending on the qla2xxx driver, that is Qlogic 2500/2600/2700/277x/2800 Series Fibre Channel Adapters
• NUMA's best effort mode, for hosts with more than one NUMA node:
  Turn it on, per host, by running xe host-param-set numa-affinity-policy=best_effort uuid=$HOST_UUID.
  Now new VMs should be able to be allocated to single NUMA nodes. To verify that it works, (re)start a VM that can fit in a single NUMA memory node, run xl debug-keys u on dom0, then check the output by running xl dmesg. The last lines of the output will show on which numa nodes each domain (VM) has memory allocated, you should see that the domain that was restarted, usually the one with the highest domain number; has several nodes below it and that all of them except one have the number 0.

Known issues

XAPI's handling of remote logging changed. XAPI now expects a configuration file in a specific location, and we haven't applied this system change yet. We'll publish the related update candidate to complement the current batch in the next days.

So: don't attempt to set up remote logging yet. If you set it up previously, then it should continue to work.

Test window before official release of the updates

~10 days. But please test as early as possible.

dinhngtu opened this pull request in xcp-ng/xcp-ng-org

closed Announce Secure Boot changes #328

@Forza What patch are you referring to that would relate to XOA?

I had this tab still open which lets me realize that despite we packaged vmfs6-tools and updated the documentation at https://docs.xcp-ng.org/installation/migrate-to-xcp-ng/#local-migration-same-host, we didn't inform you here.

Now it's done

OEL 8 & 9 wouldn't contain the fix unless they applied extra patches for this to the RHEL 8 & 9 kernel(s). I'll let the hypervisor team check the current status.

I've had this thread in my TODO list for long.

I'm not sure what the different options we have mean for users.

As r8125-module is installed by default, we'll want to avoid breaking it for existing users. Would using the new code risk causing regressions? Maybe that's what you meant but I'm not sure. Does our current r8125 driver support both 8125 and 8126, and not 8127? Or just 8125?

If we can't update the driver without causing regressions, then can we offer an alternate driver based on the new code for each chip (8125, 8126, 8127), and what amount of work might this represent?

The target would be XCP-ng 8.3. In XCP-ng 9.0 we'll have a newer kernel and newer drivers anyway.

CCing @Team-Hypervisor-Kernel for their opinion.

@olivierlambert said in Limiting access to xo-lite to a specific IP address or ssubnet:

About iptables, there's /etc/sysconfig/iptables but I'm not sure it's the right place to put manual modification, that's why I pinged the @Team-OS-Platform-Release

In a way that's the right place, but one needs to be careful with modifications there.

This changes nothing regarding brute-forcing. XAPI still listens to RPC requests on port 443.

Without changing iptables rules (that's not very flexible and could conflict with XAPI's handling of the rules), there's a way to disable the webserver.

https://docs.xcp-ng.org/management/manage-locally/xo-lite/#disabling-xo-lite

Also note that there is a simpler approach if you don't need an "all-in-one" ISO image: driver disks.

You can build driver disks using https://github.com/xcp-ng/driver-disks and load them during the installation.

You can also ask us to provide driver disks for alternate drivers for which we haven't built them yet. We make them available on https://mirrors.xcp-ng.org/isos/drivers/8.x/...

That's also something we need to document in the official docs. It's mentioned in the release notes for 8.3 but not in other sections of the documentation.

And fixes for XCP-ng 8.2 are coming but they require more work to backport the patches.

I moved this discussion to its own topic, as we need the other one for update candidate testing.

That's actually a question for @Team-XAPI-Network

@gduperrey said in XCP-ng 8.3 updates announcements and testing:

Since it doesn't indicate that an updated 8.2 is actually 8.2.1, only the major version is displayed.

Not a very good example actually (but what's true is that for 8.2 we didn't add a "LTS" mention next to the version number in the system either).

But yes, we decided not to increment the version number for XCP-ng 8.3.0. That breaks compatibility with some third party software because they wouldn't recognize a "8.3.1".

However we're working on a secondary version identifier that would allow you to know your precise patch level.