XCP-ng 8.3 Release Candidate 1

Five busy months after the release of XCP-ng 8.3 Beta 2, we are closer than ever to the final release with the XCP-ng 8.3 Release Candidate 1!

The Beta 2 announcement is still available here:

A release candidate is a product that we consider ready according to our quality standards and offer to our user base to gather the last bits of feedback. Since you, our users, represent a broader range of use cases and hardware than we could ever test internally, it's wise to have a final period of public testing before the official release.

This also means that now is the best time to join our existing wonderful testers and provide your feedback!

In this post, we'll walk you through the updates and enhancements from Beta 2 to RC 1 of XCP-ng 8.3 and discuss what the latest updates bring to the table.

But first, let's kick things off with a quick FAQ to address some of the most pressing questions you might have.

ℹ️ FAQ

Who is this release for?

1. Users who want to take part in the development of XCP-ng 8.3 through their tests and feedback.
2. Users who want to test the new features in XCP-ng 8.3 (and ideally, also provide feedback).

Where to provide feedback?

On the dedicated forum thread.

I already installed XCP-ng 8.3 Beta 2 (or any pre-release). How do I upgrade to 8.3 RC 1?

You can update as usual: through Xen Orchestra or yum update. Reboot. Check your SSH configuration (details below).

I want to upgrade an existing pool running XCP-ng 8.2.1. Can I simply update using yum?

No. Upgrading using just yum from earlier releases of XCP-ng is not supported for XCP-ng 8.3. Using the installation image to upgrade is mandatory.

So, when is the final release due?

It should be a matter of weeks now.

📀 Download

• The regular ISO is available here: https://mirrors.xcp-ng.org/isos/8.3/xcp-ng-8.3.0-rc1.iso
• And the netinstall here: https://mirrors.xcp-ng.org/isos/8.3/xcp-ng-8.3.0-rc1-netinstall.iso

SHA256 sums:

223247d0844c4877db9fbcfaba7a01584c074680bbaf47a67d6ac07067c41809  xcp-ng-8.3.0-rc1.iso
b6016acd9f9fd51684b542ac0720c926a418586218990ae79f8d542994be09c1  xcp-ng-8.3.0-rc1-netinstall.iso

✨ What's new in XCP-ng 8.3 RC1? (vs Beta2)

Since XCP-ng incorporates improvements from Xen, XAPI, and other projects, and is also closely related to XenServer, we can integrate, test, and validate changes from various sources while building a coherent and stable solution. Let's dive in!

Changes coming from XenServer and the XAPI Project

Since we released XCP-ng 8.3 Beta 2, XenServer has introduced their brand new XenServer 8 (yes, XenServer "8" is newer than Citrix Hypervisor 8.2 CU1). XCP-ng 8.3 RC1 integrates the open-source components of XenServer 8 along with various updates released since their initial version.

This includes the Xen developers' work to upgrade the core hypervisor component, Xen, from version 4.13.5 to version 4.17.4. Additionally, it incorporates the excellent work from the XAPI Project team (to which XCP-ng also contributes with code, tests, bug reports, enhancement proposals, and more).

The changelog up to xen-api 24.16.0 for XAPI is impressive, featuring numerous improvements and fixes. This includes significant groundwork to keep the component current, though many changes might not be immediately apparent to most users. Highlights include the completion of the transition to Python 3 for most Python components, reduction of log spam, introduction of a VM anti-affinity feature, various security fixes.

Also, many updates to several drivers, including: cisco-enic, cisco-fnic, intel-igc, microsemi-smartpqi and intel-ice.

While there are likely many other interesting changes worth mentioning, we also need to discuss what we at XCP-ng have been up to. So, let's address that now.

Changes coming directly from XCP-ng developers (and Xen Orchestra)

In addition to ensuring all components ported from the XAPI project and XenServer work seamlessly together (a more than significant task, taking months literally), our developers have introduced the following features and improvements:

🛰️ Xen Orchestra Lite (XO Lite) 0.2.3

We updated XO Lite to version 0.2.3. Improved treeview, some translations, and other changes.

🔐 Final touches to Guest UEFI SecureBoot Support

This has been a long journey, but it is finally coming to an end. Previously, we had to develop our own component, uefistored, which is still in use in XCP-ng 8.2.1. We decided to replace it with varstored to avoid unnecessary duplication of efforts, leading to some behavioral changes that required adaptation.

Another challenge is the license terms of the UEFI certificate databases distributed by Microsoft, which do not allow us to embed them directly in XCP-ng, as we want to maintain the Free Software spirit without additional restrictions. As a result, we need to guide our users through the process of setting up their pools for Guest Secure Boot. This process is as simple as running secureboot-certs install, but it needs to be done manually.

Here’s how it works now. Although it may seem a bit convoluted, it addresses various constraints while ensuring usability:

• For Secure Boot to be available to UEFI VMs on a pool, UEFI certificates need to be installed once.
• VMs get their copy of the certificates from the pool the first time they boot and are not updated by XCP-ng afterward.
• An unfortunate consequence is that some VMs booted before the certificates were installed on the pool or imported from another pool where they were missing are not ready for Secure Boot.

However, we expanded XAPI's features and collaborated with Xen Orchestra developers to offer the following helpers:

• A warning when attempting to enable Secure Boot on a pool that is not ready for it (already available in XO).
• Auto-detection of a VM's readiness for Secure Boot when users want to enable it (next XO release).
• A big button to propagate the pool's certificates to a VM's UEFI variable store if the certificates are missing for that particular VM (next XO release).
• Plus, links to our extensive Guest UEFI Secure Boot guide, fully updated to cover XCP-ng 8.3, will be provided in the user interface where needed.

🔑 OpenSSH, security, ciphers, and user configuration

In XCP-ng 8.3, unlike in XCP-ng 8.2.1, any updates to sshd_config or ssh_config would overwrite any changes you made to these files. This change came from XenServer 8, but we wanted to handle it differently, and we have done so.

Now, updates will not touch the files if you have modified them for your needs. To update default cipher lists, keys, and algorithms, we now make these changes directly at build time in the binaries. You can verify this with sshd -T.

💽 New (temporary) storage driver for 4kiB-block-only devices

We have added a new largeblock storage driver, a local SR driver that works around the current limitation of our storage stack with 4KiB-block-only devices by transparently emulating a 512B block size. More details can be found in this forum thread. This driver is also available in XCP-ng 8.2.1.

The goal was to make these devices quickly usable with XCP-ng, but ongoing work aims to provide a better, native storage driver based on the more recent storage stack, SMAPI v3.

🃏 PCI Passthrough API

We enhanced the API offered by XAPI by creating endpoints to add a given PCI device to the list of pass-through devices. Xen Orchestra developers also added UI components that allow you to add or remove these devices from the list. After a host reboot, these devices are released by the controller domain (dom0) and are available to be assigned to a guest.

📡 XO Enhanced coalesce detection

Those are changes introduced in Xen Orchestra after the Beta 2 of XCP-ng. Thanks to the storage API reporting improvements, we can now display many details, like if a SR is doing a coalesce operation, and even the coalesce progress in % in the task view!

🚫 XO Disk exclusion

Another visible improvement we contributed directly to XAPI is the ability to exclude disks in the VM.snapshot command. We now leverage this feature to avoid creating unnecessary disk snapshots during backups. This is implemented as the [NOBAK] feature, which conveniently excludes a virtual disk from backups. Additionally, we’ve introduced a new string, [NOSNAP], which allows you to ignore the disk during regular snapshots. You can combine both strings to exclude a disk from both VM snapshots and VM backups: [NOSNAP] [NOBAK].

🗜️ VM migration compression

For environments with slow network connections between hosts, you can now enable compression on the migration stream during live VM migrations. This feature uses additional resources on the involved hosts, but can significantly speed up the migration process if your network bandwidth is limited. However, since the effectiveness of this feature can vary, we recommend testing it in your environment before deciding to keep it enabled.

🆕 Other changes

• The installer now supports upgrading from a previous pre-release of XCP-ng 8.3.
• The installer has been made more robust when installing on used disks where ZFS or RAID metadata are still present.
• We updated the alternate "troubleshooting" kernel to version 4.19.316.
• The optional netdata package was updated to version 1.44.3.
• At the request of our technical support team, we now offer a port of the LTS 4.9 version of the mlx4_en driver for older Mellanox ConnectX cards, as a new optional driver package: mlx4-modules-alt. This is intended to address SR-IOV issues seen with the main driver.
• The web server provided by XAPI will now report the correct MIME type for SVG files, which is a requirement for XO Lite.
• We contacted all mirror providers to ensure they can offer both IPv4 and IPv6. The majority could comply, but a few could not, so we had to remove them from the list. We thank those providers for hosting XCP-ng mirrors all this time! Interested in hosting a mirror for XCP-ng? Here's how to apply!
• We updated the optional ZFS package to version 2.1.15. It is advised to run zpool upgrade after the update while all SRs on ZFS mount points are disconnected.
• Many under-the-hood changes and security fixes have been implemented.

🎯 Conclusion

The release of XCP-ng 8.3 RC1 is the result of a tremendous amount of effort over the past five months, mobilizing most of our XCP-ng team to deliver a solid and well-tested platform. Our commitment to quality is evident, as we have had mostly no incidents since the first alpha release, demonstrating our dedication to providing a turnkey and stable solution for rock-solid production environments. Despite the enormous amount of work that has gone into this release, your feedback as a user remains invaluable. We extend our heartfelt gratitude to our existing community for their tremendous feedback and support. As we enter the final lap before the official release, we encourage everyone to continue providing feedback to help us refine and perfect XCP-ng 8.3.

On a personal note (Olivier here!), I would like to thank both the XCP-ng and XO teams for working together to build, expose, and deliver all these features while maintaining a strong focus on quality and security. Kudos to them!