XCP-ng 8.3 betas and RCs feedback 🚀

archw

@wilsonqanda Stormi found the issue with edk2 and got me running!

wilsonqanda

@archw Thanks for the update yep he got me working as well. It works like a charm

stormi

New batch of updates for XCP-ng 8.3 beta

So, as several already found out, I pushed many updated packages to XCP-ng 8.3's RPM repositories at the same time. This is the result of the last six months of work on XCP-ng 8.3.

Changes coming from XenServer

This update includes everything published by XenServer between the release of XCP-ng 8.3 Beta1 and October 2023. Newer changes will be brought in after Beta2.

I am a bit late and sadly don't have time now to detail all their changes, but the most important one is maybe completion of vTPM support (and thus compatibility with Windows 11). Previously, it would work, but many features were still missing. Now, snapshots, VM export/import and live migration all are supported for UEFI VMs with a vTPM. A Windows 11 template is available, which will transparently create the required vTPM when you create a new VM.

Changes coming from XCP-ng

On XCP-ng's side, in no particular order:

We rebased all packages on XenServer 8 preview + all updates up to early October.
Installer
- rebased on latest upstream release.
- better granularity in error messages displayed to users when install fails due to wrong system date, signature issues, etc.
- re-allow interactive installation of driver disks on host during installation.
- Avoid failed services in the installer (there were benign "failed" messages for services the installer doesn't actually use)
IPv6 support continued
- IPv6 testing / automated tests. Not run automatically on our internal CI yet due to setup difficulties.
- Fixed live migration failure due to upstream bug in ocaml-uri. Fix contributed upstream to ocaml-uri.
- Allow to use a CIDR for VIFs IPv4 and IPv6 allowed IPs. Feature contributed upstream to XAPI project.
- xsconsole : allow to configure IPv6 via autoconf for the management interface. Feature contributed upstream to XAPI project.
Making XOSTOR available in XCP-ng 8.3. Another update is pending, so better wait for it.
Installer image generation: fixes and improvements.
VLAN display in xsconsole. Feature contributed upstream to XAPI project.
smartmontools updated to version 7.
Plugin added to use the data from smartmontools 7 in JSON format (initially contributed by Cécile, then improved by one of our developers)
Debian 12 template added.
Security fixes. However, no security fix was made to XCP-ng 8.3 in the last weeks, while we were busy testing and stabilizing.
Added new tests to our test suite. For example this new test which exercises the vTPM features. Windows 11 is now also tested in our CI/CD.
Deleted the old, unsupported since XCP-ng 8.1, experimental EXT4 driver. We're talking about an old experimental driver that you never used unless you installed the experimental packages in the XCP-ng 7.x era.
As in 8.2.1, lsscsi added to our repositories.
As in 8.2.1, more alternate drivers packaged.
As in 8.2.1, drivers updated and added to default installation: mpi3mr, igc, r8125 (+firmware)
xo-lite RPM added and installed by default.
UEFI and SecureBoot support:
- The version of XAPI included in XCP-ng 8.3 after Beta1 provides a new way to handle UEFI certificates at the pool level, implemented by XenServer developers while trying to take both their and our needs into account.
- We then adapted the automated tests for XCP-ng 8.3, based on new XAPI behaviour.
- Running the adapted tests revealed issues (one bug, but also more importantly that we did not fully understand each other regarding XCP-ng's needs). So we proposed a new design to the XAPI project, then implemented it. This hopefully concludes more than a year of intermittent work on this very topic.
- From a user standpoint, despite all the work behind the scene, there aren't many changes. Mostly retaining the ability we had in XCP-ng 8.2.1 for users to download and install the default certificates from Microsoft (must run secureboot-certs install at least once on the pool after this update), or install their own.
Various diagnoses and fixes.

Known issues

Several users reported issues with UEFI support. UEFI, again! Just when we thought we were done with it. According to what we found out together, the issue is not related to the changes I was mentioning above, which are limited to certificate management. Here, it's apparently a piece of software emulating a UEFI firmware for VMs, edk2, which regressed in some situations.

So, the current situation, as I know it today:

For various users, it works, but it looks like pfSense (and maybe FreeBSD) in a UEFI VM has trouble starting.
In our internal tests, UEFI works for all the OSes we tested. But currently FreeBSD is only tested in BIOS mode, so we wouldn't have seen it. We'll improve our test coverage.
For one user, it manifested in a way that no UEFI VM would ever start, whatever the OS.

For now, the workaround is to downgrade the component which seems faulty, on every host: yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3. Note that Xen Orchestra, or yum update, will attempt to reinstall the more recent version of the package, so better avoid updating again until we've solved these issues.

Updated installer (XCP-ng 8.3 Beta2)

As soon as any remaining blockers are lifted, this batch of updates will also be released in a XCP-ng 8.3 Beta2 installation ISO image.

You may download a test build of the installation ISOs: https://mirrors.xcp-ng.org/tmp/xcp-ng-8.3.0-beta2-test2.iso

SHA256SUM: 11ed1ca5d757177e0edab6c7a7a371306e3ddcdfab7df62b6f076e527a7a5015

/!\ The netinstall repository won't be updated until Beta2 is released, so any network installation using the main mirrors will actually install XCP-ng Beta1, not Beta2.

john.c

@stormi Is it possible to pull the bad edk2 package or at least for problematic systems send a package update which excludes in yum the bad update package of edk2?

Any way I have suggested to the developers of dnf to implement some kind of compatibility hold system, or possibility of a bad packages list file for repository meta-data. This info would be used by default (though overridable) when using dnf install and dnf update, so bad packages are automatically excluded from installation and/or system updates.

In principal it would operate in a similar fashion to how dandelions hold onto bad seeds, to keep them from spreading. So a bad packages list would be used by rpm repositories and dnf to keep from spreading the bad packages to more systems, thus improving system stability, reliability and/or preventing regressions.

If you wish to give update or assistance on implementing this request upstream you can find the issue here (https://github.com/rpm-software-management/dnf/issues/2029). Also may be worth considering and/or working on transitioning to dnf for XCP-ng as well as updating the appropriate parts of the software stack in the hypervisor, and Xen Orchestra.

MrGrymReaper created this issue in rpm-software-management/dnf

closed Feature Request - Compatibility safe guard holds or bad package meta data list as file. #2029

stormi

We already retain a lot of packages until they pass our internal tests successfully, but at some point we need to get feedback from users, and that's what happened here, revealing at the same time that we need to add pfSense to the OSes we test internally. Better find issues now than after the final release, and retaining packages would limit the amount of testing they receive. There's a known workaround for people who are really stuck, so I didn't pull the package back.

stormi

There is a new build of edk2 available for testing. We don't know yet if it fixes all the issues users met, so we need your feedback.

Install with:

yum update edk2 --enablerepo=xcp-ng-testing,xcp-ng-ci

If you need to rollback:

yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3

I'm especially interested in @archw's feedback, since their issue was not just with pfSense but with any UEFI VM instead. But I'm also interested in feedback from UEFI pfSense users.

Andrew

@stormi New update version edk2-20220801-1.7.3.xcpng8.3.x86_64 fixed my FreeBSD UEFI boot problem (stuck on boot countdown). XCP reboot NOT required after install. Windows 10 and Ubuntu boot correctly also (but did before too). I did not test secure boot or bitlocker.

olivierlambert

Great news

matt-plan8

Hey gents,

Has anyone else noticed that if you have a Win 11 guest (pro or enterprise) with the vtpm, that's Entra (Azure) joined, that when migrating to a diff host it breaks the Azure AD membership?

I'll try to dig up some more specifics and run a more thorough test, but I've run into it twice now on a VM I've been running on my test lab with 8.3 beta. Relatively simple solution, leave Azure domain, rejoin, but I suspect that that the token gets saved in vtpm and is either lost or invalidated by migrating hosts.

olivierlambert

That's a good question Are you migrating in the same pool or to a different pool?

matt-plan8

@olivierlambert

Same pool, I have 3 minisforum pc's I've been messing around with to learn XCP-NG, they aren't in a cluster because I got lazy after installation, but are connected via truenas iscsi as central iscsi storage.

Loving the system overall, I could never go back to hyper-v...

I was patching manually by moving machines and then using the gui single host patch button. I had an issue with rolling patches, and didn't really want to bug you guys over it, it's likely user error. Setup is installed from sources.

stormi

@matt-plan8 I don't understand. Pool and cluster are the same thing. Pool for XCP-ng is what VMWare calls cluster.

matt-plan8

@stormi sorry I consider HA a cluster, pool I consider just a group of machines in the same management domain.

They are in a single pool, they aren't HA

archw

@stormi said in XCP-ng 8.3 beta :

yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3

It didn't work. It just sits there with the attached screen.

stormi

@archw You're quoting the downgrade command. May I assume you tried yum update edk2 --enablerepo=xcp-ng-testing,xcp-ng-ci instead and got version edk2-20220801-1.7.3.xcpng8.3.x86_64?

archw

@stormi
Sorry...I cliked reply to teh wrong post.

Yes, I ran "yum update edk2 --enablerepo=xcp-ng-testing,xcp-ng-cii" and let it install. I then rebooted the VM (several of them) and tried to start them and none worked. I then ran "yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3" and the VM's fired right up.

stormi

I just published new updates for XCP-ng 8.3:

edk2-20220801-1.7.3.xcpng8.3, which some of you already tested. It fixes UEFI pfSense booting.
e2fsprogs-1.47.0-1.1.xcpng8.3 (and subpackages): brought to the same level as latest XCP-ng 8.2.1. If you installed using the latest test installation ISO, you already have it.
r8125-module-9.012.04-1.xcpng8.3 which updates the r8125 driver.

gskger

@stormi While I can not test the changes, the update went well. However, the restart took a few minutes longer than expected (or I was impatient again).

wilsonqanda

@stormi Nice thanks so I can run the complete update on XOA and not worry about having to remove that single edk2 anymore that wasn't working. Will do that later

stormi

So, @ThierryEscande found the issue which made all UEFI VMs crash on @archw's server and patched it.

This was the last issue before we could publish XCP-ng 8.3 Beta 2, so this is good news.

Now, we need you all to test the patched RPM and check your UEFI VMs still work well with it: start, function normally...

To install it:

yum update edk2-20220801-1.7.3.1.xcpng8.3 --enablerepo=xcp-ng-ci

No need to reboot, but UEFI VMs need to be restarted afterwards (when doable. If you have a specfic VM that you can't reboot yet, it's OK to leave it alone).

In case things do not work as expected, you can come back to the previous build with :

yum downgrade edk2