XCP-ng 8.3 betas and RCs feedback 🚀

exetico

@stormi I've hold back my fingers, and didn't press the "reboot button".

But, if I need to, will downgrading edk2 make anything break? I guess my pfSense instance will fail after a reboot (like described), so... Hopefully I don't need to reboot my host in the next weeks ? Maybe it's a good idea to note, to not push a update like this, so close to Christmas.

Happy Christmas, and Merry New Year .

stormi

@exetico edk2 was initially updated by XenServer to fix a VNC console corruption when vGPU was in use. As we don't support vgpu at the moment (it requires proprietary components), I doubt going back to the previous known to work version will cause any regressions. But there's always some risk.

wilsonqanda

FYI for anyone doing this the snippet of code on a 1 host 1 pool configuration no issue. If you run this code on a multiple hosts in 1 pool configuration make sure to run it all on each host that is affected by the update.

wilsonqanda

@exetico it didnt break for me lol and all 4 of my xcp-ng and multiple pfsense VMs

wilsonqanda

Though as an FYI it is annoying but it show that single upgrade needed in the pool lol that you just downgrade lol. So best not to upgrade until it is resolved in the 1st place. FYI cant move the vm even to another version of 8.3b (original iso version) that doesn't have the updated 12/21 patches as it's not same version.

Obviously after the HOSTs in the pool are updated (patch 12/21) VMs can be migrated but they all need to be on same version.

I thought 8.3b can migrate between all 8.3b of various updates but definitely not the case here. Which I think is a pretty big issues but oh well.

archw

@wilsonqanda Stormi found the issue with edk2 and got me running!

wilsonqanda

@archw Thanks for the update yep he got me working as well. It works like a charm

stormi

New batch of updates for XCP-ng 8.3 beta

So, as several already found out, I pushed many updated packages to XCP-ng 8.3's RPM repositories at the same time. This is the result of the last six months of work on XCP-ng 8.3.

Changes coming from XenServer

This update includes everything published by XenServer between the release of XCP-ng 8.3 Beta1 and October 2023. Newer changes will be brought in after Beta2.

I am a bit late and sadly don't have time now to detail all their changes, but the most important one is maybe completion of vTPM support (and thus compatibility with Windows 11). Previously, it would work, but many features were still missing. Now, snapshots, VM export/import and live migration all are supported for UEFI VMs with a vTPM. A Windows 11 template is available, which will transparently create the required vTPM when you create a new VM.

Changes coming from XCP-ng

On XCP-ng's side, in no particular order:

We rebased all packages on XenServer 8 preview + all updates up to early October.
Installer
- rebased on latest upstream release.
- better granularity in error messages displayed to users when install fails due to wrong system date, signature issues, etc.
- re-allow interactive installation of driver disks on host during installation.
- Avoid failed services in the installer (there were benign "failed" messages for services the installer doesn't actually use)
IPv6 support continued
- IPv6 testing / automated tests. Not run automatically on our internal CI yet due to setup difficulties.
- Fixed live migration failure due to upstream bug in ocaml-uri. Fix contributed upstream to ocaml-uri.
- Allow to use a CIDR for VIFs IPv4 and IPv6 allowed IPs. Feature contributed upstream to XAPI project.
- xsconsole : allow to configure IPv6 via autoconf for the management interface. Feature contributed upstream to XAPI project.
Making XOSTOR available in XCP-ng 8.3. Another update is pending, so better wait for it.
Installer image generation: fixes and improvements.
VLAN display in xsconsole. Feature contributed upstream to XAPI project.
smartmontools updated to version 7.
Plugin added to use the data from smartmontools 7 in JSON format (initially contributed by Cécile, then improved by one of our developers)
Debian 12 template added.
Security fixes. However, no security fix was made to XCP-ng 8.3 in the last weeks, while we were busy testing and stabilizing.
Added new tests to our test suite. For example this new test which exercises the vTPM features. Windows 11 is now also tested in our CI/CD.
Deleted the old, unsupported since XCP-ng 8.1, experimental EXT4 driver. We're talking about an old experimental driver that you never used unless you installed the experimental packages in the XCP-ng 7.x era.
As in 8.2.1, lsscsi added to our repositories.
As in 8.2.1, more alternate drivers packaged.
As in 8.2.1, drivers updated and added to default installation: mpi3mr, igc, r8125 (+firmware)
xo-lite RPM added and installed by default.
UEFI and SecureBoot support:
- The version of XAPI included in XCP-ng 8.3 after Beta1 provides a new way to handle UEFI certificates at the pool level, implemented by XenServer developers while trying to take both their and our needs into account.
- We then adapted the automated tests for XCP-ng 8.3, based on new XAPI behaviour.
- Running the adapted tests revealed issues (one bug, but also more importantly that we did not fully understand each other regarding XCP-ng's needs). So we proposed a new design to the XAPI project, then implemented it. This hopefully concludes more than a year of intermittent work on this very topic.
- From a user standpoint, despite all the work behind the scene, there aren't many changes. Mostly retaining the ability we had in XCP-ng 8.2.1 for users to download and install the default certificates from Microsoft (must run secureboot-certs install at least once on the pool after this update), or install their own.
Various diagnoses and fixes.

Known issues

Several users reported issues with UEFI support. UEFI, again! Just when we thought we were done with it. According to what we found out together, the issue is not related to the changes I was mentioning above, which are limited to certificate management. Here, it's apparently a piece of software emulating a UEFI firmware for VMs, edk2, which regressed in some situations.

So, the current situation, as I know it today:

For various users, it works, but it looks like pfSense (and maybe FreeBSD) in a UEFI VM has trouble starting.
In our internal tests, UEFI works for all the OSes we tested. But currently FreeBSD is only tested in BIOS mode, so we wouldn't have seen it. We'll improve our test coverage.
For one user, it manifested in a way that no UEFI VM would ever start, whatever the OS.

For now, the workaround is to downgrade the component which seems faulty, on every host: yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3. Note that Xen Orchestra, or yum update, will attempt to reinstall the more recent version of the package, so better avoid updating again until we've solved these issues.

Updated installer (XCP-ng 8.3 Beta2)

As soon as any remaining blockers are lifted, this batch of updates will also be released in a XCP-ng 8.3 Beta2 installation ISO image.

You may download a test build of the installation ISOs: https://mirrors.xcp-ng.org/tmp/xcp-ng-8.3.0-beta2-test2.iso

SHA256SUM: 11ed1ca5d757177e0edab6c7a7a371306e3ddcdfab7df62b6f076e527a7a5015

/!\ The netinstall repository won't be updated until Beta2 is released, so any network installation using the main mirrors will actually install XCP-ng Beta1, not Beta2.

john.c

@stormi Is it possible to pull the bad edk2 package or at least for problematic systems send a package update which excludes in yum the bad update package of edk2?

Any way I have suggested to the developers of dnf to implement some kind of compatibility hold system, or possibility of a bad packages list file for repository meta-data. This info would be used by default (though overridable) when using dnf install and dnf update, so bad packages are automatically excluded from installation and/or system updates.

In principal it would operate in a similar fashion to how dandelions hold onto bad seeds, to keep them from spreading. So a bad packages list would be used by rpm repositories and dnf to keep from spreading the bad packages to more systems, thus improving system stability, reliability and/or preventing regressions.

If you wish to give update or assistance on implementing this request upstream you can find the issue here (https://github.com/rpm-software-management/dnf/issues/2029). Also may be worth considering and/or working on transitioning to dnf for XCP-ng as well as updating the appropriate parts of the software stack in the hypervisor, and Xen Orchestra.

MrGrymReaper created this issue in rpm-software-management/dnf

closed Feature Request - Compatibility safe guard holds or bad package meta data list as file. #2029

stormi

We already retain a lot of packages until they pass our internal tests successfully, but at some point we need to get feedback from users, and that's what happened here, revealing at the same time that we need to add pfSense to the OSes we test internally. Better find issues now than after the final release, and retaining packages would limit the amount of testing they receive. There's a known workaround for people who are really stuck, so I didn't pull the package back.

stormi

There is a new build of edk2 available for testing. We don't know yet if it fixes all the issues users met, so we need your feedback.

Install with:

yum update edk2 --enablerepo=xcp-ng-testing,xcp-ng-ci

If you need to rollback:

yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3

I'm especially interested in @archw's feedback, since their issue was not just with pfSense but with any UEFI VM instead. But I'm also interested in feedback from UEFI pfSense users.

Andrew

@stormi New update version edk2-20220801-1.7.3.xcpng8.3.x86_64 fixed my FreeBSD UEFI boot problem (stuck on boot countdown). XCP reboot NOT required after install. Windows 10 and Ubuntu boot correctly also (but did before too). I did not test secure boot or bitlocker.

olivierlambert

Great news

matt-plan8

Hey gents,

Has anyone else noticed that if you have a Win 11 guest (pro or enterprise) with the vtpm, that's Entra (Azure) joined, that when migrating to a diff host it breaks the Azure AD membership?

I'll try to dig up some more specifics and run a more thorough test, but I've run into it twice now on a VM I've been running on my test lab with 8.3 beta. Relatively simple solution, leave Azure domain, rejoin, but I suspect that that the token gets saved in vtpm and is either lost or invalidated by migrating hosts.

olivierlambert

That's a good question Are you migrating in the same pool or to a different pool?

matt-plan8

@olivierlambert

Same pool, I have 3 minisforum pc's I've been messing around with to learn XCP-NG, they aren't in a cluster because I got lazy after installation, but are connected via truenas iscsi as central iscsi storage.

Loving the system overall, I could never go back to hyper-v...

I was patching manually by moving machines and then using the gui single host patch button. I had an issue with rolling patches, and didn't really want to bug you guys over it, it's likely user error. Setup is installed from sources.

stormi

@matt-plan8 I don't understand. Pool and cluster are the same thing. Pool for XCP-ng is what VMWare calls cluster.

matt-plan8

@stormi sorry I consider HA a cluster, pool I consider just a group of machines in the same management domain.

They are in a single pool, they aren't HA

archw

@stormi said in XCP-ng 8.3 beta :

yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3

It didn't work. It just sits there with the attached screen.

stormi

@archw You're quoting the downgrade command. May I assume you tried yum update edk2 --enablerepo=xcp-ng-testing,xcp-ng-ci instead and got version edk2-20220801-1.7.3.xcpng8.3.x86_64?