XCP-ng 8.2.1 (maintenance update) - final testing sprint

stormi

I'm creating this new thread that is the direct sequel to the previous 8.2.1 testing thread, to make it easier to start for anyone coming here from the blog to help.

XCP-ng 8.2.1 is about one last week of community testing from release. We need your help to make it happen.

What is XCP-ng 8.2.1?

It's just an update to XCP-ng 8.2. It's bigger than previous security or bugfix updates we released, but it's the same principle: a maintenance update. Once released, people running XCP-ng 8.2 will just update their hosts as usual.

Then why is it numbered 8.2.1?

Two reasons:

Among other changes, we included those from Citrix Hypervisor CU1, and the official version number changed to 8.2.1 there. You will see 8.2.1 instead of 8.2.0 as the minor version number in xsconsole, for example. But in the end it will just be an up to date 8.2 LTS.
At the same time as we will release updates for XCP-ng 8.2, we will release new installation ISOs that contain all the updated packages up to now. And also use it as an opportunity to fix a few bugs in the installer (anyone got stuck on the blue installation screen after choosing software RAID with 8.2 installation ISOs? ). So in a way, it's also a new release that deserves its own patch version number.

Is it an optional update, like Citrix Hypervisor 8.2 CU1 is?

No. Once released, it will be just another update to XCP-ng 8.2.

Are there new fancy features?

Well, it's a maintenance release so don't expect too much.

However:

Secure boot for VMs is now supported (full documentation here. Read it, really: there's a pool configuration step necessary if you want to enable secure boot). Some of you already tested this feature months ago. Now it's time for it to reach everyone. There's a but: XCP-ng's guest tools are not signed with a recent enough certificate and Microsoft's user support is so bad that we haven't been able to get a new signing certificate in months (there are issues with their own website that they have trouble finding a workaround for)! So for now the basic rule is: if you want to enable Secure Boot on a Windows VM, use the signed guest tools from Citrix.
The installation of Microsoft's infamous KB4535680 update is now fixed.
A few components like qemu are updated to pave the way to future vTPM support. I said future. It's not available yet.
Rocket Lake processors are now supported.

Other notable changes

Guest template for Windows Server 2022 added.
Log rotation. Log files should now be automatically rotated if they reach a size of 100M, without waiting for the daily log rotation. This will better handle the situations where a single log file grows up very fast to the point of filling the log partition.
Updated default drivers on the system:
- avago-mpt3sas-33.100.00.01
- intel-ice-1.6.4 (new vendor driver RPM. We were previously using the built-in driver from kernel 4.19)
- intel-ixgbe-5.9.4
- qlogic-fastlinq-8.42.10.0
- qlogic-netxtreme2-7.14.76
- r8125-module-9.003.05 added for some Realtek NICs (might be removed from the final release if we can't get enough feedback)
- igc-module-4.20.17 added for Intel i225 NICs (might be removed from the final release if we can't get enough feedback or can't fix the remaining issues)
The default console menu, xsconsole, was updated and includes an improvement that we had contributed upstream: when the XAPI service is unreachable on the host, xsconsole will try to display a useful error message, rather than displaying a misleading message saying that no network was configured.
A bug that we discovered and reported upstream regarding the handling of web pages over HTTPS on the host when HTTP support was forbidden has been fixed, so I could finally enforce HTTPS for the host's web page. Any request to get the web page on port 80 will reply with a 403 error.
@r1 has updated our alternate kernel to a much more recent maintenance release of kernel.org's 4.19 branch.
samba and openssl were updated, which fixes various CVEs. The update to the samba packages pulled several new dependencies such as gnutls, nettle, python-tdb, ...
xenserver-transfer-vm was removed.

How to test

Either install or upgrade a host using the test installation ISOs found at https://mirrors.xcp-ng.org/tmp/

Or update an existing XCP-ng 8.2 host:

create a file named xcp-ng-staging.repo.

[xcp-ng-staging]
name=XCP-ng Staging Repository
baseurl=http://mirrors.xcp-ng.org/8/8.2/staging/x86_64/ http://updates.xcp-ng.org/8/8.2/staging/x86_64/
enabled=0
gpgcheck=1
repo_gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-xcpng

Update: yum update --enablerepo=xcp-ng-staging
Usual instructions from https://xcp-ng.org/docs/updates.html still apply

What to test

As usual, anything that you need XCP-ng for.

We also would like you to give special focus to the following items:

UEFI VMs, without Secure Boot
UEFI VMs, with Secure Boot (check the docs. There's a manual command to run once on the pool, to download and install the certificates from Microsoft.)
On Windows installed from a not too recent image (otherwise the test is impossible), installation of update KB4535680, which updates the list of revocated certificates for Secure Boot. Should work without Secure Boot on, but we had reports of failures in this situation so I'm interested in finding a way to reproduce. Should also work with Secure Boot on.
Log rotation if you have a way to trigger very verbose logs.
The installer (installation, upgrade, backup restore...).
Active Directory connectivity, if you know how to make it work
The alternate kernel. Having it tested on a large variety of hardware would be good.

How to come back to the main update track after the tests?

You will have nothing to do to turn a host with this test version of X. Just continue updating your hosts normally in the future.

Ready... Test!

And of course, ask anything.

JeffBerntsen

@stormi

Testing UEFI VMs. So far working fine without secure boot. Having a problem with secure boot under Alpine Linux. They don't use MS certificates and a shim as a lot of distros do but instead have you generate a set of keys for your installation and then enroll them. It looks like this is a problem with enrolling the generated keys in the TianoCore boot firmware. I'll try with a different distro and see if it's any better with something different.

stormi

@JeffBerntsen To enrol keys for a specific VM, there's doc.

JeffBerntsen

@stormi That worked to get the auth files generated using Alpine's instructions enrolled as far as I can tell but switching the VM to secure boot after that still fails, dropping me into a UEFI shell. Alpine 3.15 is the first version with secure boot support and it's possible there are still some glitches there.

Instead of that, I'm now trying to set up a secure boot with a fresh install of OpenSUSE leap 15.3 which I know does support secure boot and will see if that works out.

stormi

@JeffBerntsen Here we have a test that generates keys and signs the boot binaries with them, if you want to check how we did. Works on many linux distros including alpine (3.12.0): https://github.com/xcp-ng/xcp-ng-tests/blob/master/tests/uefistored/test_secure_boot.py#L142

Tumbleweed 15.3 should work out of the box with the defaults certs installed by secureboot-certs install (that include the latest dbx - revocation list - from Microsoft).

JeffBerntsen

@stormi Thanks, I'll give the test script a try on my test Alpine installation and see if it works for me.

My OpenSUSE Leap 15.3 installation works just fine via secure boot with one warning/error message at boot. It's complaining that it can't generate a temporary hibernation key because of a missing EFI_RNG_PROTOCOL. Except for that, it works great under secure boot. If not being able to have hibernation support in the VM's operating system is the only issue, that's definitely minor and something I don't use and won't miss.

EDIT: I'm also going to try a fresh installation of Alpine into a VM set for secure boot and see how that works out. My test was trying to convert an existing VM that was successfully booting under UEFI without secure boot enabled.

EDIT 2: I've managed to get Alpine working as well. It appears that their Wiki entry on setting up secure boot isn't quite right yet. They have a utility which generates keys and creates a signed unified boot image. My best guess is that there is some problem with the signature on the boot image. I was able to get things working by enrolling the generated auth files for the VM uuid on the host system then booting the VM with secure boot disabled and using the sbsign utility to sign the boot image with the generated db key and certificate. It adds a second signature to the boot image which appears to be identical to the first one. Switching to secure boot mode and rebooting works on the re-signed boot image.

theAeon

Bumping my lab to staging right now-if you don't hear back, assume everything works fine.

stormi

It doesn't look like my blog post brought a lot of new testers.

There's still time (a few days) to lend a hand for this 8.2.1 release and test it. I don't think the alternate kernel got a lot of attention outside Vates. Nor AD connectivity (but maybe no one uses this, or they connect their XO instead which might be better).

I'm currently building new ISOs (test6) that will probably be the final ones. The only difference with test5 is that I removed the igc and r8125 drivers due to issues with the first one and lack of feedback on the second one. We'll continue working on improved hardware support after the release.

If you installed XCP-ng 8.2.1 using the test5 installation ISO, you need to follow these steps (other testers, just dismiss):

yum downgrade vendor-drivers
yum update vendor-drivers # should do nothing. Just in case.
yum remove igc-module r8125-module # unless you need them

gskger

@stormi Not much of a help this time, cause my job keeps me way too busy. Anyway, I upgraded my two host playlab the day you released the latest version (via the yum update route with staging repo). Everything updated fine and works as expected since then, but I cannot contribute to the specific test items you asked for.

stormi

@gskger If you can find time for it, you can just update to the latest state of the staging branch with yum update --enablerepo=xcp-ng-staging. Else no problem.

stormi

New installation ISOs (test6) are available at https://updates.xcp-ng.org/tmp/. The netinstall repository was also updated.

The only changes since the last ones are the removal of igc and r8125 drivers that I had attempted to add in test5.

These should be the final ones, so it's always good if some of you can test them one last time before the release.

JeffBerntsen

@stormi Some quick testing of the alternate kernel on my test systems seems to be working fine with the not-unexpected issue that the XOSTOR test does not come up and run on it.

gskger

@stormi That was an easy 2.8k update on both hosts with no problem. VMs continue to run without any issues so far.

rus2lan

@stormi https://www.asus.com/Motherboards-Components/Motherboards/TUF-Gaming/TUF-GAMING-Z690-PLUS-WIFI-D4/HelpDesk_QVL_CPU/ for this motherboard igc drivers work only for xcp, i have trouble in VM with VLANs: DHCP work, but no ping to gateway...

stormi

@rus2lan The igc driver we backported from the 4.20 kernel doesn't appear to be working well indeed. That's why I did not include it in the final release of XCP-ng 8.2.1 ISOs.

stormi

XCP-ng 8.2.1 is now released. A huge thanks to everyone who tested and gave feedback to us.

https://xcp-ng.org/blog/2022/02/28/xcp-ng-8-2-1-update/

apz

I upgraded 3 of my homelab hosts, all were up-to-date 8.2's before this update. One of them blurted out this right at the end of the upgrade, but I did not observe any negative consequences yet.

  Cleanup    : wsproxy-1.12.0-2.xcpng8.2.x86_64                                                                                                                                       162/162 
Traceback (most recent call last):
  File "/bin/create-guest-templates", line 17, in <module>
    loader.insert_templates()
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 189, in insert_templates
    self._insert_template(i)
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 159, in _insert_template
    conn.request("PUT", "/import_metadata?" + params, tar)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 857, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe

stormi

@apz The script that deletes then recreates the guest templates when they are updated apparently failed on your host. Are there any missing templates in your template list?

apz

@stormi The affected host has only 2 templates, 2022 Windows and Suse 12.

stormi

@apz Try to re-run the script that failed:

/usr/bin/create-guest-templates-wrapper