XCP-ng 8.2.1 (maintenance update) - final testing sprint

JeffBerntsen

Testing UEFI VMs. So far working fine without secure boot. Having a problem with secure boot under Alpine Linux. They don't use MS certificates and a shim as a lot of distros do but instead have you generate a set of keys for your installation and then enroll them. It looks like this is a problem with enrolling the generated keys in the TianoCore boot firmware. I'll try with a different distro and see if it's any better with something different.

stormi

@JeffBerntsen To enrol keys for a specific VM, there's doc.

JeffBerntsen

@stormi That worked to get the auth files generated using Alpine's instructions enrolled as far as I can tell but switching the VM to secure boot after that still fails, dropping me into a UEFI shell. Alpine 3.15 is the first version with secure boot support and it's possible there are still some glitches there.

Instead of that, I'm now trying to set up a secure boot with a fresh install of OpenSUSE leap 15.3 which I know does support secure boot and will see if that works out.

stormi

@JeffBerntsen Here we have a test that generates keys and signs the boot binaries with them, if you want to check how we did. Works on many linux distros including alpine (3.12.0): https://github.com/xcp-ng/xcp-ng-tests/blob/master/tests/uefistored/test_secure_boot.py#L142

Tumbleweed 15.3 should work out of the box with the defaults certs installed by secureboot-certs install (that include the latest dbx - revocation list - from Microsoft).

JeffBerntsen

@stormi Thanks, I'll give the test script a try on my test Alpine installation and see if it works for me.

My OpenSUSE Leap 15.3 installation works just fine via secure boot with one warning/error message at boot. It's complaining that it can't generate a temporary hibernation key because of a missing EFI_RNG_PROTOCOL. Except for that, it works great under secure boot. If not being able to have hibernation support in the VM's operating system is the only issue, that's definitely minor and something I don't use and won't miss.

EDIT: I'm also going to try a fresh installation of Alpine into a VM set for secure boot and see how that works out. My test was trying to convert an existing VM that was successfully booting under UEFI without secure boot enabled.

EDIT 2: I've managed to get Alpine working as well. It appears that their Wiki entry on setting up secure boot isn't quite right yet. They have a utility which generates keys and creates a signed unified boot image. My best guess is that there is some problem with the signature on the boot image. I was able to get things working by enrolling the generated auth files for the VM uuid on the host system then booting the VM with secure boot disabled and using the sbsign utility to sign the boot image with the generated db key and certificate. It adds a second signature to the boot image which appears to be identical to the first one. Switching to secure boot mode and rebooting works on the re-signed boot image.

theAeon

Bumping my lab to staging right now-if you don't hear back, assume everything works fine.

stormi

It doesn't look like my blog post brought a lot of new testers.

There's still time (a few days) to lend a hand for this 8.2.1 release and test it. I don't think the alternate kernel got a lot of attention outside Vates. Nor AD connectivity (but maybe no one uses this, or they connect their XO instead which might be better).

I'm currently building new ISOs (test6) that will probably be the final ones. The only difference with test5 is that I removed the igc and r8125 drivers due to issues with the first one and lack of feedback on the second one. We'll continue working on improved hardware support after the release.

If you installed XCP-ng 8.2.1 using the test5 installation ISO, you need to follow these steps (other testers, just dismiss):

yum downgrade vendor-drivers
yum update vendor-drivers # should do nothing. Just in case.
yum remove igc-module r8125-module # unless you need them

gskger

@stormi Not much of a help this time, cause my job keeps me way too busy. Anyway, I upgraded my two host playlab the day you released the latest version (via the yum update route with staging repo). Everything updated fine and works as expected since then, but I cannot contribute to the specific test items you asked for.

stormi

@gskger If you can find time for it, you can just update to the latest state of the staging branch with yum update --enablerepo=xcp-ng-staging. Else no problem.

stormi

New installation ISOs (test6) are available at https://updates.xcp-ng.org/tmp/. The netinstall repository was also updated.

The only changes since the last ones are the removal of igc and r8125 drivers that I had attempted to add in test5.

These should be the final ones, so it's always good if some of you can test them one last time before the release.

JeffBerntsen

@stormi Some quick testing of the alternate kernel on my test systems seems to be working fine with the not-unexpected issue that the XOSTOR test does not come up and run on it.

gskger

@stormi That was an easy 2.8k update on both hosts with no problem. VMs continue to run without any issues so far.

rus2lan

@stormi https://www.asus.com/Motherboards-Components/Motherboards/TUF-Gaming/TUF-GAMING-Z690-PLUS-WIFI-D4/HelpDesk_QVL_CPU/ for this motherboard igc drivers work only for xcp, i have trouble in VM with VLANs: DHCP work, but no ping to gateway...

stormi

@rus2lan The igc driver we backported from the 4.20 kernel doesn't appear to be working well indeed. That's why I did not include it in the final release of XCP-ng 8.2.1 ISOs.

stormi

XCP-ng 8.2.1 is now released. A huge thanks to everyone who tested and gave feedback to us.

https://xcp-ng.org/blog/2022/02/28/xcp-ng-8-2-1-update/

apz

I upgraded 3 of my homelab hosts, all were up-to-date 8.2's before this update. One of them blurted out this right at the end of the upgrade, but I did not observe any negative consequences yet.

  Cleanup    : wsproxy-1.12.0-2.xcpng8.2.x86_64                                                                                                                                       162/162 
Traceback (most recent call last):
  File "/bin/create-guest-templates", line 17, in <module>
    loader.insert_templates()
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 189, in insert_templates
    self._insert_template(i)
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 159, in _insert_template
    conn.request("PUT", "/import_metadata?" + params, tar)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 857, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe

stormi

@apz The script that deletes then recreates the guest templates when they are updated apparently failed on your host. Are there any missing templates in your template list?

apz

@stormi The affected host has only 2 templates, 2022 Windows and Suse 12.

stormi

@apz Try to re-run the script that failed:

/usr/bin/create-guest-templates-wrapper

apz

@stormi Result:

# /usr/bin/create-guest-templates-wrapper
Load /usr/share/xapi/vm-templates/windows-server-2012-64bit.json
Load /usr/share/xapi/vm-templates/sled-12-sp4-64bit.json
Load /usr/share/xapi/vm-templates/rhel-8.json
Load /usr/share/xapi/vm-templates/rhel-7.json
Load /usr/share/xapi/vm-templates/oel-8.json
Load /usr/share/xapi/vm-templates/sle-15-64bit.json
Load /usr/share/xapi/vm-templates/debian-9.json
Load /usr/share/xapi/vm-templates/windows-8-64bit.json
Load /usr/share/xapi/vm-templates/sles-12-sp5-64bit.json
Load /usr/share/xapi/vm-templates/base-sle-hvm.json
Load /usr/share/xapi/vm-templates/windows-10-64bit.json
Load /usr/share/xapi/vm-templates/oel-7.json
Load /usr/share/xapi/vm-templates/coreos.json
Load /usr/share/xapi/vm-templates/debian-11.json
Load /usr/share/xapi/vm-templates/windows-server-2012-r2-64bit.json
Load /usr/share/xapi/vm-templates/sles-12-sp3-64bit.json
Load /usr/share/xapi/vm-templates/windows-server-2016-64bit.json
Load /usr/share/xapi/vm-templates/gooroom-2.json
Load /usr/share/xapi/vm-templates/debian-10.json
Load /usr/share/xapi/vm-templates/windows-server-2022-64bit.json
Load /usr/share/xapi/vm-templates/other-install-media.json
Load /usr/share/xapi/vm-templates/base-sle-hvm-64bit.json
Load /usr/share/xapi/vm-templates/base-kylin-7.json
Load /usr/share/xapi/vm-templates/kylin-7.json
Load /usr/share/xapi/vm-templates/debian-8.json
Load /usr/share/xapi/vm-templates/sled-12-sp3-64bit.json
Load /usr/share/xapi/vm-templates/windows-server-2019-64bit.json
Load /usr/share/xapi/vm-templates/centos-7.json
Load /usr/share/xapi/vm-templates/base-windows-uefi.json
Load /usr/share/xapi/vm-templates/sles-12-sp4-64bit.json
Load /usr/share/xapi/vm-templates/sl-7.json
Load /usr/share/xapi/vm-templates/ubuntu-20.04.json
Load /usr/share/xapi/vm-templates/windows-10-32bit.json
Load /usr/share/xapi/vm-templates/ubuntu-16.04.json
Load /usr/share/xapi/vm-templates/rocky-8.json
Load /usr/share/xapi/vm-templates/windows-8-32bit.json
Load /usr/share/xapi/vm-templates/base-hvmlinux.json
Load /usr/share/xapi/vm-templates/almalinux-8.json
Load /usr/share/xapi/vm-templates/base-el-7.json
Load /usr/share/xapi/vm-templates/centos-8.json
Load /usr/share/xapi/vm-templates/base-windows.json
Load /usr/share/xapi/vm-templates/ubuntu-18.04.json
Load /usr/share/xapi/vm-templates/base-windows-8.json
Destroy 1c33af1c-e919-418c-ad45-85d7d6fb604a
Insert 1c33af1c-e919-418c-ad45-85d7d6fb604a
Traceback (most recent call last):
  File "/usr/bin/create-guest-templates", line 17, in <module>
    loader.insert_templates()
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 189, in insert_templates
    self._insert_template(i)
  File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 159, in _insert_template
    conn.request("PUT", "/import_metadata?" + params, tar)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
    self.send(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 857, in send
    self.sock.sendall(data)
  File "/usr/lib64/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 32] Broken pipe