XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XCP-ng 8.2.1 (maintenance update) - final testing sprint

    Scheduled Pinned Locked Moved News
    40 Posts 8 Posters 14.3k Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stormiS Offline
      stormi Vates 🪐 XCP-ng Team
      last edited by stormi

      I'm creating this new thread that is the direct sequel to the previous 8.2.1 testing thread, to make it easier to start for anyone coming here from the blog to help.

      XCP-ng 8.2.1 is about one last week of community testing from release. We need your help to make it happen.

      What is XCP-ng 8.2.1?

      It's just an update to XCP-ng 8.2. It's bigger than previous security or bugfix updates we released, but it's the same principle: a maintenance update. Once released, people running XCP-ng 8.2 will just update their hosts as usual.

      Then why is it numbered 8.2.1?

      Two reasons:

      • Among other changes, we included those from Citrix Hypervisor CU1, and the official version number changed to 8.2.1 there. You will see 8.2.1 instead of 8.2.0 as the minor version number in xsconsole, for example. But in the end it will just be an up to date 8.2 LTS.
      • At the same time as we will release updates for XCP-ng 8.2, we will release new installation ISOs that contain all the updated packages up to now. And also use it as an opportunity to fix a few bugs in the installer (anyone got stuck on the blue installation screen after choosing software RAID with 8.2 installation ISOs? 😅). So in a way, it's also a new release that deserves its own patch version number.

      Is it an optional update, like Citrix Hypervisor 8.2 CU1 is?

      No. Once released, it will be just another update to XCP-ng 8.2.

      Are there new fancy features?

      Well, it's a maintenance release so don't expect too much.

      However:

      • Secure boot for VMs is now supported (full documentation here. Read it, really: there's a pool configuration step necessary if you want to enable secure boot). Some of you already tested this feature months ago. Now it's time for it to reach everyone. There's a but: XCP-ng's guest tools are not signed with a recent enough certificate and Microsoft's user support is so bad that we haven't been able to get a new signing certificate in months (there are issues with their own website that they have trouble finding a workaround for)! So for now the basic rule is: if you want to enable Secure Boot on a Windows VM, use the signed guest tools from Citrix.
      • The installation of Microsoft's infamous KB4535680 update is now fixed.
      • A few components like qemu are updated to pave the way to future vTPM support. I said future. It's not available yet.
      • Rocket Lake processors are now supported.

      Other notable changes

      • Guest template for Windows Server 2022 added.
      • Log rotation. Log files should now be automatically rotated if they reach a size of 100M, without waiting for the daily log rotation. This will better handle the situations where a single log file grows up very fast to the point of filling the log partition.
      • Updated default drivers on the system:
        • avago-mpt3sas-33.100.00.01
        • intel-ice-1.6.4 (new vendor driver RPM. We were previously using the built-in driver from kernel 4.19)
        • intel-ixgbe-5.9.4
        • qlogic-fastlinq-8.42.10.0
        • qlogic-netxtreme2-7.14.76
        • r8125-module-9.003.05 added for some Realtek NICs (might be removed from the final release if we can't get enough feedback)
        • igc-module-4.20.17 added for Intel i225 NICs (might be removed from the final release if we can't get enough feedback or can't fix the remaining issues)
      • The default console menu, xsconsole, was updated and includes an improvement that we had contributed upstream: when the XAPI service is unreachable on the host, xsconsole will try to display a useful error message, rather than displaying a misleading message saying that no network was configured.
      • A bug that we discovered and reported upstream regarding the handling of web pages over HTTPS on the host when HTTP support was forbidden has been fixed, so I could finally enforce HTTPS for the host's web page. Any request to get the web page on port 80 will reply with a 403 error.
      • @r1 has updated our alternate kernel to a much more recent maintenance release of kernel.org's 4.19 branch.
      • samba and openssl were updated, which fixes various CVEs. The update to the samba packages pulled several new dependencies such as gnutls, nettle, python-tdb, ...
      • xenserver-transfer-vm was removed.

      How to test

      Either install or upgrade a host using the test installation ISOs found at https://mirrors.xcp-ng.org/tmp/

      Or update an existing XCP-ng 8.2 host:

      • create a file named xcp-ng-staging.repo.
        [xcp-ng-staging]
        name=XCP-ng Staging Repository
        baseurl=http://mirrors.xcp-ng.org/8/8.2/staging/x86_64/ http://updates.xcp-ng.org/8/8.2/staging/x86_64/
        enabled=0
        gpgcheck=1
        repo_gpgcheck=1
        gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-xcpng
        
      • Update: yum update --enablerepo=xcp-ng-staging
      • Usual instructions from https://xcp-ng.org/docs/updates.html still apply

      What to test

      As usual, anything that you need XCP-ng for.

      We also would like you to give special focus to the following items:

      • UEFI VMs, without Secure Boot
      • UEFI VMs, with Secure Boot (check the docs. There's a manual command to run once on the pool, to download and install the certificates from Microsoft.)
      • On Windows installed from a not too recent image (otherwise the test is impossible), installation of update KB4535680, which updates the list of revocated certificates for Secure Boot. Should work without Secure Boot on, but we had reports of failures in this situation so I'm interested in finding a way to reproduce. Should also work with Secure Boot on.
      • Log rotation if you have a way to trigger very verbose logs.
      • The installer (installation, upgrade, backup restore...).
      • Active Directory connectivity, if you know how to make it work
      • The alternate kernel. Having it tested on a large variety of hardware would be good.

      How to come back to the main update track after the tests?

      You will have nothing to do to turn a host with this test version of X. Just continue updating your hosts normally in the future.

      Ready... Test!

      And of course, ask anything.

      J gskgerG 2 Replies Last reply Reply Quote 0
      • stormiS stormi referenced this topic on
      • J Offline
        JeffBerntsen Top contributor @stormi
        last edited by

        @stormi

        Testing UEFI VMs. So far working fine without secure boot. Having a problem with secure boot under Alpine Linux. They don't use MS certificates and a shim as a lot of distros do but instead have you generate a set of keys for your installation and then enroll them. It looks like this is a problem with enrolling the generated keys in the TianoCore boot firmware. I'll try with a different distro and see if it's any better with something different.

        stormiS 1 Reply Last reply Reply Quote 0
        • stormiS Offline
          stormi Vates 🪐 XCP-ng Team @JeffBerntsen
          last edited by

          @JeffBerntsen To enrol keys for a specific VM, there's doc.

          J 1 Reply Last reply Reply Quote 0
          • J Offline
            JeffBerntsen Top contributor @stormi
            last edited by

            @stormi That worked to get the auth files generated using Alpine's instructions enrolled as far as I can tell but switching the VM to secure boot after that still fails, dropping me into a UEFI shell. Alpine 3.15 is the first version with secure boot support and it's possible there are still some glitches there.

            Instead of that, I'm now trying to set up a secure boot with a fresh install of OpenSUSE leap 15.3 which I know does support secure boot and will see if that works out.

            stormiS 1 Reply Last reply Reply Quote 0
            • stormiS Offline
              stormi Vates 🪐 XCP-ng Team @JeffBerntsen
              last edited by stormi

              @JeffBerntsen Here we have a test that generates keys and signs the boot binaries with them, if you want to check how we did. Works on many linux distros including alpine (3.12.0): https://github.com/xcp-ng/xcp-ng-tests/blob/master/tests/uefistored/test_secure_boot.py#L142

              Tumbleweed 15.3 should work out of the box with the defaults certs installed by secureboot-certs install (that include the latest dbx - revocation list - from Microsoft).

              J 1 Reply Last reply Reply Quote 0
              • J Offline
                JeffBerntsen Top contributor @stormi
                last edited by JeffBerntsen

                @stormi Thanks, I'll give the test script a try on my test Alpine installation and see if it works for me.

                My OpenSUSE Leap 15.3 installation works just fine via secure boot with one warning/error message at boot. It's complaining that it can't generate a temporary hibernation key because of a missing EFI_RNG_PROTOCOL. Except for that, it works great under secure boot. If not being able to have hibernation support in the VM's operating system is the only issue, that's definitely minor and something I don't use and won't miss.

                EDIT: I'm also going to try a fresh installation of Alpine into a VM set for secure boot and see how that works out. My test was trying to convert an existing VM that was successfully booting under UEFI without secure boot enabled.

                EDIT 2: I've managed to get Alpine working as well. It appears that their Wiki entry on setting up secure boot isn't quite right yet. They have a utility which generates keys and creates a signed unified boot image. My best guess is that there is some problem with the signature on the boot image. I was able to get things working by enrolling the generated auth files for the VM uuid on the host system then booting the VM with secure boot disabled and using the sbsign utility to sign the boot image with the generated db key and certificate. It adds a second signature to the boot image which appears to be identical to the first one. Switching to secure boot mode and rebooting works on the re-signed boot image.

                1 Reply Last reply Reply Quote 1
                • theAeonT Offline
                  theAeon
                  last edited by

                  Bumping my lab to staging right now-if you don't hear back, assume everything works fine.

                  1 Reply Last reply Reply Quote 1
                  • stormiS Offline
                    stormi Vates 🪐 XCP-ng Team
                    last edited by stormi

                    It doesn't look like my blog post brought a lot of new testers.

                    There's still time (a few days) to lend a hand for this 8.2.1 release and test it. I don't think the alternate kernel got a lot of attention outside Vates. Nor AD connectivity (but maybe no one uses this, or they connect their XO instead which might be better).

                    I'm currently building new ISOs (test6) that will probably be the final ones. The only difference with test5 is that I removed the igc and r8125 drivers due to issues with the first one and lack of feedback on the second one. We'll continue working on improved hardware support after the release.

                    If you installed XCP-ng 8.2.1 using the test5 installation ISO, you need to follow these steps (other testers, just dismiss):

                    yum downgrade vendor-drivers
                    yum update vendor-drivers # should do nothing. Just in case.
                    yum remove igc-module r8125-module # unless you need them
                    
                    J 1 Reply Last reply Reply Quote 0
                    • gskgerG Offline
                      gskger Top contributor @stormi
                      last edited by

                      @stormi Not much of a help this time, cause my job keeps me way too busy. Anyway, I upgraded my two host playlab the day you released the latest version (via the yum update route with staging repo). Everything updated fine and works as expected since then, but I cannot contribute to the specific test items you asked for.

                      stormiS 1 Reply Last reply Reply Quote 1
                      • stormiS Offline
                        stormi Vates 🪐 XCP-ng Team @gskger
                        last edited by

                        @gskger If you can find time for it, you can just update to the latest state of the staging branch with yum update --enablerepo=xcp-ng-staging. Else no problem.

                        gskgerG 1 Reply Last reply Reply Quote 0
                        • stormiS Offline
                          stormi Vates 🪐 XCP-ng Team
                          last edited by

                          New installation ISOs (test6) are available at https://updates.xcp-ng.org/tmp/. The netinstall repository was also updated.

                          The only changes since the last ones are the removal of igc and r8125 drivers that I had attempted to add in test5.

                          These should be the final ones, so it's always good if some of you can test them one last time before the release.

                          rus2lanR 1 Reply Last reply Reply Quote 0
                          • J Offline
                            JeffBerntsen Top contributor @stormi
                            last edited by

                            @stormi Some quick testing of the alternate kernel on my test systems seems to be working fine with the not-unexpected issue that the XOSTOR test does not come up and run on it.

                            1 Reply Last reply Reply Quote 1
                            • gskgerG Offline
                              gskger Top contributor @stormi
                              last edited by

                              @stormi That was an easy 2.8k update on both hosts with no problem. VMs continue to run without any issues so far.

                              1 Reply Last reply Reply Quote 1
                              • rus2lanR Offline
                                rus2lan @stormi
                                last edited by

                                @stormi https://www.asus.com/Motherboards-Components/Motherboards/TUF-Gaming/TUF-GAMING-Z690-PLUS-WIFI-D4/HelpDesk_QVL_CPU/ for this motherboard igc drivers work only for xcp, i have trouble in VM with VLANs: DHCP work, but no ping to gateway...

                                stormiS 1 Reply Last reply Reply Quote 0
                                • stormiS Offline
                                  stormi Vates 🪐 XCP-ng Team @rus2lan
                                  last edited by

                                  @rus2lan The igc driver we backported from the 4.20 kernel doesn't appear to be working well indeed. That's why I did not include it in the final release of XCP-ng 8.2.1 ISOs.

                                  1 Reply Last reply Reply Quote 1
                                  • stormiS Offline
                                    stormi Vates 🪐 XCP-ng Team
                                    last edited by

                                    XCP-ng 8.2.1 is now released. A huge thanks to everyone who tested and gave feedback to us.

                                    https://xcp-ng.org/blog/2022/02/28/xcp-ng-8-2-1-update/

                                    1 Reply Last reply Reply Quote 3
                                    • apzA Offline
                                      apz
                                      last edited by

                                      I upgraded 3 of my homelab hosts, all were up-to-date 8.2's before this update. One of them blurted out this right at the end of the upgrade, but I did not observe any negative consequences yet.

                                        Cleanup    : wsproxy-1.12.0-2.xcpng8.2.x86_64                                                                                                                                       162/162 
                                      Traceback (most recent call last):
                                        File "/bin/create-guest-templates", line 17, in <module>
                                          loader.insert_templates()
                                        File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 189, in insert_templates
                                          self._insert_template(i)
                                        File "/usr/lib/python2.7/site-packages/guesttemplates/loader.py", line 159, in _insert_template
                                          conn.request("PUT", "/import_metadata?" + params, tar)
                                        File "/usr/lib64/python2.7/httplib.py", line 1041, in request
                                          self._send_request(method, url, body, headers)
                                        File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
                                          self.endheaders(body)
                                        File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
                                          self._send_output(message_body)
                                        File "/usr/lib64/python2.7/httplib.py", line 885, in _send_output
                                          self.send(message_body)
                                        File "/usr/lib64/python2.7/httplib.py", line 857, in send
                                          self.sock.sendall(data)
                                        File "/usr/lib64/python2.7/socket.py", line 224, in meth
                                          return getattr(self._sock,name)(*args)
                                      socket.error: [Errno 32] Broken pipe
                                      
                                      stormiS 1 Reply Last reply Reply Quote 0
                                      • stormiS Offline
                                        stormi Vates 🪐 XCP-ng Team @apz
                                        last edited by

                                        @apz The script that deletes then recreates the guest templates when they are updated apparently failed on your host. Are there any missing templates in your template list?

                                        apzA 1 Reply Last reply Reply Quote 1
                                        • apzA Offline
                                          apz @stormi
                                          last edited by

                                          @stormi The affected host has only 2 templates, 2022 Windows and Suse 12.

                                          stormiS 1 Reply Last reply Reply Quote 0
                                          • stormiS Offline
                                            stormi Vates 🪐 XCP-ng Team @apz
                                            last edited by

                                            @apz Try to re-run the script that failed:

                                            /usr/bin/create-guest-templates-wrapper
                                            
                                            apzA 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post