XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XCP-ng 8.2 updates announcements and testing

    Scheduled Pinned Locked Moved News
    708 Posts 67 Posters 1.1m Views 86 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DanpD Offline
      Danp Pro Support Team
      last edited by

      Looks like these updates were released earlier today -- https://xcp-ng.org/blog/2023/05/26/may-2023-maintenance-update/

      1 Reply Last reply Reply Quote 1
      • gduperreyG Offline
        gduperrey Vates 🪐 XCP-ng Team
        last edited by stormi

        New update candidates for you to test!

        Shortly after we released the previous batch of non-urgent updates, XenServer released several updates for Citrix Hypervisor 8.2 CU1. We prepared new update candidates based on these, as well as a specific update of xcp-ng-xapi-plugins.

        There's no date for their release yet, but they're ready for your tests and feedback already.

        • xcp-ng-xapi-plugins: the updater plugin, used by Xen Orchestra to apply updates, can now also install new packages (this will be used to deploy XOSTOR from Xen Orchestra).
        • kernel: as explained in the hotfix from XenServer XS82ECU1028 "ACPI processor-related data is being reported incorrectly to the hypervisor, affecting Intel - Xeon 84xx/64xx/54xx/44xx/34xx - Sapphire Rapids and possibly other models."
        • grub: bugfix
        • lldpad:
          • The FCoE service can have a memory leak that could use up dom0 memory
          • A resource leak in the FCoE service can crash the service
          • When trying to create an LACP bond using Cisco Nexus switches, host could have intermittent connection problems
          • XenServer hotfix: XS82ECU1032
        • xen: Correct a flaw for VMx under Red Hat Enterprise 7 (and derivatives) with a large number of CPUs, that can cause migration failures when trying to migrate to AMD hosts.
          • XenServer hotfix: XS82ECU1034

        Test on XCP-ng 8.2

        From an up to date host:

        yum clean metadata --enablerepo=xcp-ng-testing
        yum update --enablerepo=xcp-ng-testing xen-* grub* lldpad kernel xcp-ng-xapi-plugins
        reboot
        

        The usual update rules apply: pool coordinator first, etc.

        Versions

        • kernel-4.19.19-7.0.16.1.xcpng8.2
        • grub-2.02-3.2.0.xcpng8.2
        • lldpad-1.0.1-10.xcpng8.2
        • xen-*4.13.5-9.32.1.xcpng8.2
        • xcp-ng-xapi-plugins-1.8.0-1.xcpng8.2

        What to test

        Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

        Test window before official release of the updates

        None defined, but early feedback is always better than late feedback, which is in turn better than no feedback 🙂

        A J 2 Replies Last reply Reply Quote 1
        • A Offline
          Andrew Top contributor @gduperrey
          last edited by

          @gduperrey I've been running the update on all my active 8.2 machines. I did not run into any of the bugs before. After the update everything is still normal.

          1 Reply Last reply Reply Quote 2
          • J Offline
            JeffBerntsen Top contributor @gduperrey
            last edited by

            @gduperrey I'm running the updates on both lab and production machines and all seems well so far.

            1 Reply Last reply Reply Quote 2
            • N Offline
              NielsH
              last edited by

              Yesterday Zenbleed (https://news.ycombinator.com/item?id=36848680) was announced. It mentions there that AMD may have already released microcode patches to fix this.
              Is this perhaps the mysterious microcode update form may 16th?

              AMD microcode (linux-firmware) and Intel microcode (microcode_ctl). AMD and Intel did not detail what they fix, but everyone is supposed to update. This is the frustrating situation with binary blobs in firmware.

              Or can we expect another update to resolve the zenbleed vulnerability in the coming days?

              1 Reply Last reply Reply Quote 0
              • gduperreyG Offline
                gduperrey Vates 🪐 XCP-ng Team
                last edited by

                We began to work on the patch yesterday evening. We will publish it for testers later today, and if everything is fine, for everyone after two days (and success in our tests, of course).

                1 Reply Last reply Reply Quote 0
                • gduperreyG Offline
                  gduperrey Vates 🪐 XCP-ng Team
                  last edited by

                  New Security Update Candidates (Xen and AMD CPUs)

                  Xen is being updated to mitigate hardware vulnerabilities in AMD CPUs.

                  • Upstream (Xen project) advisory: XSA-433
                    • Citrix Hypervisor Security Update for CVE-2023-20593

                  This issue affects systems running AMD Zen 2 CPUs. Under specific microarchitectural circumstances, it may allow an attacker to potentially access sensitive information.

                  Components are also updated to add bugfixes and enhancements:

                  • Xen:
                    • Now, MPX feature is disabled by default. Cross-pool migration and upgrade will be simplified as VMs can migrate more easily from pools with Intel SkyLake, CascadeLake, or CooperLake hardware to pools with later Intel hardware (such as IceLake).
                      A reboot is necessary after updating to benefit from this feature.
                    • Improvements to latency with a limit on the scheduler loadbalancing. This improves performance on large systems with high CPU utilization.

                  Test on XCP-ng 8.2

                  From an up to date host:

                  yum clean metadata --enablerepo=xcp-ng-testing
                  yum update "xen-*" linux-firmware --enablerepo=xcp-ng-testing
                  reboot
                  

                  Versions:

                  • xen-*: 4.13.5-9.34.1.xcpng8.2
                  • linux-firmware: 20190314-8.1.xcpng8.2

                  What to test

                  Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

                  Test window before official release of the updates

                  ~2 days.

                  A 1 Reply Last reply Reply Quote 2
                  • A Offline
                    Andrew Top contributor @gduperrey
                    last edited by

                    @gduperrey Updated multiple servers and seems to be fine. But, none are Zen2 systems.

                    1 Reply Last reply Reply Quote 2
                    • gduperreyG Offline
                      gduperrey Vates 🪐 XCP-ng Team
                      last edited by

                      Update published. Thanks for the tests!

                      https://xcp-ng.org/blog/2023/07/27/july-2023-security-update-zenbleed/

                      1 Reply Last reply Reply Quote 1
                      • T Offline
                        TodorPetkov
                        last edited by

                        Hello,

                        I saw there is new announcement on Xenbits regarding Zenbleed (https://xenbits.xen.org/xsa/advisory-433.html) - will there be new patch for XCP?

                        Thanks in advance.

                        bleaderB 1 Reply Last reply Reply Quote 0
                        • bleaderB Offline
                          bleader Vates 🪐 XCP-ng Team @TodorPetkov
                          last edited by

                          @TodorPetkov Yes, for now we do not know when this update will be released on XenServer side yet, but it will be published on XCP-ng side too.

                          What was released for now is suffering from the same issue as described in your link.

                          If I'm not mistaken:

                          • the linux-firmware update fixes the issues with zenbleed
                          • the kernel patch is working around the case where the updated firmware is not used by disabling features via the control register, and there were too much disabled in the previous patch.
                          • if you're using the updated firmware, this workaround will not be used, and therefore the updated patch is not critical.

                          You can check you're running the right microcode version via:

                          journalctl -k --grep=microcode
                          

                          Without the -k you should be able to see previous boots and ensure the patch_level= has changed. I'm unsure which version to expect there as we do not have zen2 at hand for testing this.

                          We will indeed provide an update later, likely not in a dedicated update, but with other fixes.

                          I hope that answers properly your question!

                          1 Reply Last reply Reply Quote 1
                          • gduperreyG Offline
                            gduperrey Vates 🪐 XCP-ng Team
                            last edited by

                            New Security Update Candidates (Xen)

                            Xen is being updated to correct a flaw in the latest patch (XSA-433) for Zenbleed and AMD CPUs.

                            • Upstream (Xen project) advisory: XSA-433

                            The patch provided with earlier versions was buggy by unintentionally disabling more bits than expected in the control register due to bad integer variable truncation.

                            Test on XCP-ng 8.2

                            From an up to date host:

                            yum clean metadata --enablerepo=xcp-ng-testing
                            yum update "xen-*" --enablerepo=xcp-ng-testing
                            reboot
                            

                            Version:

                            • xen-*: 4.13.5-9.35.1.xcpng8.2

                            If you didn't already applied the previous updates, we invite you to also update linux-firmware.

                            yum update linux-firmware
                            reboot
                            

                            Version:

                            • linux-firmware: 20190314-8.1.xcpng8.2

                            One reboot for the two updates is enough.

                            What to test

                            Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

                            Test window before official release of the updates

                            ~1 days. We'll release before the WE if our internal tests are fine.

                            1 Reply Last reply Reply Quote 1
                            • gduperreyG Offline
                              gduperrey Vates 🪐 XCP-ng Team
                              last edited by

                              Update published. Thanks for the tests!

                              https://xcp-ng.org/blog/2023/08/04/erratum-july-2023-security-update-zenbleed/

                              N 1 Reply Last reply Reply Quote 2
                              • N Offline
                                NielsH @gduperrey
                                last edited by

                                @gduperrey Thanks!

                                I see that shortly after this some more security related patched have become available:
                                XSA-432: https://www.openwall.com/lists/oss-security/2023/08/08/3
                                XSA-434: https://www.openwall.com/lists/oss-security/2023/08/08/4
                                XSA-435: https://www.openwall.com/lists/oss-security/2023/08/08/5

                                Some of this also seems pretty serious. We were currently busy running patches on our systems. This takes us about 1-2 weeks to complete. Do you think these 3 patches will also become available for XCP-ng? In that case we'd rather wait for those to become available so we don't have to spend another 1-2 weeks immediately after completing the current set of patches.

                                Thank you!

                                1 Reply Last reply Reply Quote 0
                                • gduperreyG Offline
                                  gduperrey Vates 🪐 XCP-ng Team
                                  last edited by

                                  Hello,

                                  Yes, these patches will become available in XCP-ng. We're working on it to release as soon as possible. We'd like to release them this week, so we do everything we can for that.

                                  There will be a post here for the tests and for the final release.

                                  1 Reply Last reply Reply Quote 2
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    Hello @NielsH I don't want to sound moralistic, but if you are using XCP-ng in production without any subscription, and being worried about patches coming fast enough for your production system, you should really think about getting support for it 🙂 That's exactly what the subscription money is made for! (well, in part but absolutely used for that).

                                    I mean, you are free to not doing it, and even if we do our best to treat everyone fairly (paying or not) for our patches, we won't be against more support so the project can continue to grow 🙂

                                    N 1 Reply Last reply Reply Quote 0
                                    • N Offline
                                      NielsH @olivierlambert
                                      last edited by

                                      @olivierlambert said in Updates announcements and testing:

                                      Hello @NielsH I don't want to sound moralistic, but if you are using XCP-ng in production without any subscription, and being worried about patches coming fast enough for your production system, you should really think about getting support for it 🙂 That's exactly what the subscription money is made for! (well, in part but absolutely used for that).

                                      I mean, you are free to not doing it, and even if we do our best to treat everyone fairly (paying or not) for our patches, we won't be against more support so the project can continue to grow 🙂

                                      Hey,

                                      No worries, I understand your point.
                                      For us specifically, we use local storage and the time it takes to migrate all VMs to a different host, reboot, next host, etc... is about 1-1.5 weeks during working hours. This is why, if another batch of updates is coming soon, we would rather wait and do everything at once instead of update twice in a row. I'm aware of the Rolling Pool Upgrade feature but it is not compatible with local storage.

                                      Regarding XCP-ng Pro support, currently the pricing is a bit too steep for us to be feasable due to the amount of hosts we have. We do however pay for XOA Premium which is more affordable and we do find it valuable to support the developers.

                                      I believe I did see some of your posts around regarding a possible all-in pricing plan which may be interesting for us, so once more details regarding that are available it is something for us to look into 🙂

                                      C 1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        No worries, and indeed, new pricing might be a good fit for you 🙂

                                        1 Reply Last reply Reply Quote 0
                                        • gduperreyG Offline
                                          gduperrey Vates 🪐 XCP-ng Team
                                          last edited by gduperrey

                                          New Security Update Candidates (kernel, Xen, linux-firmware, microcode_ctl, XAPI...)

                                          Xen is being updated to mitigate some vulnerabilities:

                                          • XSA-432: CVE-2023-34319. Under Linux, a buffer overrun in netback can be triggered due to unusual packets. This behavior was due to the fix of the XSA-423 which didn't account an extreme case of an entire packet being split into as many pieces as permitted by the protocol and still being smaller than the area that's dealt with to keep all headers together. It is possible to crash a host from a vm, with malicious and privileged code.

                                          • XSA-434: CVE-2023-20569. Researchers from ETH Zurich have extended their prior research (XSA-422, Branch Type Confusion, a.k.a Retbleed) and have discovered INCEPTION, also known as RAS (Return Address Stack) Poisoning, and Speculative Return Stack Overflow. An attacker might be able to infer the contents of memory belonging to other guests.

                                          • XSA-435: CVE-2022-40982. A security issue in certain Intel CPUs may allow an attacker to infer data from different contexts on the same core.

                                          Components are also updated to add bugfixes and enhancements:

                                          • guest-templates-json: Added Debian 12 Bookworm

                                          • XAPI:

                                            • Several hotfixes and improvements from XS82ECU1033
                                            • From XS82ECU1045 Significant performance improvements on a set of CPU features for servers with Cascade Lake or later Intel CPUs.
                                          • microcode_ctl: Update to IPU 2023.3

                                          • linux-firmware: Expose additional features for Intel CPUs, especially for Cascade Lake or later Intel CPUs. Updated to latest AMD firmware for processor family 19h.

                                          • Xen: Expose MSR_ARCH_CAPS to guests on all Intel hardware by default.

                                          • blktap, nbd: An update of the packages for Xostor.

                                          Test on XCP-ng 8.2

                                          From an up to date host:

                                          yum clean metadata --enablerepo=xcp-ng-testing
                                          yum update "xen-*" microcode_ctl linux-firmware kernel forkexecd gpumon message-switch "ocaml-*" rrd2csv rrdd-plugins sm-cli squeezed varstored-guard vhd-tool wsproxy "xapi-*" xcp-networkd xcp-rrdd "xenopsd*" xs-opam-repo "guest-templates-*" blktap xcp-ng-linstor nbd tzdata grub* lldpad xcp-ng-xapi-plugins --enablerepo=xcp-ng-testing
                                          reboot
                                          

                                          Version:

                                          • forkexecd: 1.18.3-2.1.xcpng8.2
                                          • gpumon: 0.18.0-10.1.xcpng8.2
                                          • kernel: 4.19.19-7.0.17.1.xcpng8.2
                                          • linux-firmware: 20190314-9.1.xcpng8.2
                                          • message-switch: 1.23.2-9.1.xcpng8.2
                                          • microcode_ctl: 2.1-26.xs26.1.xcpng8.2
                                          • ocaml-rrd-transport: 1.16.1-7.1.xcpng8.2
                                          • ocaml-rrdd-plugin: 1.9.1-7.1.xcpng8.2
                                          • ocaml-tapctl: 1.5.1-7.1.xcpng8.2
                                          • ocaml-xcp-idl: 1.96.5-1.1.xcpng8.2
                                          • ocaml-xen-api-client: 1.9.0-10.1.xcpng8.2
                                          • ocaml-xen-api-libs-transitional: 2.25.5-4.1.xcpng8.2
                                          • rrd2csv: 1.2.6-7.1.xcpng8.2
                                          • rrdd-plugins: 1.10.9-4.1.xcpng8.2
                                          • sm-cli: 0.23.0-53.1.xcpng8.2
                                          • squeezed-0.27.0-10.1.xcpng8.2
                                          • varstored-guard: 0.6.2-7.xcpng8.2
                                          • vhd-tool: 0.43.0-10.1.xcpng8.2
                                          • wsproxy: 1.12.0-11.xcpng8.2
                                          • xapi: 1.249.32-1.1.xcpng8.2
                                          • xapi-nbd: 1.11.0-9.1.xcpng8.2
                                          • xapi-storage: 11.19.0_sxm2-9.xcpng8.2
                                          • xapi-storage-script: 0.34.1-8.1.xcpng8.2
                                          • xcp-networkd: 0.56.2-7.xcpng8.2
                                          • xcp-rrdd: 1.33.2-6.1.xcpng8.2
                                          • xen: 4.13.5-9.36.1.xcpng8.2
                                          • xenopsd: 0.150.17-1.1.xcpng8.2
                                          • xs-opam-repo: 6.35.11-1.xcpng8.2
                                          • guest-templates-json: 1.9.6-1.3.xcpng8.2
                                          • blktap-3.37.4-1.0.2.xcpng8.2
                                          • tzdata-2022a-1.el7
                                          • xcp-ng-linstor-1.1-3.xcpng8.2
                                          • nbd-3.24-1.xcpng8.2
                                          • grub-2.02-3.2.0.xcpng8.2

                                          What to test

                                          Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

                                          Test window before official release of the updates

                                          ~2 days.

                                          gskgerG 1 Reply Last reply Reply Quote 2
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by

                                            42 packages and 147M worth of updates 😄

                                            Installed here and worked on my HPE EPYC host 🙂

                                            1 Reply Last reply Reply Quote 2
                                            • First post
                                              Last post