XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XCP-ng 8.2.0 beta now available!

    Scheduled Pinned Locked Moved News
    42 Posts 15 Posters 11.5k Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stormiS Offline
      stormi Vates 🪐 XCP-ng Team
      last edited by stormi

      Hello to everyone.

      The Beta release of XCP-ng 8.2 is available right now šŸš€ .

      Stability status

      Our internal tests have not revealed any major bugs.

      Note : support for UEFI VMs is currently incomplete and will be completed during the beta phase via an update.

      Long Term Support (LTS)

      XCP-ng 8.2 will be our first LTS release.

      Help us make it rock-solid through your tests!

      What changed

      Here are the main changes since XCP-ng 8.1.

      Based on Citrix Hypervisor 8.2 (https://docs.citrix.com/en-us/citrix-hypervisor/whats-new.html).

      Base components:

      • Xen 4.13.1 + patches
      • Kernel 4.19 + patches

      Citrix announces:

      • Support for SLES 12 SP5 and Ubuntu 20.04 added
      • Support for Windows 7, Windows Server 2008 SP2 and Windows Server 2008 R2 SP1 removed. They may still work, but are not supported officially nor tested anymore by Citrix. Consider upgrading.
      • Maximum host RAM raised to 6 TB and maximum number of logical processors per host raised to 448 CPUs.
      • Support for installing TLS certificates on hosts (see https://docs.citrix.com/en-us/citrix-hypervisor/hosts-pools.html#install-a-tls-certificate-on-your-server)
      • TLS 1.2 protocol enforced for HTTPS traffic, and between XCP-ng components. Support for legacy SSL mode and TLS 1.0/1.1 protocols have been removed.

      Other changes:

      • Bug fixed for hosts configured with DHCP. xcp-networkd used not to send the hostname along with the DHCP request. We contributed a fix to the xcp-networkd package, that was included in Citrix Hypervisor 8.2 and thus in XCP-ng 8.2 (note: packages with that fix are also available for XCP-ng 8.0 and 8.1 in the xcp-ng-testing repository).
      • Backup restore fixed for UEFI hosts. This upstream bug that our community had found and that we reported to Citrix teams has been fixed by them.

      Changes specific to XCP-ng:

      • Support for Intel IceLake and CometLake CPUs.
      • Reimplementation of UEFI VM support : See dedicated section below for details.
      • New storage drivers: ZFS, GlusterFS, CephFS... and other changes. See dedicated section below for details.
      • The updated Xen brings support for Core Scheduling. We added support for changing the host configuration through the XAPI and contributed it upstream. See dedicated section below.
      • Guest tools ISO: Citrix doesn't provide a guest tools ISO anymore in Citrix Hypervisor. They replaced it with downloads from their internet website. We chose to retain the feature and still provide a guest tools ISO that you can mount to your VMs. Many thanks to the XAPI developers who have kept the related source code for us in the XAPI project, rather than delete it.
      • OpenFlow support in the SDN controller: support added out of the box for Xen Orchestra to leverage it. The related code is being contributed upstream to the XAPI project.
      • We replaced Citrix's gpumon package, not built by us, by a mock build of gpumon sources, without the proprietary nvidia developer kit. See dedicated section below
      • Alternate kernel updated to version 4.19.140.
      • Additional packages:
        • ZFS updated to 0.8.4.
        • glusterfs 8.1 added to the repositories
        • New additional driver package: r8125-module

      Reimplementation of UEFI VM support

      The component that handles UEFI variables for VMs, varstored, was redistributable but not open source. So we had to implement our own, named uefistored.

      It is not entirely ready yet, that's why XCP-ng 8.2 beta's UEFI VM support is incomplete. Systems like Alpine linux will work, but neither Windows nor Debian will install for now.

      This is being worked on and you should see updates very soon.

      Since the component was entirely rewritten, we will ask the community to give special attention to UEFI VM tests.

      New storage drivers: ZFS, GlusterFS, CephFS... and other changes

      We added three new experimental storage drivers: zfs, glusterfs and cephfs.

      We also decided to remove the sm-additional-drivers package (that only contained the xfs SR driver) and include all SR drivers by default, including experimental ones. However, we do not install all the dependencies on dom0 by default: xfsprogs, gluster-server, ceph-common, zfs... They need to be installed using yum for you to use the related SR drivers.

      zfs

      We already provided zfs packages in our repositories, but there was no dedicated SR driver. Users would use the file driver, which has a major drawback: if the zpool is not active, that driver may believe that the SR suddenly became empty, and forget all VDI metadata.

      So we developed a dedicated zfs SR driver that checks whether zfs is present before drawing such conclusions.

      • Required dependency: zfs
      • Documentation: https://xcp-ng.org/docs/storage.html#zfs

      glusterfs

      Use this driver to connect to an existing Gluster storage as a shared SR.

      • Required dependency: glusterfs-server
      • Documentation: https://xcp-ng.org/docs/storage.html#glusterfs

      cephfs

      Use this driver to connect to an existing Ceph storage through the CephFS storage interface.

      • Required dependency: ceph-common (not provided in XCP-ng repositories. Check the docs.)
      • Documentation: https://xcp-ng.org/docs/storage.html#cephfs

      xfs

      Use this driver to create and use an XFS SR.

      • Required dependency: xfsprogs
      • Documentation : to be written. There's nothing special about it, it works exactly like the ext driver, but uses xfs instead of ext4 as a filesystem.

      Core Scheduling

      The updated Xen in XCP-ng 8.2 brings support for Core Scheduling.

      As you probably know, Hyper Threading defeats all mitigations of CPU vulnerabilities related to side-channel attacks. That's why it was required to disable it as part of the mitigations. The reason is that with Hyper Threading enabled you can't protect a VM's vCPUs from attacks originating from other VMs that have workloads scheduled on the same physical core.

      With Core Scheduling, you now have another solution: you can choose to leave Hyper Threading enabled and ask the scheduler to always group vCPUs of a given VM together on the same physical core(s). This will remove the vulnerability to a class of attacks from other VMs, but will leave the VM processes vulnerables to attacks from malevolent processes from within itself. To be usedonly with entirely trusted workloads.

      How to enable it:

      • In a future release of Xen Orchestra, you will be able to enable it with a simple click in the host's advanced settings
      • Until then, you can enable it manually by adding sched-gran=core to Xen's command line parameters in grub.cfg and reboot. See Xen's documentation.

      gpumon's mock build (tests with nVIDIA GPUs required)

      gpumon is an opensource component that can't be built without the appropriate proprietary nvidia toolkit. It is supposed to be only used for nvidia vgpus, which we don't support (proprietary stuff too). We already tried to remove it from XCP-ng in 8.1, however you may remember than we discovered, after the release of XCP-ng 8.1, that the XAPI would not manage to start without it on systems that have nVIDIA GPUs.

      This time, we chose another approach: we built a mock version of gpumon (working, but giving fake statistics about the GPUs), so the binary is present on XCP-ng and the XAPI is happy enough with it to start in presence of an nVIDIA GPU.

      However, there still is a risk that the XAPI would try to use gpumon later, not only during the startup process, so we need users with nVIDIA GPUs to install XCP-ng 8.2 and test it in various situations.

      How to upgrade from previous releases

      Since XCP-ng 8.2.0 is a minor release, both upgrade methods are supported:

      • From the installation ISO
      • From command line using yum (from XCP-ng 8.0 or 8.1 only!)

      Refer to the upgrade howto.

      Downloads:

      • Standard ISO: http://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.0-beta.iso
      • Net-install ISO: http://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.0-beta-netinstall.iso
      • SHA256SUMS: http://mirrors.xcp-ng.org/isos/8.2/SHA256SUMS
      • Signatures of the sums: http://mirrors.xcp-ng.org/isos/8.2/SHA256SUMS.asc

      It's a good habit to check your downloaded ISO against the SHA256 sum and for better security also check the signature of those sums. Although our mirror redirector does try to detect file changes on mirrors, it's should always be envisioned that a mirror (or in the worst case our source mirror) gets hacked and managed to provide both fake ISOs and SHA256 sums. But they can't fake the signature. See https://xcp-ng.org/docs/mirrors.html#security.

      After the installation, stay updated

      Run yum update regularly. We'll fix bugs that are detected regularly until the final release. Subscribe to this thread (and make sure to activate mail notifications in the forum parameters if you want them): we'll announce every important update to the beta here.

      What to test

      Everything and anything! A community effort to list things to be tested has been started at https://github.com/xcp-ng/xcp/wiki/Test-XCP.

      In addition, we ask you to give special attention to:

      • UEFI VM support
      • The new storage drivers

      Report or ask anything as replies to this thread.

      Feedback is always useful, both about successes and failures.

      H 1 Reply Last reply Reply Quote 4
      • H Offline
        hoerup @stormi
        last edited by

        @stormi whats the opinion on using the new gluster & ceph drivers for a hyper converged solution? Ok / should be ok but not tested / not recommended ??

        stormiS J 2 Replies Last reply Reply Quote 0
        • stormiS Offline
          stormi Vates 🪐 XCP-ng Team @hoerup
          last edited by

          @hoerup Should be ok but not tested enough.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            This is not trivial if you don't have enough Ceph/Gluster knowledge and a little bit of toolstack knowledge. Also, it won't survive an upgrade.

            1 Reply Last reply Reply Quote 0
            • K Offline
              krakers
              last edited by

              @olivierlambert do you have similar timeline plans to 8.1 for 8.2? I mean ~2 months between beta and final release?

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                This will depend on what's going on, but that sound a reasonable timeframe (no promises though šŸ˜‰ )

                1 Reply Last reply Reply Quote 0
                • T Offline
                  technot
                  last edited by technot

                  Is there a Xcp-NG Center compatible version compiled yet ?

                  I had a little problems upgrading to the 8.2 beta.
                  The main tty showed a normal bootup untill a point where it showed a bunch of ACPI errors,
                  it worked for 10-15 min on starting previously disabled devices, and then just stopped.
                  I let it work another 10-15 min but all that came was this error, over and over again:

                  [  639.270804] EDAC sbridge: Seeking for: PCI ID 8086:3cf5
                  [  639.270804] EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
                  [  639.270804] EDAC sbridge: Couldn't find mci handler
                  [  639.270804] EDAC sbridge: Failed to register device with error -19.
                  
                  

                  I switched to tty2 and lunched the init script from /opt/xensources and managed to upgrade from there. Wasnt too easy as the screen kept getting garbled from debug info that I assume default to tty2 output.

                  After the upgrade and reboot, it boots. But it takes a long long time.
                  Here is a small snippit from the first boot dmesg:

                  [   29.618412] usb 1-1.1.1: Product: G19 Gaming Keyboard
                  [   29.623569] input: G19 Gaming Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.1/1-1.1.1:1.0/0003:046D:C228.0003/input/input7
                  [   30.290247] hid-generic 0003:046D:C228.0003: input,hidraw2: USB HID v1.10 Keyboard [G19 Gaming Keyboard] on usb-0000:00:1a.0-1.1.1/input0
                  [   30.293954] input: G19 Gaming Keyboard Consumer Control as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.1/1-1.1.1:1.1/0003:046D:C228.0004/input/input8
                  [   30.958193] hid-generic 0003:046D:C228.0004: input,hiddev97,hidraw3: USB HID v1.10 Device [G19 Gaming Keyboard] on usb-0000:00:1a.0-1.1.1/input1
                  [  129.058404] EXT4-fs (sdb1): mounting ext3 file system using the ext4 subsystem
                  [  129.063909] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
                  [  169.228203] e1000e 0000:00:19.0 0000:00:19.0 (uninitialized): registered PHC clock
                  [  169.356510] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) 8c:89:a5:a3:b6:f3
                  [  169.356512] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection
                  

                  And its not done and ready untill it tries to initialize samba SR:

                  [  453.242190] xenbr1: port 1(eth1) entered blocking state
                  [  453.242190] xenbr1: port 1(eth1) entered forwarding state
                  [  476.236967] FS-Cache: Loaded
                  [  476.254307] FS-Cache: Netfs 'cifs' registered for caching
                  [  476.254307] Key type cifs.spnego registered
                  [  476.254307] Key type cifs.idmap registered
                  
                  

                  All in all this is much better then 8.1 and 8.0 where I had similar problems with ACPI, but it never got any further, and just locked up, no tty2 or such.
                  I actually had to use ACPI=off and/or ACPI=strict, and either way, the whole experience after upgrading was super slow vms, but with no aparant error messages so I returned to 7.6.
                  Well see how 8.2 fares, but so far so good šŸ™‚
                  I have attached both the dmesg and xl-dmesg from the installer, incase you want to review what takes so long during boot.
                  xl-dmesg-log.txt
                  dmesg-log.txt

                  Thanks for all you hard work šŸ™‚

                  stormiS 1 Reply Last reply Reply Quote 0
                  • brezlordB Offline
                    brezlord
                    last edited by brezlord

                    I upgraded to 8.2.0-beta to fix this issue nfs-storage-latency. Bellow are some tests done, if you need any specific tests just let me know.

                    I’m running XO from sources xo-server 5.67.0 xo-web 5.71.0

                    XEN-HOST-01 Dell R720
                    XEN-HOST-02 Dell R710
                    TrueNAS NFS Storage over 10GbE

                    I upgraded from 8.1.0 patched to 8.2.0-beta via ISO install. Install went through with no troubles. The host booted with no problems. All networks and storage was present.

                    All VMs booted and came back up with no issues and guest tools running. VMs consist of CentOS 7 & 8, Debian 9 & 10, Windows 10Pro

                    Successfully live migrated VMs from 8.1.0 host to 8.2.0-beta with no issues.

                    Attempted VM migration from 8.2.0-beta to 8.1.0 host failed as expected.

                    Successfully migrated VMs between 8.2.0-beta hosts.

                    Could not change pool master from 8.2.0-beta to 8.1.0 host. Get Unknown error from peer.

                    Successfully changed pool master with both hosts on 8.2.0-beta.

                    Both host on 8.2.0-beta, shut down host-02 which was not the pool master and the shared storage shows partially connected and no stats available for storage. Reboot host-02 and shared storage is now healthy and stats are back. Disabled host-02 and shutdown and still had the same issue as above. The only was to get stats back and get shares storage back to healthy and stats working is to forget the storage attached to host-02 that was shut down.

                    XCP-ng Center v20.04.00.32 will not connect to 8.2.0-beta hosts.

                    Had to reinstall xscontainer package to have docker container visibility in XO.

                    stormiS 1 Reply Last reply Reply Quote 1
                    • stormiS Offline
                      stormi Vates 🪐 XCP-ng Team @technot
                      last edited by

                      @technot said in XCP-ng 8.2.0 beta now available!:

                      Is there a Xcp-NG Center compatible version compiled yet ?

                      I don't think there is one. It depends on the community. Previous releases were built by @borzel. I don't know if he will have the time to do it this time.

                      T borzelB 2 Replies Last reply Reply Quote 0
                      • D Offline
                        demanzke
                        last edited by

                        I just updated using the ISO-method because the yum method didn't boot successfully.
                        What would be the best way to migrate from a ZFS file-SR to ZFS-SR? I assume setting the SR type after creation isn't intended and I can't find that in the XAPI docs either.
                        Is the cleanest solution to create a new ZFS dataset, ZFS-SR on top and move the virtual drives?

                        1 Reply Last reply Reply Quote 0
                        • stormiS Offline
                          stormi Vates 🪐 XCP-ng Team
                          last edited by stormi

                          I would have been interested in debugging why the system didn't boot successfully after an update using yum. Can you give us more details?

                          To migrate the ZFS SR, the only clean way I am aware of is to create a new one. There may be a hacky way but it would probably involve modifying XAPI database objects dans metadata written in the SR itself so I wouldn't attempt it.

                          1 Reply Last reply Reply Quote 0
                          • stormiS Offline
                            stormi Vates 🪐 XCP-ng Team @brezlord
                            last edited by

                            @brezlord said in XCP-ng 8.2.0 beta now available!:

                            Could not change pool master from 8.2.0-beta to 8.1.0 host. Get Unknown error from peer.

                            When you get such error in XO, there's usually more information in XO's log page. However, it is expected that you can't move the master to a host running a lower release.

                            Both host on 8.2.0-beta, shut down host-02 which was not the pool master and the shared storage shows partially connected and no stats available for storage. Reboot host-02 and shared storage is now healthy and stats are back. Disabled host-02 and shutdown and still had the same issue as above. The only was to get stats back and get shares storage back to healthy and stats working is to forget the storage attached to host-02 that was shut down.

                            Ok, I don't think it's a regression but it may be interesting to create a GitHub issue with all the details.

                            Had to reinstall xscontainer package to have docker container visibility in XO.

                            That is expected, unless you upgrade using yum.

                            if you need any specific tests just let me know.

                            If your host has an nVIDIA GPU, we would be interested in knowing it because there used to be bugs specific to hosts with such a GPU, even when not used.

                            And be ready for testing UEFI VMs soon! I'll post here when the updated uefistored package is available.

                            T 1 Reply Last reply Reply Quote 1
                            • brezlordB Offline
                              brezlord
                              last edited by

                              Thanks for the info. So doing a ISO update is essentially like a fresh install but keeps settings.

                              I don't have any nVIDIA GPUs, sorry can't help there.

                              I'll boot the second host and get more info to raise an issue for the SR problem.

                              1 Reply Last reply Reply Quote 0
                              • brezlordB Offline
                                brezlord
                                last edited by

                                I have raised issue https://github.com/xcp-ng/xcp/issues/439 for the above SR problem.

                                brezlord created this issue in xcp-ng/xcp

                                open Shared storage & XCP-ng tools show partially connected when 1 of 2 hosts shutdown. #439

                                1 Reply Last reply Reply Quote 1
                                • T Offline
                                  technot @stormi
                                  last edited by

                                  And be ready for testing UEFI VMs soon! I'll post here when the updated uefistored package is available.

                                  Does the host have to be booted in UEFI mode for UEFI VMs to function?

                                  If it does, is there an easy way to make the change from legacy to UEFI?
                                  All I know, is the install docs clearly state you should never change mode when upgrading.
                                  Is the only way to export all the vms, and reinstall, then import the vms back?

                                  stormiS 1 Reply Last reply Reply Quote 0
                                  • stormiS Offline
                                    stormi Vates 🪐 XCP-ng Team @technot
                                    last edited by

                                    @technot said in XCP-ng 8.2.0 beta now available!:

                                    Does the host have to be booted in UEFI mode for UEFI VMs to function?

                                    No, it's not related to the host's mode.

                                    1 Reply Last reply Reply Quote 1
                                    • T Offline
                                      technot @stormi
                                      last edited by

                                      @stormi
                                      Is it only lacking to get built/compiled? If so I could do it sometime during the weekend. Or has the code not been updated to reflect any changes done with xcp-ng 8.2. If the latter, I am afraid I cant help.

                                      1 Reply Last reply Reply Quote 0
                                      • stormiS Offline
                                        stormi Vates 🪐 XCP-ng Team
                                        last edited by

                                        It's not ready yet so you can't help now. Our developer who wrote that piece of code (uefistored) is chasing the last known bugs and then I'll build an update and push it to our RPM repository for XCP-ng 8.2. I'll announce it when it's ready.

                                        T 1 Reply Last reply Reply Quote 1
                                        • T Offline
                                          technot
                                          last edited by

                                          I dont know if this is expected behaviour, after upgrading all the way from 7.6 to 8.2 beta. But on a windows guest I have an AMD gpu in full pci passthrough, the device didnt show up after boot. And instead there was another device showing fault.
                                          Intel 8237sb pci to usb universal host controller, was throwing an error 43.
                                          And under Display adapters, there was only the microsoft remote emulated and a "Microsoft Basic" devices showing.
                                          I reinstalled the AMD driver, and rebooted. And it came back to life.

                                          Also, I must say the vms feel snappier. Faster boots, and less delay/latency. But I have not meassured pre/post, so its only a feeling!:D

                                          1 Reply Last reply Reply Quote 0
                                          • T Offline
                                            technot @stormi
                                            last edited by

                                            @stormi
                                            Im sorry, it wasnt clear. What I meant is XCP-ng Center. If that code has been updated, or just a question of compiling it.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post