XCP-ng 8.2.0 beta now available!

stormi

Hello to everyone.

The Beta release of XCP-ng 8.2 is available right now .

Stability status

Our internal tests have not revealed any major bugs.

Note : support for UEFI VMs is currently incomplete and will be completed during the beta phase via an update.

Long Term Support (LTS)

XCP-ng 8.2 will be our first LTS release.

Help us make it rock-solid through your tests!

What changed

Here are the main changes since XCP-ng 8.1.

Based on Citrix Hypervisor 8.2 (https://docs.citrix.com/en-us/citrix-hypervisor/whats-new.html).

Base components:

Xen 4.13.1 + patches
Kernel 4.19 + patches

Citrix announces:

Support for SLES 12 SP5 and Ubuntu 20.04 added
Support for Windows 7, Windows Server 2008 SP2 and Windows Server 2008 R2 SP1 removed. They may still work, but are not supported officially nor tested anymore by Citrix. Consider upgrading.
Maximum host RAM raised to 6 TB and maximum number of logical processors per host raised to 448 CPUs.
Support for installing TLS certificates on hosts (see https://docs.citrix.com/en-us/citrix-hypervisor/hosts-pools.html#install-a-tls-certificate-on-your-server)
TLS 1.2 protocol enforced for HTTPS traffic, and between XCP-ng components. Support for legacy SSL mode and TLS 1.0/1.1 protocols have been removed.

Other changes:

Bug fixed for hosts configured with DHCP. xcp-networkd used not to send the hostname along with the DHCP request. We contributed a fix to the xcp-networkd package, that was included in Citrix Hypervisor 8.2 and thus in XCP-ng 8.2 (note: packages with that fix are also available for XCP-ng 8.0 and 8.1 in the xcp-ng-testing repository).
Backup restore fixed for UEFI hosts. This upstream bug that our community had found and that we reported to Citrix teams has been fixed by them.

Changes specific to XCP-ng:

Support for Intel IceLake and CometLake CPUs.
Reimplementation of UEFI VM support : See dedicated section below for details.
New storage drivers: ZFS, GlusterFS, CephFS... and other changes. See dedicated section below for details.
The updated Xen brings support for Core Scheduling. We added support for changing the host configuration through the XAPI and contributed it upstream. See dedicated section below.
Guest tools ISO: Citrix doesn't provide a guest tools ISO anymore in Citrix Hypervisor. They replaced it with downloads from their internet website. We chose to retain the feature and still provide a guest tools ISO that you can mount to your VMs. Many thanks to the XAPI developers who have kept the related source code for us in the XAPI project, rather than delete it.
OpenFlow support in the SDN controller: support added out of the box for Xen Orchestra to leverage it. The related code is being contributed upstream to the XAPI project.
We replaced Citrix's gpumon package, not built by us, by a mock build of gpumon sources, without the proprietary nvidia developer kit. See dedicated section below
Alternate kernel updated to version 4.19.140.
Additional packages:
- ZFS updated to 0.8.4.
- glusterfs 8.1 added to the repositories
- New additional driver package: r8125-module

Reimplementation of UEFI VM support

The component that handles UEFI variables for VMs, varstored, was redistributable but not open source. So we had to implement our own, named uefistored.

It is not entirely ready yet, that's why XCP-ng 8.2 beta's UEFI VM support is incomplete. Systems like Alpine linux will work, but neither Windows nor Debian will install for now.

This is being worked on and you should see updates very soon.

Since the component was entirely rewritten, we will ask the community to give special attention to UEFI VM tests.

New storage drivers: ZFS, GlusterFS, CephFS... and other changes

We added three new experimental storage drivers: zfs, glusterfs and cephfs.

We also decided to remove the sm-additional-drivers package (that only contained the xfs SR driver) and include all SR drivers by default, including experimental ones. However, we do not install all the dependencies on dom0 by default: xfsprogs, gluster-server, ceph-common, zfs... They need to be installed using yum for you to use the related SR drivers.

`zfs`

We already provided zfs packages in our repositories, but there was no dedicated SR driver. Users would use the file driver, which has a major drawback: if the zpool is not active, that driver may believe that the SR suddenly became empty, and forget all VDI metadata.

So we developed a dedicated zfs SR driver that checks whether zfs is present before drawing such conclusions.

Required dependency: zfs
Documentation: https://xcp-ng.org/docs/storage.html#zfs

`glusterfs`

Use this driver to connect to an existing Gluster storage as a shared SR.

Required dependency: glusterfs-server
Documentation: https://xcp-ng.org/docs/storage.html#glusterfs

`cephfs`

Use this driver to connect to an existing Ceph storage through the CephFS storage interface.

Required dependency: ceph-common (not provided in XCP-ng repositories. Check the docs.)
Documentation: https://xcp-ng.org/docs/storage.html#cephfs

`xfs`

Use this driver to create and use an XFS SR.

Required dependency: xfsprogs
Documentation : to be written. There's nothing special about it, it works exactly like the ext driver, but uses xfs instead of ext4 as a filesystem.

Core Scheduling

The updated Xen in XCP-ng 8.2 brings support for Core Scheduling.

As you probably know, Hyper Threading defeats all mitigations of CPU vulnerabilities related to side-channel attacks. That's why it was required to disable it as part of the mitigations. The reason is that with Hyper Threading enabled you can't protect a VM's vCPUs from attacks originating from other VMs that have workloads scheduled on the same physical core.

With Core Scheduling, you now have another solution: you can choose to leave Hyper Threading enabled and ask the scheduler to always group vCPUs of a given VM together on the same physical core(s). This will remove the vulnerability to a class of attacks from other VMs, but will leave the VM processes vulnerables to attacks from malevolent processes from within itself. To be usedonly with entirely trusted workloads.

How to enable it:

In a future release of Xen Orchestra, you will be able to enable it with a simple click in the host's advanced settings
Until then, you can enable it manually by adding sched-gran=core to Xen's command line parameters in grub.cfg and reboot. See Xen's documentation.

`gpumon`'s mock build (tests with nVIDIA GPUs required)

gpumon is an opensource component that can't be built without the appropriate proprietary nvidia toolkit. It is supposed to be only used for nvidia vgpus, which we don't support (proprietary stuff too). We already tried to remove it from XCP-ng in 8.1, however you may remember than we discovered, after the release of XCP-ng 8.1, that the XAPI would not manage to start without it on systems that have nVIDIA GPUs.

This time, we chose another approach: we built a mock version of gpumon (working, but giving fake statistics about the GPUs), so the binary is present on XCP-ng and the XAPI is happy enough with it to start in presence of an nVIDIA GPU.

However, there still is a risk that the XAPI would try to use gpumon later, not only during the startup process, so we need users with nVIDIA GPUs to install XCP-ng 8.2 and test it in various situations.

How to upgrade from previous releases

Since XCP-ng 8.2.0 is a minor release, both upgrade methods are supported:

From the installation ISO
From command line using yum (from XCP-ng 8.0 or 8.1 only!)

Refer to the upgrade howto.

Downloads:

Standard ISO: http://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.0-beta.iso
Net-install ISO: http://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.0-beta-netinstall.iso
SHA256SUMS: http://mirrors.xcp-ng.org/isos/8.2/SHA256SUMS
Signatures of the sums: http://mirrors.xcp-ng.org/isos/8.2/SHA256SUMS.asc

It's a good habit to check your downloaded ISO against the SHA256 sum and for better security also check the signature of those sums. Although our mirror redirector does try to detect file changes on mirrors, it's should always be envisioned that a mirror (or in the worst case our source mirror) gets hacked and managed to provide both fake ISOs and SHA256 sums. But they can't fake the signature. See https://xcp-ng.org/docs/mirrors.html#security.

After the installation, stay updated

Run yum update regularly. We'll fix bugs that are detected regularly until the final release. Subscribe to this thread (and make sure to activate mail notifications in the forum parameters if you want them): we'll announce every important update to the beta here.

What to test

Everything and anything! A community effort to list things to be tested has been started at https://github.com/xcp-ng/xcp/wiki/Test-XCP.

In addition, we ask you to give special attention to:

UEFI VM support
The new storage drivers

Report or ask anything as replies to this thread.

Feedback is always useful, both about successes and failures.

hoerup

@stormi whats the opinion on using the new gluster & ceph drivers for a hyper converged solution? Ok / should be ok but not tested / not recommended ??

stormi

@hoerup Should be ok but not tested enough.

olivierlambert

This is not trivial if you don't have enough Ceph/Gluster knowledge and a little bit of toolstack knowledge. Also, it won't survive an upgrade.

krakers

@olivierlambert do you have similar timeline plans to 8.1 for 8.2? I mean ~2 months between beta and final release?

olivierlambert

This will depend on what's going on, but that sound a reasonable timeframe (no promises though )

technot

Is there a Xcp-NG Center compatible version compiled yet ?

I had a little problems upgrading to the 8.2 beta.
The main tty showed a normal bootup untill a point where it showed a bunch of ACPI errors,
it worked for 10-15 min on starting previously disabled devices, and then just stopped.
I let it work another 10-15 min but all that came was this error, over and over again:

[  639.270804] EDAC sbridge: Seeking for: PCI ID 8086:3cf5
[  639.270804] EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled
[  639.270804] EDAC sbridge: Couldn't find mci handler
[  639.270804] EDAC sbridge: Failed to register device with error -19.

I switched to tty2 and lunched the init script from /opt/xensources and managed to upgrade from there. Wasnt too easy as the screen kept getting garbled from debug info that I assume default to tty2 output.

After the upgrade and reboot, it boots. But it takes a long long time.
Here is a small snippit from the first boot dmesg:

[   29.618412] usb 1-1.1.1: Product: G19 Gaming Keyboard
[   29.623569] input: G19 Gaming Keyboard as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.1/1-1.1.1:1.0/0003:046D:C228.0003/input/input7
[   30.290247] hid-generic 0003:046D:C228.0003: input,hidraw2: USB HID v1.10 Keyboard [G19 Gaming Keyboard] on usb-0000:00:1a.0-1.1.1/input0
[   30.293954] input: G19 Gaming Keyboard Consumer Control as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1.1/1-1.1.1:1.1/0003:046D:C228.0004/input/input8
[   30.958193] hid-generic 0003:046D:C228.0004: input,hiddev97,hidraw3: USB HID v1.10 Device [G19 Gaming Keyboard] on usb-0000:00:1a.0-1.1.1/input1
[  129.058404] EXT4-fs (sdb1): mounting ext3 file system using the ext4 subsystem
[  129.063909] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[  169.228203] e1000e 0000:00:19.0 0000:00:19.0 (uninitialized): registered PHC clock
[  169.356510] e1000e 0000:00:19.0 eth0: (PCI Express:2.5GT/s:Width x1) 8c:89:a5:a3:b6:f3
[  169.356512] e1000e 0000:00:19.0 eth0: Intel(R) PRO/1000 Network Connection

And its not done and ready untill it tries to initialize samba SR:

[  453.242190] xenbr1: port 1(eth1) entered blocking state
[  453.242190] xenbr1: port 1(eth1) entered forwarding state
[  476.236967] FS-Cache: Loaded
[  476.254307] FS-Cache: Netfs 'cifs' registered for caching
[  476.254307] Key type cifs.spnego registered
[  476.254307] Key type cifs.idmap registered

All in all this is much better then 8.1 and 8.0 where I had similar problems with ACPI, but it never got any further, and just locked up, no tty2 or such.
I actually had to use ACPI=off and/or ACPI=strict, and either way, the whole experience after upgrading was super slow vms, but with no aparant error messages so I returned to 7.6.
Well see how 8.2 fares, but so far so good
I have attached both the dmesg and xl-dmesg from the installer, incase you want to review what takes so long during boot.
xl-dmesg-log.txt
dmesg-log.txt

Thanks for all you hard work

brezlord

I upgraded to 8.2.0-beta to fix this issue nfs-storage-latency. Bellow are some tests done, if you need any specific tests just let me know.

I’m running XO from sources xo-server 5.67.0 xo-web 5.71.0

XEN-HOST-01 Dell R720
XEN-HOST-02 Dell R710
TrueNAS NFS Storage over 10GbE

I upgraded from 8.1.0 patched to 8.2.0-beta via ISO install. Install went through with no troubles. The host booted with no problems. All networks and storage was present.

All VMs booted and came back up with no issues and guest tools running. VMs consist of CentOS 7 & 8, Debian 9 & 10, Windows 10Pro

Successfully live migrated VMs from 8.1.0 host to 8.2.0-beta with no issues.

Attempted VM migration from 8.2.0-beta to 8.1.0 host failed as expected.

Successfully migrated VMs between 8.2.0-beta hosts.

Could not change pool master from 8.2.0-beta to 8.1.0 host. Get Unknown error from peer.

Successfully changed pool master with both hosts on 8.2.0-beta.

Both host on 8.2.0-beta, shut down host-02 which was not the pool master and the shared storage shows partially connected and no stats available for storage. Reboot host-02 and shared storage is now healthy and stats are back. Disabled host-02 and shutdown and still had the same issue as above. The only was to get stats back and get shares storage back to healthy and stats working is to forget the storage attached to host-02 that was shut down.

XCP-ng Center v20.04.00.32 will not connect to 8.2.0-beta hosts.

Had to reinstall xscontainer package to have docker container visibility in XO.

stormi

@technot said in XCP-ng 8.2.0 beta now available!:

Is there a Xcp-NG Center compatible version compiled yet ?

I don't think there is one. It depends on the community. Previous releases were built by @borzel. I don't know if he will have the time to do it this time.

demanzke

I just updated using the ISO-method because the yum method didn't boot successfully.
What would be the best way to migrate from a ZFS file-SR to ZFS-SR? I assume setting the SR type after creation isn't intended and I can't find that in the XAPI docs either.
Is the cleanest solution to create a new ZFS dataset, ZFS-SR on top and move the virtual drives?

stormi

I would have been interested in debugging why the system didn't boot successfully after an update using yum. Can you give us more details?

To migrate the ZFS SR, the only clean way I am aware of is to create a new one. There may be a hacky way but it would probably involve modifying XAPI database objects dans metadata written in the SR itself so I wouldn't attempt it.

stormi

@brezlord said in XCP-ng 8.2.0 beta now available!:

Could not change pool master from 8.2.0-beta to 8.1.0 host. Get Unknown error from peer.

When you get such error in XO, there's usually more information in XO's log page. However, it is expected that you can't move the master to a host running a lower release.

Both host on 8.2.0-beta, shut down host-02 which was not the pool master and the shared storage shows partially connected and no stats available for storage. Reboot host-02 and shared storage is now healthy and stats are back. Disabled host-02 and shutdown and still had the same issue as above. The only was to get stats back and get shares storage back to healthy and stats working is to forget the storage attached to host-02 that was shut down.

Ok, I don't think it's a regression but it may be interesting to create a GitHub issue with all the details.

Had to reinstall xscontainer package to have docker container visibility in XO.

That is expected, unless you upgrade using yum.

if you need any specific tests just let me know.

If your host has an nVIDIA GPU, we would be interested in knowing it because there used to be bugs specific to hosts with such a GPU, even when not used.

And be ready for testing UEFI VMs soon! I'll post here when the updated uefistored package is available.

brezlord

Thanks for the info. So doing a ISO update is essentially like a fresh install but keeps settings.

I don't have any nVIDIA GPUs, sorry can't help there.

I'll boot the second host and get more info to raise an issue for the SR problem.

brezlord

I have raised issue https://github.com/xcp-ng/xcp/issues/439 for the above SR problem.

brezlord created this issue in xcp-ng/xcp

open Shared storage & XCP-ng tools show partially connected when 1 of 2 hosts shutdown. #439

technot

And be ready for testing UEFI VMs soon! I'll post here when the updated uefistored package is available.

Does the host have to be booted in UEFI mode for UEFI VMs to function?

If it does, is there an easy way to make the change from legacy to UEFI?
All I know, is the install docs clearly state you should never change mode when upgrading.
Is the only way to export all the vms, and reinstall, then import the vms back?

stormi

@technot said in XCP-ng 8.2.0 beta now available!:

Does the host have to be booted in UEFI mode for UEFI VMs to function?

No, it's not related to the host's mode.

technot

@stormi
Is it only lacking to get built/compiled? If so I could do it sometime during the weekend. Or has the code not been updated to reflect any changes done with xcp-ng 8.2. If the latter, I am afraid I cant help.

stormi

It's not ready yet so you can't help now. Our developer who wrote that piece of code (uefistored) is chasing the last known bugs and then I'll build an update and push it to our RPM repository for XCP-ng 8.2. I'll announce it when it's ready.

technot

I dont know if this is expected behaviour, after upgrading all the way from 7.6 to 8.2 beta. But on a windows guest I have an AMD gpu in full pci passthrough, the device didnt show up after boot. And instead there was another device showing fault.
Intel 8237sb pci to usb universal host controller, was throwing an error 43.
And under Display adapters, there was only the microsoft remote emulated and a "Microsoft Basic" devices showing.
I reinstalled the AMD driver, and rebooted. And it came back to life.

Also, I must say the vms feel snappier. Faster boots, and less delay/latency. But I have not meassured pre/post, so its only a feeling!:D

technot

@stormi
Im sorry, it wasnt clear. What I meant is XCP-ng Center. If that code has been updated, or just a question of compiling it.