XCP-ng 8.2.0 RC now available!

stormi

Hello to everyone.

The RC (Release Candidate) release of XCP-ng 8.2 is available right now .

Stability status

Our internal tests have not revealed any major bugs. Community feedback during the beta phase was positive.

We expect the RC phase to be quite short (around two weeks), so don't wait too much or you will not be able to help us test it!

Long Term Support (LTS)

XCP-ng 8.2 will be our first LTS release.

Help us make it rock-solid through your tests!

What changed

Here are the main changes since XCP-ng 8.1. Changes since the beta are in bold characters.

XCP-ng 8.2 is based on the open source components of Citrix Hypervisor 8.2 (https://docs.citrix.com/en-us/citrix-hypervisor/whats-new.html).

Base components:

Xen 4.13.1 + patches
Kernel 4.19 + patches

Citrix announces:

Support for SLES 12 SP5 and Ubuntu 20.04 added
Support for Windows 7, Windows Server 2008 SP2 and Windows Server 2008 R2 SP1 removed. They may still work, but are not supported officially nor tested anymore by Citrix. Consider upgrading.
Maximum host RAM raised to 6 TB and maximum number of logical processors per host raised to 448 CPUs.
Support for installing TLS certificates on hosts (see https://docs.citrix.com/en-us/citrix-hypervisor/hosts-pools.html#install-a-tls-certificate-on-your-server)
TLS 1.2 protocol enforced for HTTPS traffic, and between XCP-ng components. Support for legacy SSL mode and TLS 1.0/1.1 protocols have been removed.

Other changes:

Bug fixed for hosts configured with DHCP. xcp-networkd used not to send the hostname along with the DHCP request. We contributed a fix to the xcp-networkd package, that was included in Citrix Hypervisor 8.2 and thus in XCP-ng 8.2 (note: packages with that fix are also available for XCP-ng 8.0 and 8.1 in the xcp-ng-testing repository).
Backup restore fixed for UEFI hosts. This upstream bug that our community had found and that we reported to Citrix teams has been fixed by them.

Changes specific to XCP-ng:

Support for Intel IceLake and CometLake CPUs.
Reimplementation of UEFI VM support : See dedicated section below for details.
New storage drivers: ZFS, GlusterFS, CephFS... and other changes. See dedicated section below for details.
The updated Xen brings support for Core Scheduling. We added support for changing the host configuration through the XAPI and contributed it upstream. See dedicated section below.
Guest tools ISO: Citrix doesn't provide a guest tools ISO anymore in Citrix Hypervisor. They replaced it with downloads from their internet website. We chose to retain the feature and still provide a guest tools ISO that you can mount to your VMs. Many thanks to the XAPI developers who have kept the related source code for us in the XAPI project, rather than delete it.
OpenFlow support in the SDN controller: support added out of the box for Xen Orchestra to leverage it. The related code is being contributed upstream to the XAPI project.
We replaced Citrix's gpumon package, not built by us, by a mock build of gpumon sources, without the proprietary nvidia developer kit. See dedicated section below
Alternate kernel updated to version 4.19.140.
Intel's e1000e driver updated to version 3.8.4 in order to support more devices
rsyslog synced from latest CentOS 7.8 security and bugfix update because several memory leaks have been patched. Please give special attention to the logging system during your tests..
Additional packages:
- ZFS updated to 0.8.5
- zstd updated to 1.4.5
- glusterfs 8.1 added to the repositories
- New additional driver package: r8125-module
- Alternate driver package intel-igb-alt updated to version 5.4.6

Reimplementation of UEFI VM support

The component that handles UEFI variables for VMs, varstored, was redistributable but not open source. So we had to implement our own, named uefistored.

Since the component was entirely rewritten, we ask the community to give special attention to UEFI VM tests.

Support for Secure Boot in VMs will also arrive soon.

New storage drivers: ZFS, GlusterFS, CephFS... and other changes

We added three new experimental storage drivers: zfs, glusterfs and cephfs.

We also decided to remove the sm-additional-drivers package (that only contained the xfs SR driver) and include all SR drivers by default, including experimental ones. However, we do not install all the dependencies on dom0 by default: xfsprogs, gluster-server, ceph-common, zfs... They need to be installed using yum for you to use the related SR drivers.

`zfs`

We already provided zfs packages in our repositories, but there was no dedicated SR driver. Users would use the file driver, which has a major drawback: if the zpool is not active, that driver may believe that the SR suddenly became empty, and forget all VDI metadata.

So we developed a dedicated zfs SR driver that checks whether zfs is present before drawing such conclusions.

Required dependency: zfs
Documentation: https://xcp-ng.org/docs/storage.html#zfs

`glusterfs`

Use this driver to connect to an existing Gluster storage as a shared SR.

Required dependency: glusterfs-server
Documentation: https://xcp-ng.org/docs/storage.html#glusterfs

`cephfs`

Use this driver to connect to an existing Ceph storage through the CephFS storage interface.

Required dependency: ceph-common (not provided in XCP-ng repositories. Check the docs.)
Documentation: https://xcp-ng.org/docs/storage.html#cephfs

`xfs`

Use this driver to create and use an XFS SR.

Required dependency: xfsprogs
Documentation : to be written. There's nothing special about it, it works exactly like the ext driver, but uses xfs instead of ext4 as a filesystem.

Core Scheduling

The updated Xen in XCP-ng 8.2 brings support for Core Scheduling.

As you probably know, Hyper Threading defeats all mitigations of CPU vulnerabilities related to side-channel attacks. That's why it was required to disable it as part of the mitigations. The reason is that with Hyper Threading enabled you can't protect a VM's vCPUs from attacks originating from other VMs that have workloads scheduled on the same physical core.

With Core Scheduling, you now have another solution: you can choose to leave Hyper Threading enabled and ask the scheduler to always group vCPUs of a given VM together on the same physical core(s). This will remove the vulnerability to a class of attacks from other VMs, but will leave the VM processes vulnerables to attacks from malevolent processes from within itself. To be usedonly with entirely trusted workloads.

How to enable it:

The latest version of Xen Orchestra, released at the same time as XCP-ng 8.2 RC, now offers a way to change the scheduling mode for a host.
You can also enable it manually by adding sched-gran=core to Xen's command line parameters in grub.cfg and reboot. See Xen's documentation.

`gpumon`'s mock build (tests with nVIDIA GPUs required)

gpumon is an opensource component that can't be built without the appropriate proprietary nvidia toolkit. It is supposed to be only used for nvidia vgpus, which we don't support (proprietary stuff too). We already tried to remove it from XCP-ng in 8.1, however you may remember than we discovered, after the release of XCP-ng 8.1, that the XAPI would not manage to start without it on systems that have nVIDIA GPUs.

This time, we chose another approach: we built a mock version of gpumon (working, but giving fake statistics about the GPUs), so the binary is present on XCP-ng and the XAPI is happy enough with it to start in presence of an nVIDIA GPU.

However, there still is a risk that the XAPI would try to use gpumon later, not only during the startup process, so we need users with nVIDIA GPUs to install XCP-ng 8.2 and test it in various situations.

How to upgrade from previous releases

Since XCP-ng 8.2.0 is a minor release, both upgrade methods are supported:

From the installation ISO
From command line using yum (from XCP-ng 8.0, 8.1 or 8.2 beta only!)

Refer to the upgrade howto.

Downloads:

Standard ISO: http://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.0-RC1.iso
Net-install ISO: http://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.0-RC1-netinstall.iso
SHA256SUMS: http://mirrors.xcp-ng.org/isos/8.2/SHA256SUMS
Signatures of the sums: http://mirrors.xcp-ng.org/isos/8.2/SHA256SUMS.asc

It's a good habit to check your downloaded ISO against the SHA256 sum and for better security also check the signature of those sums. Although our mirror redirector does try to detect file changes on mirrors, it's should always be envisioned that a mirror (or in the worst case our source mirror) gets hacked and managed to provide both fake ISOs and SHA256 sums. But they can't fake the signature. See https://xcp-ng.org/docs/mirrors.html#security.

After the installation, stay updated

Run yum update regularly. We'll fix bugs that are detected regularly until the final release. Subscribe to this thread (and make sure to activate mail notifications in the forum parameters if you want them): we'll announce every important update to the RC here.

What to test

Everything and anything! A community effort to list things to be tested has been started at https://xcp-ng.org/docs/develprocess.html#tests.

In addition, we ask you to give special attention to:

UEFI VM support
Logging (rsyslog update)
The new storage drivers
Hosts with nVIDIA GPUs

Report or ask anything as replies to this thread.

Feedback is always useful, both about successes and failures.

Rocky

Can I upgrade from Citrix Xen Server 7.1? I have an old host running that that I can test with.

update: putting 8.2RC1 ISO on a USB stick and trying it

update (upgrade from 7.1 using 8.2RC1 iso):

on first boot of the install ISO I got a nominal error of

Installer mode (UEFI) is mismatched with target boot mode (legacy)

Booting the usb stick from non UEFI resolved this and I was able to preform the upgrade install, on first boot I saw error on the screen that sat there for a while but then the console came up after a couple mins

ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.SAT0.SPT2._GTF.DSSP], AE_NOT_FOUND (20180810/psargs-330)

in XOA it says I need 1 patch but isn't showing a patch.
Installed Windows 10 from newest ISO, UEFI boot seems to be working well, installed Citrix Hypervisor PV Tools (managementagentx64.msi)
I noticed I can change the video resolution of my windows 10 VM from 1024x768? Any Idea how I fix this? (there was a post about needing to maybe change this in the UEFI settings but how do I get to those? (tried pressing a lot of F keys on boot and del and such)

Answer is here: https://xcp-ng.org/forum/topic/3498/windows-resolution-not-higher-than-1024x768/10

stormi

Yes, you can using the installation ISO.

_danielgurgel

Reconnection via iSCSI is failing. Upgrade pool from 8.0 to 8.2 (Starting with the Master host).

Nov  2 10:36:59 CLI219 SM: [10134] Warning: vdi_[de]activate present for dummy
Nov  2 10:37:00 CLI219 SM: [10200] Setting LVM_DEVICE to /dev/disk/by-scsid/36005076d0281000108000000000000d7
Nov  2 10:37:00 CLI219 SM: [10200] Setting LVM_DEVICE to /dev/disk/by-scsid/36005076d0281000108000000000000d7
Nov  2 10:37:00 CLI219 SM: [10200] Raising exception [97, Unable to retrieve the host configuration ISCSI IQN parameter]
Nov  2 10:37:00 CLI219 SM: [10200] ***** LVHD over iSCSI: EXCEPTION <class 'SR.SROSError'>, Unable to retrieve the host configuration ISCSI IQN parameter
Nov  2 10:37:00 CLI219 SM: [10200]   File "/opt/xensource/sm/SRCommand.py", line 376, in run
Nov  2 10:37:00 CLI219 SM: [10200]     sr = driver(cmd, cmd.sr_uuid)
Nov  2 10:37:00 CLI219 SM: [10200]   File "/opt/xensource/sm/SR.py", line 147, in __init__
Nov  2 10:37:00 CLI219 SM: [10200]     self.load(sr_uuid)
Nov  2 10:37:00 CLI219 SM: [10200]   File "/opt/xensource/sm/LVMoISCSISR", line 86, in load
Nov  2 10:37:00 CLI219 SM: [10200]     iscsi = BaseISCSI.BaseISCSISR(self.original_srcmd, sr_uuid)
Nov  2 10:37:00 CLI219 SM: [10200]   File "/opt/xensource/sm/SR.py", line 147, in __init__
Nov  2 10:37:00 CLI219 SM: [10200]     self.load(sr_uuid)
Nov  2 10:37:00 CLI219 SM: [10200]   File "/opt/xensource/sm/BaseISCSI.py", line 150, in load
Nov  2 10:37:00 CLI219 SM: [10200]     raise xs_errors.XenError('ConfigISCSIIQNMissing')
Nov  2 10:37:00 CLI219 SM: [10200]

olivierlambert

Upgrade via yum or via ISO?

_danielgurgel

@olivierlambert ISO. Our Storage is IBM v7000.

olivierlambert

xe host-list uuid=<host uuid> params=other-config

Let's see if a host lost its IQN.

ChuckNorrison

Still testing here since beta, runs very smooth on my single host system

Window 10 UEFI is still fine
NAS Remote Backup works awesome
Backup Notifications via Mail and Slack works as expected
Xen Orchestra works awesome

_danielgurgel

@olivierlambert said in XCP-ng 8.2.0 RC now available!:

xe host-list uuid=<host uuid> params=other-config

[13:01 CLI220 ~]# xe host-list uuid=e1d9e417-961b-43f4-8750-39300e197692 params=other-config
other-config (MRW)    : agent_start_time: 1604332891.; boot_time: 1604323953.; iscsi_iqn: iqn.2020-11.com.svm:cli219; rpm_patch_installation_time: 1604320091.898; last_blob_sync_time: 1570986114.22; multipathing: true; MAINTENANCE_MODE_EVACUATED_VMS_SUSPENDED: ; MAINTENANCE_MODE_EVACUATED_VMS_HALTED: ; MAINTENANCE_MODE_EVACUATED_VMS_MIGRATED: ; multipathhandle: dmp

[13:02 CLI219 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-11.com.svm:cli219
InitiatorAlias=CLI219

olivierlambert

@_danielgurgel said in XCP-ng 8.2.0 RC now available!:

@olivierlambert said in XCP-ng 8.2.0 RC now available!:

xe host-list uuid=<host uuid> params=other-config

[13:01 CLI220 ~]# xe host-list uuid=e1d9e417-961b-43f4-8750-39300e197692 params=other-config
other-config (MRW)    : agent_start_time: 1604332891.; boot_time: 1604323953.; iscsi_iqn: iqn.2020-11.com.svm:cli219; rpm_patch_installation_time: 1604320091.898; last_blob_sync_time: 1570986114.22; multipathing: true; MAINTENANCE_MODE_EVACUATED_VMS_SUSPENDED: ; MAINTENANCE_MODE_EVACUATED_VMS_HALTED: ; MAINTENANCE_MODE_EVACUATED_VMS_MIGRATED: ; multipathhandle: dmp

[13:02 CLI219 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2020-11.com.svm:cli219
InitiatorAlias=CLI219

So the IQN was correctly "saved" during the upgrade, it sounds correct to me. Is that host in a pool? Can you check for the other hosts?

_danielgurgel

@olivierlambert said in XCP-ng 8.2.0 RC now available!:

So the IQN was correctly "saved" during the upgrade, it sounds correct to me. Is that host in a pool? Can you check for the other hosts?

All others are correct and with other guests running.
I'll do some testing by removing the iSCSI connections altogether and replugging the host that has been updated.

If you have any tips on how to solve it, i appreciate your help.

olivierlambert

Frankly, I never saw that error before.

It's in there:

https://github.com/xapi-project/sm/blob/master/drivers/BaseISCSI.py#L143-L155

You have the field in XAPI, so I don't get why it fails with an error.

stormi

@Rocky said in XCP-ng 8.2.0 RC now available!:

on first boot of the install ISO I got a nominal error of
Installer mode (UEFI) is mismatched with target boot mode (legacy)

I would expect this if you're booting the USB stick in UEFI mode but have your existing system configured to boot in legacy mode (though I don't know exactly how that is detected).

Maybe @beshleman would know.

Update: I think that's a message from the installer itself, actually. It detected that the existing XenServer system was installed in BIOS mode, not UEFI, so it refuses to go on.

gskger

Upgraded a three host homelab:

ISO upgrade from XCP-ng 8.1 fully patched to XCP-ng 8.2.0 RC
ISO upgrade from XCP-ng 8.2.0 beta fully patched to XCP-ng 8.2.0 RC
ISO fresh install of XCP-ng 8.2.0 RC

My quick-and-dirty setup of Let's Encrypt on one of the hosts survived the ISO upgrade - nice

Did a yum update on all hosts and tested:

Add/remove hosts to XO from source
Join/remove hosts to a pool.
Copy/Move of VMs (within and across pools)
Move VDI between local/shared SR
Create/Change VMs (Linux, Windows (but not UEFI))
Add, Remove, Re-add NFS shares (ISO and storage)

Rocky

@stormi yea, I wasn't concerned about it was just documenting how it's normal I guess. Actually it's a helpful error even because then I knew to boot the USB the legacy (non-uefi) way.

Rocky

Shared nothing windows 10 Live migrations working well in my 2 hosts 8.2RC pool. both uefi VM and BIOS vm.

Rocky

So one error I got was on a XO backup job from a VM I moved from an 8.1 pool to a my test 8.2RC pool.

Error: INTERNAL_ERROR((Failure "Expected string, got '{}'"))

Update: I get this error on a new backup job also. This is on XO from the sources, I'll test with XOA also.

Update: this is working from XOA, not sure why I got the error in XO (from sources)... and I got 86.73 MiB/s. Which I think is faster than 8.1.

Update: this is now working for XO (from the sources) after I updated to 5.70.0 / 5.74.0 thanks for the great products.

olivierlambert

Thanks for the feedback! It's very helpful

beshleman

@stormi The installer just checks for the existence of "/sys/firmware/efi/", which is created when the kernel EFI driver reads that EFI is enabled from the boot_params passed to it from GRUB2. This is how it detects the "Installer mode"

The "target boot mode" is UEFI if an EFI (ESP) partition is discovered on the target disk, otherwise it's determined to be legacy.

A mismatch between the two will cause this error. This system is a UEFI system, but doesn't appear to have a ESP on the target disk.

stormi

Important update about XCP-ng 8.2

There were several updates pushed to the repositories for XCP-ng 8.2 RC. More than planned, because we chose to sync with the latest Citrix Hypervisor 8.2 hotfixes before the final release.

So please update (yum update, see https://xcp-ng.org/docs/updates.html if needed), reboot the hosts, and give us your feedback.

You're not supposed to notice anything changed. These changes are under the hood.

This is the last sprint. If all goes well, XCP-ng 8.2 final will be released real soon.

XCP-ng 8.2.0 RC now available!

Stability status

Long Term Support (LTS)

What changed

Reimplementation of UEFI VM support

New storage drivers: ZFS, GlusterFS, CephFS... and other changes

zfs

glusterfs

cephfs

xfs

Core Scheduling

gpumon's mock build (tests with nVIDIA GPUs required)

How to upgrade from previous releases

After the installation, stay updated

What to test

`zfs`

`glusterfs`

`cephfs`

`xfs`

`gpumon`'s mock build (tests with nVIDIA GPUs required)