XCP-ng 8.2 updates announcements and testing

HeMaN

@stormi
I did not test the updates in the test-repository this time because I wanted to test the Rolling Pool Update function from XO. I recently created a pool of 2 hosts.

I noticed the patches were available according to XO on the recently installed host, but not on the host I am using already for a longer time (and used to install the test-patches before). This host is also the pool master.
Only after a yum clean all and a yum update the updates were visible on the first host and thus the pool.

After fixing that, the Rolling Pool Update went very smooth. I like this feature!

HeMaN

@stormi said in Updates announcements and testing:

yum clean metadata --enablerepo=xcp-ng-testing
yum update ca-certificates

After the rolling pool update of the released production patches, I wanted to test the ca-certificates from the testing repository as well.
Maybe I was to fast but I got no updates on both hosts

[18:30 xenserver-2 ~]# yum clean metadata --enablerepo=xcp-ng-testing
Loaded plugins: fastestmirror
Cleaning repos: xcp-ng-base xcp-ng-testing xcp-ng-updates
9 metadata files removed
8 sqlite files removed
0 metadata files removed
[18:30 xenserver-2 ~]# yum update ca-certificates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
xcp-ng-base/signature                                                 |  473 B  00:00:00
xcp-ng-base/signature                                                 | 3.0 kB  00:00:00 !!!
xcp-ng-updates/signature                                              |  473 B  00:00:00
xcp-ng-updates/signature                                              | 3.0 kB  00:00:00 !!!
(1/2): xcp-ng-updates/primary_db                                      |  46 kB  00:00:00
(2/2): xcp-ng-base/primary_db                                         | 1.2 MB  00:00:01
No packages marked for update

[18:34 xenserver-3 ~]# yum list installed ca-certificates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Installed Packages
ca-certificates.noarch                                                                2018.2.22-70.0.el7_5                                                                @install/$releasever

stormi

@heman You're right, I had forgotten an --enablerepo switch in my post. I edited it to add it.

HeMaN

@stormi said in Updates announcements and testing:

@heman You're right, I had forgotten an --enablerepo switch in my post. I edited it to add it.

I am not at my best today I noticed, I should have seen that

Anyway, installed without issue. No strange behaviour afterwards

olivierlambert

@heman said in Updates announcements and testing:

I noticed the patches were available according to XO on the recently installed host, but not on the host I am using already for a longer time (and used to install the test-patches before). This host is also the pool master.
Only after a yum clean all and a yum update the updates were visible on the first host and thus the pool.

After fixing that, the Rolling Pool Update went very smooth. I like this feature!

Thanks for the feedback. I think we must add a feature to do that from the plugin ("force refresh updates"). Pinging @nraynaud about this.

jmccoy555

@stormi Applied ca-certificates along with the security patch and all is good in my pool.

stormi

The blog post, as promised: https://xcp-ng.org/blog/2021/01/25/january-2021-security-update/

stormi

A new update of the Xen packages which is not a security update this time is available for tests. It fixes crash analysis with xen-crashdump-analyser (this runs automatically when the host crashes and puts results in /var/crash).

When the Release tag (e.g. 9.8.2.xcpng8.2) of the RPM was longer than a certain number of characters (last digit of Xen version + - + release tag <= 16 chars), it was truncated in the xen_extra field (xl info xen_extra) which caused xen-crashdump-analyser to not find some required files from /boot to conduct its crash analysis). The updated packages fix that by removing the .xcpng8.2 or .xcpng8.1 suffix from the filenames in /boot and from the xen_extra value.

Installation:

yum clean metadata --enablerepo=xcp-ng-testing
yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing

Main objective of the tests: as usual, detect obvious regressions.

If you want to test the fixed behaviour in case of crash, see https://github.com/xcp-ng/xcp/issues/476 (basically, provoke a crash with the command I gave in the comments, then check that the xen-crashdump-analyser.log file in /var/crash is OK).

ghost created this issue in xcp-ng/xcp

closed Kdump not generating proper Xen Crashdump Analyzer file on XCP-ng 8.2 #476

stormi

I've promoted the sudo (https://xcp-ng.org/blog/2021/01/28/security-issue-in-sudo/) and ca-certificate update candidates to official updates.

The Xen update is on hold until it's been sufficiently tested.

jmccoy555

A bit late to the party....... Updated my pool and no oddities to report.

Arraylist

Is a host reboot really necessary for the sudo and ca-certificate updates (as noted in the blog post)?
On an ordinary linux system I wouldn't see a need to restart after updating these packages.

stormi

@arraylist Good point. I'm updating the blog post.

HeMaN

@stormi I did an update for sudo on the hosts with XO CE and after the update I got the warning a reboot is required. I do not know if that is by default after installing updates or that it is a property of the package?

stormi

That's the default behaviour from XO because we currently don't have that kind of information about each updated package available to XO.

olivierlambert

Indeed. There's some plans to get a way to have more info on which packages really need a reboot. But it's not ultra straight forward.

stormi

A new batch of updates arrived in the testing repository, for XCP-ng 8.2

Xen (bugfixes)
xcp-ng-release-* for a fix to the ssh and sshd configuration in order to limit the list of accepted ciphers only to those that are considered secure enough. See list at https://support.citrix.com/article/CTX292897
xcp-python-libs: "A misconfigured PCI interface-rename rule leaves all host interfaces inaccessible." (quoting Citrix)
xenserver-status-report and bugtool-conn-tests: "On slower systems, xen-bugtool can experience time outs." (quoting Citrix again)

To install:

yum clean metadata --enablerepo=xcp-ng-testing
yum update bugtool-conn-tests xcp-python-libs xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenserver-status-report xcp-ng-release xcp-ng-release-config xcp-ng-release-presets --enablerepo=xcp-ng-testing

As usual, we're mainly interested in the verification that there's no obvious regression after the installation and a reboot.

A specific test: please check that your /etc/ssh/sshd_config and /etc/ssh/ssh_config have been updated by the update (there's a chance they aren't, if you have modified them in a way that makes the patching fail... And there won't be any warning unfortunately). Check for the presence of:

in sshd_config:

# Ciphers, MACs, KEX Algorithms & HostKeyAlgorithms
Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

and also GSSAPIAuthentication no (uncommented)

in ssh_config:

        Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
        MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
        KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
        HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

gskger

@stormi Had some time at hand and updated my three host playlab (8.2.0 fully patched). No problem with the update so far and creating linux VMs, live migrate, copy, delete, snapshot (with/without ram), backup and restore of linux and a windows 10 VM is working as expected.

Here is a diff of my sshd_config

[22:37 xcp01 ~]# diff -u /etc/ssh/sshd_config.pre /etc/ssh/sshd_config.post
--- /etc/ssh/sshd_config.pre    2021-02-04 19:57:46.121049198 +0100
+++ /etc/ssh/sshd_config.post   2021-02-04 22:37:18.283422751 +0100
@@ -24,7 +24,12 @@
 HostKey /etc/ssh/ssh_host_ecdsa_key
 HostKey /etc/ssh/ssh_host_ed25519_key

-# Ciphers and keying
+# Ciphers, MACs, KEX Algorithms & HostKeyAlgorithms
+Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
+MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
+KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
+HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa
+
 #RekeyLimit default none

 # Logging

and ssh_config file on host xcp01.

[22:37 xcp01 ~]# diff -u /etc/ssh/ssh_config.pre /etc/ssh/ssh_config.post
--- /etc/ssh/ssh_config.pre     2021-02-04 19:58:18.282487154 +0100
+++ /etc/ssh/ssh_config.post    2021-02-04 22:37:09.447028887 +0100
@@ -66,3 +66,8 @@
        SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
        SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
        SendEnv XMODIFIERS
+
+       Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
+       MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
+       KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
+       HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

Both files have not been modified. Made copies of the files before (pre) and after (post) the update.

stormi

@gskger Thanks. Looks good

stormi

New security updates to test for 8.1 and 8.2

Plus all the other update candidates queued for release if you haven't tested them yet (listed below).

New update candidates are available for testing and due to be released as official updates very soon, as is usually the case for security updates.

kernel security update, protecting against host DoS (unresponsiveness or crash)
a fix for the network performance issues - mainly visible with FreeBSD VMs - caused by a previous security fix
a fix for Xen's crash analysis after a reboot
(8.2 only) the fixes described in this post

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update bugtool-conn-tests kernel xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xcp-python-libs xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenserver-status-report --enablerepo=xcp-ng-testing
reboot

Test on XCP-ng 8.1

yum clean metadata --enablerepo=xcp-ng-testing
yum update kernel xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

What to test

The main goal is to avoid obvious regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

Between 24h and 48h.

gskger

@stormi Updated my two host playlab (8.2.0 fully patched, the third host currently serves as a Covid-19 homeoffice workstation) with no error. Rebooted and ran the usual tests (create, live migrate, copy and delete a linux and a windows 10 VM as well as create / revert snapshot (with/without ram) ). Fooled myself with a VM_LACKS_FEATURE error on the windows 10 VM until I realized that I forgot to install the Guest tools - I need more sleep. Will try a restore after tonights backup.

Edit: restore from backup worked as well