XCP-ng 8.2 updates announcements and testing

HeMaN

@stormi I did an update for sudo on the hosts with XO CE and after the update I got the warning a reboot is required. I do not know if that is by default after installing updates or that it is a property of the package?

stormi

That's the default behaviour from XO because we currently don't have that kind of information about each updated package available to XO.

olivierlambert

Indeed. There's some plans to get a way to have more info on which packages really need a reboot. But it's not ultra straight forward.

stormi

A new batch of updates arrived in the testing repository, for XCP-ng 8.2

Xen (bugfixes)
xcp-ng-release-* for a fix to the ssh and sshd configuration in order to limit the list of accepted ciphers only to those that are considered secure enough. See list at https://support.citrix.com/article/CTX292897
xcp-python-libs: "A misconfigured PCI interface-rename rule leaves all host interfaces inaccessible." (quoting Citrix)
xenserver-status-report and bugtool-conn-tests: "On slower systems, xen-bugtool can experience time outs." (quoting Citrix again)

To install:

yum clean metadata --enablerepo=xcp-ng-testing
yum update bugtool-conn-tests xcp-python-libs xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenserver-status-report xcp-ng-release xcp-ng-release-config xcp-ng-release-presets --enablerepo=xcp-ng-testing

As usual, we're mainly interested in the verification that there's no obvious regression after the installation and a reboot.

A specific test: please check that your /etc/ssh/sshd_config and /etc/ssh/ssh_config have been updated by the update (there's a chance they aren't, if you have modified them in a way that makes the patching fail... And there won't be any warning unfortunately). Check for the presence of:

in sshd_config:

# Ciphers, MACs, KEX Algorithms & HostKeyAlgorithms
Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

and also GSSAPIAuthentication no (uncommented)

in ssh_config:

        Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
        MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
        KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
        HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

gskger

@stormi Had some time at hand and updated my three host playlab (8.2.0 fully patched). No problem with the update so far and creating linux VMs, live migrate, copy, delete, snapshot (with/without ram), backup and restore of linux and a windows 10 VM is working as expected.

Here is a diff of my sshd_config

[22:37 xcp01 ~]# diff -u /etc/ssh/sshd_config.pre /etc/ssh/sshd_config.post
--- /etc/ssh/sshd_config.pre    2021-02-04 19:57:46.121049198 +0100
+++ /etc/ssh/sshd_config.post   2021-02-04 22:37:18.283422751 +0100
@@ -24,7 +24,12 @@
 HostKey /etc/ssh/ssh_host_ecdsa_key
 HostKey /etc/ssh/ssh_host_ed25519_key

-# Ciphers and keying
+# Ciphers, MACs, KEX Algorithms & HostKeyAlgorithms
+Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
+MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
+KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
+HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa
+
 #RekeyLimit default none

 # Logging

and ssh_config file on host xcp01.

[22:37 xcp01 ~]# diff -u /etc/ssh/ssh_config.pre /etc/ssh/ssh_config.post
--- /etc/ssh/ssh_config.pre     2021-02-04 19:58:18.282487154 +0100
+++ /etc/ssh/ssh_config.post    2021-02-04 22:37:09.447028887 +0100
@@ -66,3 +66,8 @@
        SendEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
        SendEnv LC_IDENTIFICATION LC_ALL LANGUAGE
        SendEnv XMODIFIERS
+
+       Ciphers chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
+       MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
+       KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
+       HostKeyAlgorithms ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa

Both files have not been modified. Made copies of the files before (pre) and after (post) the update.

stormi

@gskger Thanks. Looks good

stormi

New security updates to test for 8.1 and 8.2

Plus all the other update candidates queued for release if you haven't tested them yet (listed below).

New update candidates are available for testing and due to be released as official updates very soon, as is usually the case for security updates.

kernel security update, protecting against host DoS (unresponsiveness or crash)
a fix for the network performance issues - mainly visible with FreeBSD VMs - caused by a previous security fix
a fix for Xen's crash analysis after a reboot
(8.2 only) the fixes described in this post

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update bugtool-conn-tests kernel xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xcp-python-libs xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenserver-status-report --enablerepo=xcp-ng-testing
reboot

Test on XCP-ng 8.1

yum clean metadata --enablerepo=xcp-ng-testing
yum update kernel xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

What to test

The main goal is to avoid obvious regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

Between 24h and 48h.

gskger

@stormi Updated my two host playlab (8.2.0 fully patched, the third host currently serves as a Covid-19 homeoffice workstation) with no error. Rebooted and ran the usual tests (create, live migrate, copy and delete a linux and a windows 10 VM as well as create / revert snapshot (with/without ram) ). Fooled myself with a VM_LACKS_FEATURE error on the windows 10 VM until I realized that I forgot to install the Guest tools - I need more sleep. Will try a restore after tonights backup.

Edit: restore from backup worked as well

olivierlambert

Thanks again @gskger

At some point, you'll earn a "XCP-ng QA team" badge

gskger

@olivierlambert It is a pleasure to help. I highly appreciate your teams work on making XCP-ng better and more secure as well as the support you give.

jmccoy555

@stormi Only updated my test host so far, but all looks good to me.... no passthrough issues!!!

Will try and update my pool over the weekend and run some FreeBSD speed tests before and after (been waiting for that patch to make it out into the world).

HeMaN

@stormi said in Updates announcements and testing:

yum update bugtool-conn-tests kernel xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xcp-python-libs xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenserver-status-report --enablerepo=xcp-ng-testing

Updated both host in the pool (xcp-ng 8.2 fully patched), did my usual stuff and found no anomalities.
Looking good for me!

stormi

Many thanks to all testers. The updates have been published on Friday, along with the blog post: https://xcp-ng.org/blog/2021/02/26/february-2021-security-updates/

jmccoy555

Updating my pool now.... quite a substantial speed increase on FreeBSD

root@FILE001:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.125 port 5001 connected with 10.10.1.126 port 42952
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.1 sec   694 MBytes   576 Mbits/sec
^Croot@FILE001:~ # iperf -c 10.10.1.126
------------------------------------------------------------
Client connecting to 10.10.1.126, TCP port 5001
TCP window size:  105 KByte (default)
------------------------------------------------------------
[  3] local 10.10.1.125 port 50605 connected with 10.10.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   202 MBytes   169 Mbits/sec
root@FILE001:~ # iperf -c 10.10.1.126
------------------------------------------------------------
Client connecting to 10.10.1.126, TCP port 5001
TCP window size: 80.8 KByte (default)
------------------------------------------------------------
[  3] local 10.10.1.125 port 45072 connected with 10.10.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  4.06 GBytes  3.49 Gbits/sec
root@FILE001:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.125 port 5001 connected with 10.10.1.126 port 47101
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.17 GBytes  1.86 Gbits/sec
^Croot@FILE001:~ #

stormi

It's testing time again!

We changed the way we build the guest tools ISO and so we need as much feedback as we can get regarding the new guest tools. The main goal is to look for regressions. Details and feedback on https://xcp-ng.org/forum/topic/4378/new-guest-tools-iso-for-linux-and-freebsd-can-you-help-with-the-tests.

stormi

New security updates to test for 8.1 and 8.2

New update candidates are available for testing and due to be released as official updates very soon, as is usually the case for security updates.

kernel security and bugfix update, prevents DoS attacks from the guests and brings fixes to event handling.
openvswitch security update, prevents malicious network traffic to cause packets to be dropped.
Fixed ixgbe driver to avoid the memory leaks discussed at https://xcp-ng.org/forum/topic/2507/alert-control-domain-memory-usage

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update kernel openvswitch openvswitch-ipsec intel-ixgbe --enablerepo=xcp-ng-testing
reboot

Note: this won't update openvswitch-ipsec if not present on the system, that is, unless you're using XO's private networks.

Test on XCP-ng 8.1

This will be the last security update for XCP-ng 8.1, whose end of life is officially today, March 31st.

yum clean metadata --enablerepo=xcp-ng-testing
yum update kernel openvswitch openvswitch-ipsec intel-ixgbe --enablerepo=xcp-ng-testing
reboot

What to test

The main goal is to avoid obvious regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

Note: as the the code for event handling was changed, pay special attention to network performance in VMs, especially FreeBSD VMs since they are outside the scope of Citrix testing.

Test window before official release of the updates

Between 24h and 48h.

gskger

@stormi Successfully updated my two host playlab (8.2.0 full patched, but with the new guest tools ISO). Rebooted both hosts and ran my usual test program (create, live migrate (without (only Debian) and with guest-tools installed (Debian 10 and Windows 10) avoiding the VM_LACKS_FEATURE error on Windows 10 - more sleep this time ), copy and delete as well as create / revert to snapshot (with/without ram) and (live/halted=shutdown) storage migration remote to local SR and back). Also restored a Debian 10 VM from a pre-update backup with no problem. Looks good . Will see how backup runs tonight, but I am confident that this will work as well.

Edit #1: typos

Edit #2: Thanks to Xen Orchestra, out-of-band management even for Optiplex 9010s, a notebook and a very nice evening on the balcony, I extended my tests to Ubuntu 20 LTS. And you might guess it - works as well

Edit #3: Deleted a subsequent post on an asyncMap is not defined error because it is not related to the security update.

gskger

This post is deleted!

tzerod

Tested new security patches on 8.2.

Network performance seems just fine. Have 4 Windows server 2019 and 2 Fedora Server VMs recording video feed. Haven't observed any difference in performance.

Supermicro SYS-1029U-TR4 with 768gig ram.
Network usage: about 400mbps constant.
ZFS

jmccoy555

Only updated one host in my pool so far, but it has my TrueNAS VM and network performance looks comparable

edit:

Actually quite an improvement in comparison to my results above

root@FILE001:~ # iperf -c 10.10.1.126
------------------------------------------------------------
Client connecting to 10.10.1.126, TCP port 5001
TCP window size: 80.8 KByte (default)
------------------------------------------------------------
[  3] local 10.10.1.125 port 35576 connected with 10.10.1.126 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.1 sec  6.30 GBytes  5.36 Gbits/sec
root@FILE001:~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.125 port 5001 connected with 10.10.1.126 port 54773
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  3.57 GBytes  3.06 Gbits/sec

Also done a VM reboot now and no passthrough issues