XCP-ng 8.2 updates announcements and testing

stormi

Announcement live: https://xcp-ng.org/blog/2019/05/21/xcp-ng-security-bulletin-mds-hardware-vulnerabilities-in-intel-cpus/

Was a long write but should be a short read

stormi

I just released a batch of updates for XCP-ng 7.6. It is not a security update (except the microcode_ctl update which brings updated microcode from Intel for some CPUs, such as the SandyBridge family).

Here's the list:

microcode_ctl-2.1-26.xs5.1.xcpng: updated microcodes from Intel regarding the MDS attacks. They mainly include microcodes for the SandyBridge family of CPUs which were not available previously.
sm-1.25.0-1.0.3.el7.centos:
- partial ext4 and XFS support for local storage. Nothing new if you already installed the update candidate previously, I'm just pushing it to everyone. Installing the experimental sm-additional-drivers package is still required if you want to use such filesystems in local storage repositories.
- NFS 4.1 support
xapi-1.110.1-1.7.xcpng:
- enables storage migration during pool upgrade. Will allow storage migration when upgrading a pool from 7.6 to 8.0 or later. Tested on a few VMs, but we can't promise zero risk, so, as usual, backup before and start with the less critical VMs. If you have shared storage, move your VMs to shared storage before the pool upgrade, this will save you much work. Or turn the VMs down if you can afford it.
- automatically opens the VxLAN network port for XAPI to use it. This is needed to use VxLAN with the new SDN controller from Xen Orchestra.
xcp-emu-manager-1.1.2-1.xcpng: new implementation of xcp-emu-manager, the same as the one that has been tested in XCP-ng 8.0 beta. Might fix a few corner cases, but overall what's expected is that it just works as well as before, and provides better logs if anything goes wrong.
xcp-ng-generic-lib-1.1.1-1.xcpng: a dependency of the new xcp-emu-manager
~~xenopsd-0.66.0-1.1.xcpng: fixes live migration from older releases when the VM has no platform:device_id defined (see upstream bug).~~ Update withdrawn: does fix the bug, but causes a transient live migration error during the update when the hosts have different patch levels. If you already installed the update and rebooted your hosts no problem, you're already past the issue. If you installed it but haven't moved your VMs around to reboot your hosts, make sure all your hosts have the same patch level and restart their toolstacks before migrating VMs. If you haven't installed the update but you want to install it, it is still available in the xcp-ng-updates_testing repository. Thanks to lavamind for helping me debug this over IRC.
xcp-ng-xapi-plugins-1.4.0-1.xcpng: replaces xcp-ng-updater. Includes the updater plugin that allows Xen Orchestra to install updates, as well as the new ZFS pool discovery plugin and the HyperThreading detection plugin (see latest Xen Orchestra blog post)
xcp-ng-deps-7.6.0-5: update needed due to the renaming of xcp-ng-updater into xcp-ng-xapi-plugins

Some updates bring no change from user point of view. This is just some needed clean-up of the packaging, and I'm using the opportunity offered by this batch of update to include them:

conversion-plugin-8.0.0-1.1
vendor-update-keys-1.3.6-1.1

Note that most of these update candidates have been available for a long time (months), but I did not get enough feedback so it had to wait until I was able to give them more testing myself. So don't hesitate to subscribe to this thread, make sure e-mail notifications are active, and take part in the testing of updates candidates to help us release in a timely manner.

As a reminder, the security update for XCP-ng 7.5 regarding MDS attacks, that took one of our developers some time to backport, is still an update candidate that can't be pushed to everyone unless it gets tested by the community (but maybe there's no real need to update it).

MajorTom

stormi I've just done yum update (Complete!) and rebooted.
Seems OK.

Comparing before and after the reboot, the only differences in output of xl dmesg (apart timestamps) are:

<  Detected 3504.246 MHz processor.
---
>  Detected 3504.212 MHz processor.

<   Init. ramdisk: 000000026ee32000->000000026ffff04b
---
>   Init. ramdisk: 000000026ee32000->000000026ffff164

Also, microcodes don't seem to be newer:

# xl dmesg | grep -i microcode
(XEN) [    0.000000] microcode: CPU0 updated from revision 0x8e to 0xb4, date = 2019-04-01
(XEN) [   18.296976] microcode: CPU2 updated from revision 0x8e to 0xb4, date = 2019-04-01
(XEN) [   18.373666] microcode: CPU4 updated from revision 0x8e to 0xb4, date = 2019-04-01
(XEN) [   18.450559] microcode: CPU6 updated from revision 0x8e to 0xb4, date = 2019-04-01

but it's OK, isnt it?

stormi

MajorTom only some microcodes have been updated since the last microcode update, yes.

maxcuttins

stormi just updated.
After the upgrade the reboot of the master failed with this screen:

stormi

maxcuttins Is there anything in the logs that would give information on what the "failure" was?

maxcuttins

I'm trying to find the log

maxcuttins

Another error while I was live migrating a VM from an host to another to reboot the host that need rebooting after the upgrade.

Migrating VM 'XXXXXX'
Internal Error:
Xenops_interface.Internal_error("Domain.Emu_manager_failure("Received error from emu-manager:xenguestInvalid Argument")")

stormi

maxcuttins Could you produce a system status report (on both hosts), upload them somewhere and send the link to me (in private if you prefer the host data not to be publicly available)?

stormi

So after a look at the logs, we are not sure where the issue comes from. Could be caused by faulty DIMM RAM because there are error messages related to RAM errors (but it's ECC RAM and it claims to have corrected them), but we're not sure.

stormi

I have withdrawn the update to xenopsd.

It does fix live migration from older releases when the VM has no platform:device_id defined (see upstream bug), but causes a transient live migration error during the update when the hosts have different patch levels. If you already installed the update and rebooted your hosts no problem, you're already past the issue. If you installed it but haven't moved your VMs around to reboot your hosts, make sure all your hosts have the same patch level and restart their toolstacks before migrating VMs. If you haven't installed the update but you want to install it, it is still available in the xcp-ng-updates_testing repository. Thanks to lavamind for helping me debug this over IRC.

Digitalllama

stormi could you provide some more details about the "transient live migration error" with the xenopsd update?

I have been experiencing issues live migrating larger VM's (20-40GB Ram) especially when the memory is in active use (eg MariaDB database). As far as I can tell my issue is related to the VM dynamic memory being reduced to below the actual memory use of the VM at which point a random processes crash - but not sure why this is happening since both the source and target XCP host are under allocated on memory. We upgraded all hosts in the pool before xenopsd was withdrawn and are currently scheduling in downtime for each VM so we can migrate them in an off state which seems to bypass the issue.

stormi

Digitalllama The issue you had is probably not related to that update. If that was caused by the update, then you'd have seen the VM reboot at the end of the migration process and it would have been duplicated on both hosts.

A dynamic memory minimum value set too low can cause the kind of process crash you experienced: the available memory for the VM being low, you may reach a point where the VM's kernel needs to fire the Out Of Memory Killer which will select a running process and kill it. If you want to be sure about that, you can try to set the same value for minimum and maximum dynamic memory and see if the migration issue still occurs.

stormi

To people not having updated their hosts yet with the latest update: wait a few more days! There's a kernel security update on its way, so you'll probably want to reboot only then.

Note that the security update will be mostly useful for people who put their hosts on a network that is reachable from a potential attacker.

stormi

The new kernel update candidate is available. As usual, I need some feedback before I can push it to everyone.

Citrix advisory: https://support.citrix.com/article/CTX256725

XCP-ng 7.5: install it with yum update kernel --enablerepo='xcp-ng-updates_testing'
XCP-ng 7.6: install it with yum update kernel --enablerepo='xcp-ng-updates_testing'
XCP-ng 8.0 beta/RC1: simply yum update

It is a security update. A distant attacker could manage to crash your host or raise its memory usage significantly with specially crafted network requests. Hosts isolated from public networks are safe, unless the attacker managed to get into your private network.

Reboot required (we do not support live patching at the moment, due to a closed source component in XenServer / Citrix Hypervisor).

Edit: update pushed: https://xcp-ng.org/blog/2019/07/12/xcp-ng-security-bulletin-kernel-update-sack-vulnerability/

stormi

Anyone available to test the security update on 7.5 and/or 7.6? It is a security update, so quite urgent.

cnaumer

Installed it a view minutes ago. Will report back.

cnaumer

Updated 3 hosts and so far no Problems. Transferred some machines etc no bad effects.

stormi

Hello everyone. I'm back from holidays with update candidates that need testing!

XCP-ng 7.6

xcp-ng-xapi-plugins

yum update xcp-ng-xapi-plugins --enablerepo=xcp-ng-updates_testing

This is the most important update. It fixes host memory consumption issues that could go as far as crashing several hosts at the same time (especially if EPEL repositories were active, which shouldn't be the case but often was prior to XCP-ng 8.0 where they are already present but disabled by default). Already fixed in XCP-ng 8.0.

Post-install: xe-toolstack-restart

microcode_ctl

yum update microcode_ctl --enablerepo=xcp-ng-updates_testing

Microcode update for the SandyBridge family of CPUs regarding the MDS attacks.

Post-install: reboot if you want it to be taken into account.

xcp-ng-pv-tools

yum update xcp-ng-pv-tools --enablerepo=xcp-ng-updates_testing

Linux guest tools: support for SLES 15 SP1, updated README, support for recent CoreOS.

Post-install: nothing to do.

xen

yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-updates_testing

Avoids possible memory corruption when forcibly shutting down a VM with AMD MxGPU attached. Or when the guest crashes.

Post-install: reboot to apply the changes.

XCP-ng 8.0

xen + guest templates

yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools guest-templates-json guest-templates-json-data-windows guest-templates-json-data-xenapp guest-templates-json-data-linux guest-templates-json-data-other --enablerepo=xcp-ng-testing

Avoid doing that on several hosts of the same pool at the same time, because the guest-template-json* updates will need to update the XAPI database at the same time. The /usr/bin/create-guest-templates tool that is called post-update is not designed to run concurrently (thanks to Silmaril on IRC for finding out at the cost of a broken XAPI database).

Changes:

same fix as in XCP-ng 7.6 regarding VMs with AMD MxGPU attached
fix a host crash that can occur when you force-shutdown a Windows VM that is in an unclean state
Windows VMs could hang for more than a minute after live migration
Windows VMs with the viridian_reference_tsc flag enabled could crash during live migration. This fix opens the door to possible performance improvements for your Windows VMs, because following that fix now Citrix advises to set viridian_reference_tsc and viridian_stimer flags to true for better performance.
Updated Windows VM templates with new default settings that set viridian_* to true.

Post-install:

reboot the host to apply the xen changes
consider modifying your existing Windows VM settings for possible better performance. See "After installing this hotfix" in https://support.citrix.com/article/CTX258320

microcode_ctl

yum update microcode_ctl --enablerepo=xcp-ng-testing

Microcode update for the SandyBridge family of CPUs regarding the MDS attacks. XCP-ng 8.0 already contained updated microcodes from Intel when released, before Citrix released a hotfix, but their update contains one additional file so we synced with their package.

Post-install: reboot if you want it to be taken into account.

What we need

As usual, Vates tests the updates internally, but we also rely on the community to widen the test cases and hardware tested, so we need you to install the updates and give us feedback, either positive or negative, before we can consider pushing those updates to everyone!

MajorTom

stormi said in Updates announcements and testing:

XCP-ng 7.6

[...]

microcode_ctl
yum update microcode_ctl --enablerepo=xcp-ng-updates_testing
Microcode update for the SandyBridge family of CPUs regarding the MDS attacks.

Post-install: reboot if you want it to be taken into account.

xcp-ng-pv-tools
yum update microcode_ctl --enablerepo=xcp-ng-updates_testing

I guess that above command for xcp-ng-pv-tools should not be the same as the one for microcode_ctl

XCP-ng 8.0

[...]

microcode_ctl

yum update microcode_ctl --enablerepo=xcp-ng-updates_testing

This command returns error:

# yum update microcode_ctl --enablerepo=xcp-ng-updates_testing
Loaded plugins: fastestmirror


Error getting repository data for xcp-ng-updates_testing, repository not found

I guess it meant to be:

# yum update microcode_ctl --enablerepo=xcp-ng-testing

as this worked for me.

HTH