XCP-ng 8.2 updates announcements and testing

rechotech

since this is unlikely the cause of our crash, where would you suggest we look to determine why our xcp servers crashed a few days ago?

stormi

I suggest opening a dedicated thread with as much information as possible.

stormi

As a starting point, see https://github.com/xcp-ng/xcp/wiki/Logfiles and look for whatever happened by the time of the crashes, if logging was still operational at that moment.

stormi

For XCP-ng 7.6 only

New security update for the latest Intel CPU vulnerabilities:

http://xenbits.xen.org/xsa/advisory-297.html

Install it with:

yum update --enablerepo='xcp-ng-updates_testing' microcode_ctl-2.1-26.xs5.x86_64 xen-dom0-libs-4.7.6-6.5.1.xcpng.x86_64 xen-dom0-tools-4.7.6-6.5.1.xcpng.x86_64 xen-hypervisor-4.7.6-6.5.1.xcpng.x86_64 xen-libs-4.7.6-6.5.1.xcpng.x86_64 xen-tools-4.7.6-6.5.1.xcpng.x86_64

There's nothing fancy in the way the updates were built so no breakage is expected, but we still need feedback from early-adopters before we push them to everyone.

Reboot required.

Downgrade if needed with:

yum downgrade microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools

What to test:

basic functions of the hypervisor, VMs...

Read the advisory carefully: installing the updates is not enough. There are other steps to be taken for complete protection.

Citrix announcement: https://support.citrix.com/article/CTX251995

Ultra2D

@stormi said in Updates announcements and testing:

yum update --enablerepo='xcp-ng-updates_testing' microcode_ctl-2.1-26.xs5.x86_64 xen-dom0-libs-4.7.6-6.5.1.xcpng.x86_64 xen-dom0-tools-4.7.6-6.5.1.xcpng.x86_64 xen-hypervisor-4.7.6-6.5.1.xcpng.x86_64 xen-libs-4.7.6-6.5.1.xcpng.x86_64 xen-tools-4.7.6-6.5.1.xcpng.x86_64

Should the pool master be updated first? Or it doesn't matter for there updates?

stormi

@Ultra2D Yes, always the pool master first.

stormi

@Ultra2D Technically, some updates could be installed in any order, but it's safer and simpler to consider that master is always updated first.

Ultra2D

@stormi Sure, but I'd rather test on a pool member first if trying as an early adopter .

stormi

@Ultra2D For this update, I think you could try with a slave first, since it's only an update of xen (and microcode_ctl), so I don't expect incompatibilities in pool management. But I wouldn't guarantee that cross-host operations such as migrations would work fine.

Ultra2D

@stormi I'll try it on a master, no worries. But not on a Friday.

borzel

@stormi installed just now all latest patches from updates testing via yum on my testbox at work
--> server bootet fine
--> VMs bootet (Debian 9, Ubuntu 18.04, all PVHVM)

stormi

Upping this thread in the hope for more test results.

rizaemet 0

@stormi installed on our pool hosts. Hosts boot normal. No vm boot issue Linux and Windows.

stormi

I have pushed the update to the updates repo. Security bulletin should follow shortly on the blog.

Ultra2D

Installing now, seems to run fine.

stormi

Announcement live: https://xcp-ng.org/blog/2019/05/21/xcp-ng-security-bulletin-mds-hardware-vulnerabilities-in-intel-cpus/

Was a long write but should be a short read

stormi

I just released a batch of updates for XCP-ng 7.6. It is not a security update (except the microcode_ctl update which brings updated microcode from Intel for some CPUs, such as the SandyBridge family).

Here's the list:

microcode_ctl-2.1-26.xs5.1.xcpng: updated microcodes from Intel regarding the MDS attacks. They mainly include microcodes for the SandyBridge family of CPUs which were not available previously.
sm-1.25.0-1.0.3.el7.centos:
- partial ext4 and XFS support for local storage. Nothing new if you already installed the update candidate previously, I'm just pushing it to everyone. Installing the experimental sm-additional-drivers package is still required if you want to use such filesystems in local storage repositories.
- NFS 4.1 support
xapi-1.110.1-1.7.xcpng:
- enables storage migration during pool upgrade. Will allow storage migration when upgrading a pool from 7.6 to 8.0 or later. Tested on a few VMs, but we can't promise zero risk, so, as usual, backup before and start with the less critical VMs. If you have shared storage, move your VMs to shared storage before the pool upgrade, this will save you much work. Or turn the VMs down if you can afford it.
- automatically opens the VxLAN network port for XAPI to use it. This is needed to use VxLAN with the new SDN controller from Xen Orchestra.
xcp-emu-manager-1.1.2-1.xcpng: new implementation of xcp-emu-manager, the same as the one that has been tested in XCP-ng 8.0 beta. Might fix a few corner cases, but overall what's expected is that it just works as well as before, and provides better logs if anything goes wrong.
xcp-ng-generic-lib-1.1.1-1.xcpng: a dependency of the new xcp-emu-manager
~~xenopsd-0.66.0-1.1.xcpng: fixes live migration from older releases when the VM has no platform:device_id defined (see upstream bug).~~ Update withdrawn: does fix the bug, but causes a transient live migration error during the update when the hosts have different patch levels. If you already installed the update and rebooted your hosts no problem, you're already past the issue. If you installed it but haven't moved your VMs around to reboot your hosts, make sure all your hosts have the same patch level and restart their toolstacks before migrating VMs. If you haven't installed the update but you want to install it, it is still available in the xcp-ng-updates_testing repository. Thanks to lavamind for helping me debug this over IRC.
xcp-ng-xapi-plugins-1.4.0-1.xcpng: replaces xcp-ng-updater. Includes the updater plugin that allows Xen Orchestra to install updates, as well as the new ZFS pool discovery plugin and the HyperThreading detection plugin (see latest Xen Orchestra blog post)
xcp-ng-deps-7.6.0-5: update needed due to the renaming of xcp-ng-updater into xcp-ng-xapi-plugins

Some updates bring no change from user point of view. This is just some needed clean-up of the packaging, and I'm using the opportunity offered by this batch of update to include them:

conversion-plugin-8.0.0-1.1
vendor-update-keys-1.3.6-1.1

Note that most of these update candidates have been available for a long time (months), but I did not get enough feedback so it had to wait until I was able to give them more testing myself. So don't hesitate to subscribe to this thread, make sure e-mail notifications are active, and take part in the testing of updates candidates to help us release in a timely manner.

As a reminder, the security update for XCP-ng 7.5 regarding MDS attacks, that took one of our developers some time to backport, is still an update candidate that can't be pushed to everyone unless it gets tested by the community (but maybe there's no real need to update it).

MajorTom

@stormi I've just done yum update (Complete!) and rebooted.
Seems OK.

Comparing before and after the reboot, the only differences in output of xl dmesg (apart timestamps) are:

<  Detected 3504.246 MHz processor.
---
>  Detected 3504.212 MHz processor.

<   Init. ramdisk: 000000026ee32000->000000026ffff04b
---
>   Init. ramdisk: 000000026ee32000->000000026ffff164

Also, microcodes don't seem to be newer:

# xl dmesg | grep -i microcode
(XEN) [    0.000000] microcode: CPU0 updated from revision 0x8e to 0xb4, date = 2019-04-01
(XEN) [   18.296976] microcode: CPU2 updated from revision 0x8e to 0xb4, date = 2019-04-01
(XEN) [   18.373666] microcode: CPU4 updated from revision 0x8e to 0xb4, date = 2019-04-01
(XEN) [   18.450559] microcode: CPU6 updated from revision 0x8e to 0xb4, date = 2019-04-01

but it's OK, isnt it?

stormi

@MajorTom only some microcodes have been updated since the last microcode update, yes.

maxcuttins

@stormi just updated.
After the upgrade the reboot of the master failed with this screen: