XCP-ng 8.2 updates announcements and testing

JeffBerntsen

@stormi Following the warning in this patch set's announcement, I did not install it on my XOSTOR setup. Is it safe to do so now?

stormi

@JeffBerntsen Yes, we published the XOSTOR-ready version of the update at the same time. The only difference you should notice with other hosts is the installed version of sm (and subpackages) will be 2.30.8-7.1.0.linstor.2.xcpng8.2

JeffBerntsen

@stormi Thanks for the quick reply. That's what I'd figured but I like to be cautious so thought I should ask.

stormi

Urgent security update candidate (only a few hours to provide feedback)

There's an escalation of privilege vulnerability in Intel CPUs of the last few years. It was silently mitigated by Intel in previous microcode updates for the most recent CPU generations, but older affected CPUs were only fixed in yesterday's microcode.

See https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00950.html

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update microcode_ctl --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

microcode_ctl: 2.1-26.xs26.2.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

If you don't have time for more, just installing the update, rebooting, and checking one VM can start, will be enough.

Test window before official release of the updates

A few hours.

Gheppy

The server (Lenovo System x3650 M5) has started and all 4 VMs are started and functional

Andrew

@stormi Microcode updated on affected Gen11 i7. Running normally.

stormi

Thanks for the feedback! Update published: https://xcp-ng.org/blog/2023/11/15/november-2023-security-update/

The blog post also contains information about two vulnerabilities in Xen, but which don't affect XCP-ng in a supported and/or default configuration.

Users of PV guests who still haven't converted them to HVM should consider it, though.

stormi

New security update candidates

As promised in the announcement of the previous security update, here's a new one which includes changes for previously missing XSA updates as well as an updated AMD microcode.

Security updates

xen-*:
- Fix XSA-445 - x86/AMD: mismatch in IOMMU quarantine page table levels. On x86 AMD systems with IOMMU hardware, a device in quarantine mode, using dom_io, could access leaked data from previously quarantined pages. This is not enabled by default in XCP-ng, but can still be enabled at Xen boot time.
- Fix XSA-446 - x86: BTC/SRSO fixes not fully effective. A PV guest could infer memory content from other guests. We do not recommand using PV guests and have been suggesting switching to HVM for a while, so we do hope most users were not impacted by this.
linux-firmware: Update AMD microcode to 2023-10-19 drop, updating the family 19h, so Zen 3, Zen3+ and Zen 4. AMD Advisory here.

Other updates

We plan to also push other, non security, updates at the same time, to pave the way for the upcoming refreshed installation ISOs.

gpumon: suppression of logs which were needlessly written every 5s into /var/log/daemon.log.
tzdata: updated timezones.
vendor-drivers: pull new drivers into XCP-ng:
- igc-module: Intel device drivers for I225/I226
- r8125-module: Realtek r8125 device drivers
- mpi3mr-module: Broadcom mpi3mr RAID device driver

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" linux-firmware gpumon vendor-drivers tzdata --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.13.5-9.38.1.xcpng8.2
linux-firmware: 20190314-10.1.xcpng8.2 (Update: now 20190314-10.2.xcpng8.2, which adds firmware for rtl8125)
gpumon: 0.18.0-11.2.xcpng8.2
tzdata: 2023c-1.el7
vendor-drivers: 1.0.2-1.6.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates
~4 days

Samuel, along with David and Gaël

olivierlambert

Update done and reboot successful

JeffBerntsen

The update is installed and seems to be working without problems on my two test systems.

Andrew

@stormi Updated on several Intel Xeon servers. Updated on new Intel and AMD (zen3) systems with IGC and r8125 chips. One issue... the base install does not include the standard firmware for the 8125.

gskger

@stormi My two host cluster (HP ProDesk 600 G6) updated without an issue. Let's see how the cluster is performing during the coming days.

stormi

@Andrew I just pushed an updated linux-firmware to testing, with firmware for rtl8125. Should be available within 10 minutes.

Andrew

@stormi r8125 firmware loads.

stormi

So, we found out the AMD vulnerability actually doesn't affect XCP-ng directly, because Xen doesn't use AMD's SEV features currently.

The other two vulnerabilities still need fixing, but they both can only be exploited if XCP-ng is used in an either unlikely or unsupported way. We'll fix them in due course, but won't push the update to everyone today as initially planned. We will delay them slightly to give them a chance to be grouped with future updates and thus cause less maintenance for users.

Thanks for the tests anyway: we will be able to publish these packages whenever we need now.

bleader

New security update candidates (kernel)

A new XSA was published on the 23rd of January, so we have a new security update to include it.

Security updates

kernel:
* Fix XSA-448 - Linux: netback processing of zero-length transmit fragment. An unprivileged guest can cause Denial of Service (DoS) of the host bysending network packets to the backend, causing the backend to crash. This was discovered through issues when using pfSense with wireguard causing random crashes of the host.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update kernel --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

kernel: 4.19.19-7.0.23.1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates
~2 days due to security updates.

stormi

Did anyone install it? The 2 days delay is over and we'll publish today.

Andrew

@stormi Yes, I installed it on a few running hosts. I did not have any kernel crashes before, and none after...

olivierlambert

Installed here, works

CJ

@NielsH Kind of off topic but figured I'd mention it as I only recently discovered this.

Not sure what VMs you're running, but if they can survive being off for a short time (redundancy of services or planned outage) you can reboot the host using Smart Reboot under the Advanced tab. While it incurs some downtime, it allows for a much faster reboot time than migrating the VMs to another server and back.

I use local storage as well and it's been a game changer for dealing with pool patches.