XCP-ng 8.2 updates announcements and testing

gskger

@bleader Installed on my playlab. Everything looks normal, let's see how it goes.

stormi

I just pushed an update for microcode_ctl, in xcp-ng-candidates again.

It only changes one microcode file, 06-a5-03 (CML-S62, Core Gen 10), which Intel had forgotten to update in their initial release.

(We found out and reported it to them)

bleader

Update published https://xcp-ng.org/blog/2024/08/16/august-security-update/

Thank you all for testing

manilx

@bleader All pools updated via RPU without issues.

bleader

New security update candidates (xen, microcode_ctl)

A new XSA was published on September 24th 2024.
Intel published a microcode update on the September 10th 2024.
We also included an updated xcp-ng-release for testing, althrough not related to security.

XSA-462 a malicious HVM or PVH guest can trigger a DoS of the host.

SECURITY UPDATES

xen-*:
* Fix XSA-462 - x86: Deadlock in vlapic_error(). The handling of x86's APIC (Advanced Programmable Interrupt Controller) allows a guest to configure an illegal vector to handle error interrupts. This causes the vlapic_error() to recurse, this is protected, but the lock used for this protection will try to be taken recursiveley, leading to a deadlock.
microcode_ctl:
    * Latest Intel microcode update, still named IPU 2024.3, including security updates for:
        * INTEL-SA-01103
        * INTEL-SA-01097

Other updates

xcp-ng-release:
    * Update the "XOA Quick deploy" feature in the host's web page.
    * Point at repo.vates.tech for CentOS since mirrorlist.centos.org was cut
    * Add "(EOL)" to repo descriptions for EOL repos
    * Drop unused repos

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" microcode_ctl xcp-ng-release --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.13.5-9.44.1.xcpng8.2
microcode_ctl: microcode_ctl-2.1-26.xs29.5.xcpng8.2
xcp-ng-release: xcp-ng-release-8.2.1-13

What to test

Normal use and anything else you want to test.

Test window before official release of the update

~ 1 day because of security updates.

gskger

@bleader Update worked well on my two node homelab and everything looks and works normal after reboot. I did some basic stuff like VM and Storage migration, but nothing in depth. Let's see how things work out.

Andrew

@bleader I installed it on many 8.2 machines. On one I did get a warning:

  Cleanup    : xen-libs-4.13.5-9.40.3.xcpng8.2.x86_64                     14/14
warning: %posttrans(microcode_ctl-2:2.1-26.xs29.5.xcpng8.2.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package 2:microcode_ctl-2.1-26.xs29.5.xcpng8.2.x86_64
  Verifying  : xcp-ng-release-8.2.1-13.x86_64                              1/14

But it did not seem to have any effect. Nothing extra in the yum.log

bleader

@Andrew If I understand correctly that is ran after regenerating the initrd, but to be sure, can you check the date of your /boot/initrd-4.19.0+1.img on this host?

stormi

Here's what the posttrans scriptlet exactly executes:

    dracut -f /boot/initrd-4.19.0+1.img 4.19.0+1 
    udevadm trigger --attr-nomatch=driver && udevadm settle -t 30

bleader

So if the initrd has been regenerated, it's likely the second part that fails, otherwise it is dracut, which is more of an issue.

Andrew

@bleader @stormi Microcode was updated and initrd was rebuilt correctly and the system rebooted with the updates, so I don't know what the error was. Running the commands manually produces no errors. Maybe the system was busy and it was the shorter 30 second settle timeout (vs. the default 120 seconds), at this time it returns quickly.

The warning was only on one system (out of 5 identical pool hosts) but caused no actual issues.

bleader

Update published: https://xcp-ng.org/blog/2024/09/27/september-2024-security-updates/

Thank you for the tests!

gduperrey

New update candidates for you to test!

As you may know, we group non-urgent updates together for a collective release, in order not to cause unnecessary maintenance for our users.

The moment to release such a batch has come, so here they are, ready for user tests before the final release.

XAPI:
- Synced with XS82ECU1074
  - Enhancement: robustification of the command xe host-emergency-ha-disable
  - Correction of different issues:
    - Performing a hard shutdown of a VM may hang due to unnecessary RBCA permission checks. An icon (yellow triangle) may then be displayed on some management applications, indicating that the shutdown process did not complete successfully.
    - Canceling a hard shutdown of a hung VM fails because the cancel function only checks for proper shutdowns.
    - Migrating VMs from 8.2.1 to 8.3 with the xe vm-migrate command may fail with the error 'Failure: Unknown tag/contents'.
    - You may encounter a 500 error (internal server error) when trying to retrieve RRD measurements from a powered off virtual machine.
xsconsole:
- Synced with XS82ECU1074: Fix for a time-out when creating an iSCSI SR.
guest-templates-json:
- Add generic templates for Linux BIOS and UEFI
- Synced with XS82ECU1085:
  - Oracle 8 requires minimum 2 vCPUS.
  - Added template for Ubuntu 24.04
sm:
- Synced with XS82ECU1075
  - Updated multipath.conf for several SANs
  - Fix for CA-393194: Find the real PV in a VG before removing the VG.
blktap: Synced with XS82ECU1075: Improvements on coalesce performance.
curl: Backport fixes for several CVE: CVE-2024-2004, CVE-2024-2379, CVE-2024-2398, CVE-2024-2466, CVE-2024-6197, CVE-2024-7264
openssl: Update to version 1.0.2k-26 from CentOS 7 updates and backports of available CVE fixes from openssl upstream. Update from CentOS 7 Includes fixes for CVE-2021-3712, CVE-2022-2078 and CVE-2023-0286. Backport are fixing CVE-2019-1547, CVE-2019-1551 and CVE-2019-1563.
xs-openssl: Rebased on version 1.1.1k-12 from CentOS 8 Stream. Include fixes for CVE-2023-5678, CVE-2023-3446, CVE-2023-3817 and a proper fix for CVE-2020-25659.
zstd: Update to version 1.5.5 to avoid an extremely rare cases of corruption
xcp-ng-xapi-plugins: Enhance error reporting when a command run on a host fails.
xenserver-status-report: Update to latest version, synced with XS82ECU1058
python-defusedxml: Added as a new dependency of xenserver-status-report.
Alternate Drivers: Updated to newer versions.
- intel-i40e-alt: From version 2.22.20-3.1 to 2.22.20-5.1
- mellanox-mlnxen-alt: From version 5.9.0.5.5.0-1.1 to 5.9.0.5.5.0-1.2
- More information about drivers and current versions is on the drivers page: (https://github.com/xcp-ng/xcp/wiki/Drivers).
New alternate driver mlx4-modules-alt: To resolve some issues with CX3 cards and SR-IOV, we added an updated version 4.9-7.1.0.0-LTS of this driver.
kernel-alt: Backport of a fix to correct cooling fan rotation speed on some Lenovo servers. For more information, you can read this thread on the forum.

As a dependency of XOSTOR:

http-nbd-transfer:
- Try to open device and start HTTP server before notifying the user
- Install pyc and pyo files

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

blktap: 3.37.4-4.1.xcpng8.2
curl: 8.6.0-2.2.xcpng8.2
forkexecd: 1.18.3-12.1.xcpng8.2
gpumon: 0.18.0-20.1.xcpng8.2
guest-templates-json: 1.10.7-1.2.xcpng8.2
http-nbd-transfer: 1.4.0-1.xcpng8.2
message-switch: 1.23.2-19.1.xcpng8.2
ocaml-rrd-transport: 1.16.1-17.1.xcpng8.2
ocaml-rrdd-plugin: 1.9.1-17.1.xcpng8.2
ocaml-tapctl: 1.5.1-17.1.xcpng8.2
ocaml-xcp-idl: 1.96.7-6.1.xcpng8.2
ocaml-xen-api-client: 1.9.0-20.1.xcpng8.2
ocaml-xen-api-libs-transitional: 2.25.7-1.1.xcpng8.2
openssl: 1.0.2k-26.2.xcpng8.2
python-defusedxml: 0.7.1-1.xcpng8.2
rrd2csv: 1.2.6-17.1.xcpng8.2
rrdd-plugins: 1.10.9-14.1.xcpng8.2
sm: 2.30.8-13.1.xcpng8.2
sm-cli: 0.23.0-63.1.xcpng8.2
squeezed: 0.27.0-20.1.xcpng8.2
varstored-guard: 0.6.2-17.xcpng8.2
vhd-tool: 0.43.0-20.1.xcpng8.2
wsproxy: 1.12.0-21.xcpng8.2
xapi: 1.249.38-1.11.xcpng8.2
xapi-nbd: 1.11.0-19.1.xcpng8.2
xapi-storage: 11.19.0_sxm2-19.xcpng8.2
xapi-storage-script: 0.34.1-18.1.xcpng8.2
xcp-networkd: 0.56.2-17.xcpng8.2
xcp-rrdd: 1.33.4-6.1.xcpng8.2
xenopsd: 0.150.19-5.1.xcpng8.2
xenserver-status-report: 1.3.16-3.xcpng8.2
xcp-ng-xapi-plugins: 1.10.1-1.xcpng8.2
xs-opam-repo: 6.35.13-4.xcpng8.2
xs-openssl: 1.1.1k-12.3.xcpng8.2
xsconsole: 10.1.13.1-2.1.xcpng8.2
zstd: 1.5.5-1.el7

Optional packages:

kernel-alt: 4.19.265-2.xcpng8.2
Alternates drivers:
- intel-i40e-alt: 2.22.20-5.1.xcpng8.2
- mellanox-mlnxen-alt: 5.9_0.5.5.0-1.2.xcpng8.2

New optional package:

mlx4-modules-alt: 4.9_7.1.0.0-1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~1 week.

bleader

Updated my machine at home, no issue so far.

olivierlambert

The same

JeffBerntsen

This seems to be working well for me too.

Andrew

@gduperrey Single 8.2.1 hosts and pools updated. All running fine. VMs working, CR CBT/NBD backup working.

gduperrey

New security update candidates for XCP-ng 8.2 LTS (xen, microcode_ctl)

Two new XSAs were published on November 12th 2024.
Intel published a microcode update on the November 12th 2024.

NOTE: This was published soon after the previous testing updates, therefore, we did not publish the previous and everything will be published altogether.

XSA-463 an unprivileged guest making two quick accesses to the VGA memory can deadlock a host.
XSA-464 an unprivileged PVH guest may access sensitive information from the host, control domain or other guests.

SECURITY UPDATES

xen-*:
- Fix XSA-463 - Deadlock in x86 HVM standard VGA handling. A mistake in the locking of process of the "standard" VGA memory makes it possible for a guest to make 2 quick accesses and create a deadlock that will hang the host.
- Fix XSA-464 - libxl leaks data to PVH guests via ACPI tables. The ACPI tables for PVH guests initialization left the excess memory space with its previous content, which was then copied to the guest memory as it was, resulting in possible leak of sensitive information. This doesn't affect XCP-ng in its normal configuration, as only HVM and PV-in-PVH (not affected) guests are supported.
microcode_ctl:
- Latest Intel microcode update, published on November the 12th:
  - Security updates for INTEL-SA-01101
  - Security updates for INTEL-SA-01079
  - Updated security updates for INTEL-SA-01097
  - Updated security updates for INTEL-SA-01103
  - Multiple other updates for functional issues.

Test on XCP-ng 8.2

Install all the current update candidates: the ones we already made users test to prepare for the next train of updates, and the new ones which fix security vulnerabilities. We will release them all together.

yum clean metadata --enablerepo=xcp-ng-candidates,xcp-ng-testing
yum update  --enablerepo=xcp-ng-candidates,xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

Security updates:

xen: 4.13.5-9.45.1.xcpng8.2
microcode_ctl: 2.1-26.xs29.6.xcpng8.2

The rest of the packages is described in the previous update candidate announcement.

What to test

Normal use and anything else you want to test.

Test window before official release of the update

~ 2 days because of security updates.

stormi

We also need you to test updates for XCP-ng 8.3. I created a new thread dedicated to XCP-ng 8.3, and renamed this present thread to make it dedicated to XCP-ng 8.2.

Please ensure you follow the new thread if you plan to help on testing XCP-ng 8.3.

New thread: https://xcp-ng.org/forum/topic/9964/xcp-ng-8-3-updates-announcements-and-testing

Andrew

@gduperrey Installed and running on active pools.