XCP-ng 8.2 updates announcements and testing

Andrew

@bleader Updated an running on newer and older Intel machines. Running normally so far.

olivierlambert

Tested in my own lab, no issues so far

bleader

Update published https://xcp-ng.org/blog/2024/07/18/july-2024-security-updates/

Thank you everyone for your tests!

manilx

@bleader 2 production polls updated via RPU without issues.

bleader

New security update candidates (xen, microcode_ctl)

A new XSA was published on August 14th 2024.
Intel published an updated microcode on August 13th 2024.

XSA-460 passing through some PCI devices after guest creation could lead to any kind of security issue (DoS, privilege escalation, information leak,…) the actual risk depends on the systems and the devices passed through.
XSA-461 some pass-through use cases when on one or more device when they share resources are actually not possible to be made secure. This XSA updates the Xen documentation to reflect that.

SECURITY UPDATES

xen-*:
- Fix XSA-460 - error handling in x86 IOMMU identity mapping. The handling of errors in the case of mapping Reserved Memory Regions was flawed, potentially keeping the mapping in place, this could allow guests to access memory region they should not be allowed to access. These mapping are typically used when doing pass-through of legacy USB emulation.
- Document XSA-461 - PCI device pass-through with shared resources Files. According to the Xen Project security team, PCI pass-through of devices that share ressources is not possible to be made safe. This updates the documentation to reflect it. Look at the "MITIGATION" section of the XSA for more information regarding the safe cases.
microcode_ctl: Security updates from Intel:
- INTEL-SA-01083
- INTEL-SA-01118
- INTEL-SA-01100
- INTEL-SA-01038
- INTEL-SA-01046
- Plus fixes for a lot of functional issues, see the Release Notes for more details.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" microcode_ctl --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.13.5-9.40.3.xcpng8.2
microcode_ctl: 2.1-26.xs29.3.xcpng8.2

What to test

Normal use and anything else you want to test. Feedback about PCI pass-through is also welcome.

Test window before official release of the update

~ 2 day because of security updates.

Andrew

@bleader Installed and running on pools of 8.2.1 systems.

JeffBerntsen

@bleader They're installed and running well on my test lab systems.

gskger

@bleader Installed on my playlab. Everything looks normal, let's see how it goes.

stormi

I just pushed an update for microcode_ctl, in xcp-ng-candidates again.

It only changes one microcode file, 06-a5-03 (CML-S62, Core Gen 10), which Intel had forgotten to update in their initial release.

(We found out and reported it to them)

bleader

Update published https://xcp-ng.org/blog/2024/08/16/august-security-update/

Thank you all for testing

manilx

@bleader All pools updated via RPU without issues.

bleader

New security update candidates (xen, microcode_ctl)

A new XSA was published on September 24th 2024.
Intel published a microcode update on the September 10th 2024.
We also included an updated xcp-ng-release for testing, althrough not related to security.

XSA-462 a malicious HVM or PVH guest can trigger a DoS of the host.

SECURITY UPDATES

xen-*:
* Fix XSA-462 - x86: Deadlock in vlapic_error(). The handling of x86's APIC (Advanced Programmable Interrupt Controller) allows a guest to configure an illegal vector to handle error interrupts. This causes the vlapic_error() to recurse, this is protected, but the lock used for this protection will try to be taken recursiveley, leading to a deadlock.
microcode_ctl:
    * Latest Intel microcode update, still named IPU 2024.3, including security updates for:
        * INTEL-SA-01103
        * INTEL-SA-01097

Other updates

xcp-ng-release:
    * Update the "XOA Quick deploy" feature in the host's web page.
    * Point at repo.vates.tech for CentOS since mirrorlist.centos.org was cut
    * Add "(EOL)" to repo descriptions for EOL repos
    * Drop unused repos

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" microcode_ctl xcp-ng-release --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.13.5-9.44.1.xcpng8.2
microcode_ctl: microcode_ctl-2.1-26.xs29.5.xcpng8.2
xcp-ng-release: xcp-ng-release-8.2.1-13

What to test

Normal use and anything else you want to test.

Test window before official release of the update

~ 1 day because of security updates.

gskger

@bleader Update worked well on my two node homelab and everything looks and works normal after reboot. I did some basic stuff like VM and Storage migration, but nothing in depth. Let's see how things work out.

Andrew

@bleader I installed it on many 8.2 machines. On one I did get a warning:

  Cleanup    : xen-libs-4.13.5-9.40.3.xcpng8.2.x86_64                     14/14
warning: %posttrans(microcode_ctl-2:2.1-26.xs29.5.xcpng8.2.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package 2:microcode_ctl-2.1-26.xs29.5.xcpng8.2.x86_64
  Verifying  : xcp-ng-release-8.2.1-13.x86_64                              1/14

But it did not seem to have any effect. Nothing extra in the yum.log

bleader

@Andrew If I understand correctly that is ran after regenerating the initrd, but to be sure, can you check the date of your /boot/initrd-4.19.0+1.img on this host?

stormi

Here's what the posttrans scriptlet exactly executes:

    dracut -f /boot/initrd-4.19.0+1.img 4.19.0+1 
    udevadm trigger --attr-nomatch=driver && udevadm settle -t 30

bleader

So if the initrd has been regenerated, it's likely the second part that fails, otherwise it is dracut, which is more of an issue.

Andrew

@bleader @stormi Microcode was updated and initrd was rebuilt correctly and the system rebooted with the updates, so I don't know what the error was. Running the commands manually produces no errors. Maybe the system was busy and it was the shorter 30 second settle timeout (vs. the default 120 seconds), at this time it returns quickly.

The warning was only on one system (out of 5 identical pool hosts) but caused no actual issues.

bleader

Update published: https://xcp-ng.org/blog/2024/09/27/september-2024-security-updates/

Thank you for the tests!

gduperrey

New update candidates for you to test!

As you may know, we group non-urgent updates together for a collective release, in order not to cause unnecessary maintenance for our users.

The moment to release such a batch has come, so here they are, ready for user tests before the final release.

XAPI:
- Synced with XS82ECU1074
  - Enhancement: robustification of the command xe host-emergency-ha-disable
  - Correction of different issues:
    - Performing a hard shutdown of a VM may hang due to unnecessary RBCA permission checks. An icon (yellow triangle) may then be displayed on some management applications, indicating that the shutdown process did not complete successfully.
    - Canceling a hard shutdown of a hung VM fails because the cancel function only checks for proper shutdowns.
    - Migrating VMs from 8.2.1 to 8.3 with the xe vm-migrate command may fail with the error 'Failure: Unknown tag/contents'.
    - You may encounter a 500 error (internal server error) when trying to retrieve RRD measurements from a powered off virtual machine.
xsconsole:
- Synced with XS82ECU1074: Fix for a time-out when creating an iSCSI SR.
guest-templates-json:
- Add generic templates for Linux BIOS and UEFI
- Synced with XS82ECU1085:
  - Oracle 8 requires minimum 2 vCPUS.
  - Added template for Ubuntu 24.04
sm:
- Synced with XS82ECU1075
  - Updated multipath.conf for several SANs
  - Fix for CA-393194: Find the real PV in a VG before removing the VG.
blktap: Synced with XS82ECU1075: Improvements on coalesce performance.
curl: Backport fixes for several CVE: CVE-2024-2004, CVE-2024-2379, CVE-2024-2398, CVE-2024-2466, CVE-2024-6197, CVE-2024-7264
openssl: Update to version 1.0.2k-26 from CentOS 7 updates and backports of available CVE fixes from openssl upstream. Update from CentOS 7 Includes fixes for CVE-2021-3712, CVE-2022-2078 and CVE-2023-0286. Backport are fixing CVE-2019-1547, CVE-2019-1551 and CVE-2019-1563.
xs-openssl: Rebased on version 1.1.1k-12 from CentOS 8 Stream. Include fixes for CVE-2023-5678, CVE-2023-3446, CVE-2023-3817 and a proper fix for CVE-2020-25659.
zstd: Update to version 1.5.5 to avoid an extremely rare cases of corruption
xcp-ng-xapi-plugins: Enhance error reporting when a command run on a host fails.
xenserver-status-report: Update to latest version, synced with XS82ECU1058
python-defusedxml: Added as a new dependency of xenserver-status-report.
Alternate Drivers: Updated to newer versions.
- intel-i40e-alt: From version 2.22.20-3.1 to 2.22.20-5.1
- mellanox-mlnxen-alt: From version 5.9.0.5.5.0-1.1 to 5.9.0.5.5.0-1.2
- More information about drivers and current versions is on the drivers page: (https://github.com/xcp-ng/xcp/wiki/Drivers).
New alternate driver mlx4-modules-alt: To resolve some issues with CX3 cards and SR-IOV, we added an updated version 4.9-7.1.0.0-LTS of this driver.
kernel-alt: Backport of a fix to correct cooling fan rotation speed on some Lenovo servers. For more information, you can read this thread on the forum.

As a dependency of XOSTOR:

http-nbd-transfer:
- Try to open device and start HTTP server before notifying the user
- Install pyc and pyo files

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

blktap: 3.37.4-4.1.xcpng8.2
curl: 8.6.0-2.2.xcpng8.2
forkexecd: 1.18.3-12.1.xcpng8.2
gpumon: 0.18.0-20.1.xcpng8.2
guest-templates-json: 1.10.7-1.2.xcpng8.2
http-nbd-transfer: 1.4.0-1.xcpng8.2
message-switch: 1.23.2-19.1.xcpng8.2
ocaml-rrd-transport: 1.16.1-17.1.xcpng8.2
ocaml-rrdd-plugin: 1.9.1-17.1.xcpng8.2
ocaml-tapctl: 1.5.1-17.1.xcpng8.2
ocaml-xcp-idl: 1.96.7-6.1.xcpng8.2
ocaml-xen-api-client: 1.9.0-20.1.xcpng8.2
ocaml-xen-api-libs-transitional: 2.25.7-1.1.xcpng8.2
openssl: 1.0.2k-26.2.xcpng8.2
python-defusedxml: 0.7.1-1.xcpng8.2
rrd2csv: 1.2.6-17.1.xcpng8.2
rrdd-plugins: 1.10.9-14.1.xcpng8.2
sm: 2.30.8-13.1.xcpng8.2
sm-cli: 0.23.0-63.1.xcpng8.2
squeezed: 0.27.0-20.1.xcpng8.2
varstored-guard: 0.6.2-17.xcpng8.2
vhd-tool: 0.43.0-20.1.xcpng8.2
wsproxy: 1.12.0-21.xcpng8.2
xapi: 1.249.38-1.11.xcpng8.2
xapi-nbd: 1.11.0-19.1.xcpng8.2
xapi-storage: 11.19.0_sxm2-19.xcpng8.2
xapi-storage-script: 0.34.1-18.1.xcpng8.2
xcp-networkd: 0.56.2-17.xcpng8.2
xcp-rrdd: 1.33.4-6.1.xcpng8.2
xenopsd: 0.150.19-5.1.xcpng8.2
xenserver-status-report: 1.3.16-3.xcpng8.2
xcp-ng-xapi-plugins: 1.10.1-1.xcpng8.2
xs-opam-repo: 6.35.13-4.xcpng8.2
xs-openssl: 1.1.1k-12.3.xcpng8.2
xsconsole: 10.1.13.1-2.1.xcpng8.2
zstd: 1.5.5-1.el7

Optional packages:

kernel-alt: 4.19.265-2.xcpng8.2
Alternates drivers:
- intel-i40e-alt: 2.22.20-5.1.xcpng8.2
- mellanox-mlnxen-alt: 5.9_0.5.5.0-1.2.xcpng8.2

New optional package:

mlx4-modules-alt: 4.9_7.1.0.0-1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~1 week.