XCP-ng 8.3 updates announcements and testing

stormi

This is a thread dedicated to testing update candidates for XCP-ng 8.3 before they are released to everyone.

We will announce it here everytime there is a new update candidate in our testing repositories, so that you can test them and give feedback before they are pushed to everyone through the updates repository.

Follow this thread

Use the bell on top of this thread to watch it, and make sure you enable email notifications in your forum settings. This way, you will be notified each time there's a new update candidate that needs feedback.

How to install the update candidate

This will be described in each announcement.

If a package breaks something, you can downgrade to the previous version:

yum downgrade package1 [package2 ...]

Then run any tests you find appropriate for the installed updates, and report here.

Most update candidates won't stay for long in the testing stage, so each update is to be tested as soon as possible.

What to test

The most important task is to make sure any update introduces no regressions. Test basic functionality related to the updated component, test that your setup is still functional. As a bonus, you can also test more complicated scenarios that involve the component.

If you can, when the update fixes a bug or security issue, try to reproduce before installing the update, then try to ensure the update does what it says it does.

If the update brings new features, it's good to test them too.

If you can only test parts of the above, it's still good. Just say so when you report here.

How to report

Say what and how you tested, and give the results, either positive or negative. When in doubt about your results, just ask!

Let's start

Now see you at the end of this thread, for any updates candidates currently being tested!

stormi

New security update candidates for XCP-ng 8.3 LTS (xen, intel-microcode)

Two new XSAs were published on November 12th 2024.
Intel published a microcode update on the November 12th 2024.

XSA-463 an unprivileged guest making two quick accesses to the VGA memory can deadlock a host.
XSA-464 an unprivileged PVH guest may access sensitive information from the host, control domain or other guests.

SECURITY UPDATES

xen-*:
- Fix XSA-463 - Deadlock in x86 HVM standard VGA handling. A mistake in the locking of process of the "standard" VGA memory makes it possible for a guest to make 2 quick accesses and create a deadlock that will hang the host.
- Fix XSA-464 - libxl leaks data to PVH guests via ACPI tables. The ACPI tables for PVH guests initialization left the excess memory space with its previous content, which was then copied to the guest memory as it was, resulting in possible leak of sensitive information. This doesn't affect XCP-ng in its normal configuration, as only HVM and PV-in-PVH (not affected) guests are supported.
intel-microcode:
- Latest Intel microcode update, published on November the 12th:
  - Security updates for INTEL-SA-01101
  - Security updates for INTEL-SA-01079
  - Updated security updates for INTEL-SA-01097
  - Updated security updates for INTEL-SA-01103
  - Multiple other updates for functional issues.

Other updates

XO Lite: updated to version 0.5.0, fixing its loading without internet access and bringing some other improvements. Changelog: https://github.com/vatesfr/xen-orchestra/blob/xo-lite-v0.5.0/%40xen-orchestra/lite/CHANGELOG.md

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-candidates
yum update  --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

Security updates:

xen: 4.17.5-4.xcpng8.3
intel-microcode: 20241016-1.xcpng8.3

Maintenance update:

xo-lite: 0.5.0-1.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the update

~ 2 day because of security updates.

flakpyro

@stormi Updated a test machine running only couple VMs. Everything installed fine and rebooted without issue.

Machine is:
Intel Xeon E-2336
SuperMicro board.
One VM happens to be windows based with an Nvidia GPU passed though to it running Blue Iris using the MSR fixed found elsewhere on these forums, fix continues to work with this version of Xen.

Andrew

@stormi Installed on several test and pre-production machines.

XCP-ng-JustGreat

Latest version 8.3 candidate updates installed and are working fine on three-host home lab pool. Received a couple of repo errors for a certain mirror, but yum tried another mirror and it completed successfully. After updates were applied, performed live migrations between hosts with no problems and updated a Windows 11 Version 24H2 VM to the November 2024 cumulative update without problems. (VM is currently running Citrix Tools 9.3.2 without issues.)

gduperrey

Update published: https://xcp-ng.org/blog/2024/11/15/november-2024-security-update-for-xcp-ng-8-3/

Thank you for the tests!

flakpyro

@gduperrey Installed on a 2 host AMD based test pool, as well as our 5 host Intel based production pool without issue using rolling pool update. Everything migrated, updated then migrated back perfectly.

Also installed on my home server without issue.

Greg_E

Has the kernel changed to a newer version, or still 4.xx.xx?

gduperrey

@Greg_E No, the kernel version has not changed.
There is no kernel update in this update series either.

dxym

The blog(https://xcp-ng.org/blog/2024/11/15/november-2024-security-update-for-xcp-ng-8-3/) states the following:

Host reboots are necessary after this update.

However, the command output indicates:

# needs-restarting -r
No core libraries or services have been updated.
Reboot is probably not necessary.

Which one is correct?
It might be better to reboot the host, but not everyone checks the blog regularly.

gduperrey

@dxym It is always important to follow the instructions given on the forum or on the blog. In both cases, we indicate that the hosts must be restarted.
This way, we are sure that the hosts will apply the changes coming from the updates, like here changes on Xen and the Intel microcode.

stormi

needs-restarting is a tool from CentOS, which is not aware of the reality of XCP-ng. It's not even able to detect that a Xen or a microcode update requires a reboot. So, as Gaël says.

gduperrey

New update candidates for you to test!

A new batch of non-urgent updates is ready for user tests before a future collective release. Below are the details about these.

amd-microcode: Update AMD microcode to the 2024-11-21 drop
- Updates firmware for families 17h and 19h CPUs. For the first time, AMD published updates for non-server CPUs. One can assume that they started supporting microcode update for these, contrarily to what they did in the past, and that these updates thus fix various bugs and vulnerabilities. This is only (sensible) speculation at the moment, though.
grub: Backport VLAN networking support for UEFI PXE boot.
iperf3: Upgrade to version 3.9-13 from CentOS 7
- Includes a security fix for CVE-2023-38403
kernel: Backport of a fix to correct cooling fan rotation speed on some Lenovo servers. For more information, you can read this thread on the forum.
kexec-tools: Backport of a patch removing kernel_version(). Fixing a bug for kernel with a patchlevel greater than 255.
netdata: Fixed an issue that could occur when quickly uninstalling the package, right after an unfinished installation, and leave a service in an undetermined status.
slang: Fixed display and input issues in optional package mc.
sm: Contains a fix for leaf coalesce where the size of the leaf to coalesce was wrongly computed before deciding if it was a live coalesce or not, it resulted in some leaf having too much data to coalesce not successing the live coalesce and staying in this state indefinitely.
xapi:
- Fixed a malfunction related to the absence of a certificate, which could cause a loop.
- Fixed and improved various points in IPv6, related to management, reboot and re-initialization.
xo-lite: Update to version 0.6.0. For more details, you can consult the blog post on the latest release of Xen Orchestra.

Optional packages:

kernel-alt: Backport of a fix to correct cooling fan rotation speed on some Lenovo servers. For more information, you can read this thread on the forum.
socat: Update the package to version 1.7.4.1 which includes a fix for a buffer overflow and security fixes.
traceroute: Updated to version 2.1.5.
Alternate Drivers: Updated to newer versions.
- broadcom-bnxt-en-alt: From version 1.10.2_227.0.130.0 to 1.10.3_231.0.162.0
- intel-i40e-alt: From version 2.22.20-3.1 to 2.26.8
- More information about drivers and current versions is on the drivers page: (https://github.com/xcp-ng/xcp/wiki/Drivers).

Test on XCP-ng 8.3

From an up-to-date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

amd-microcode: 20240503-1.1.xcpng8.3
grub: 2.06-4.0.2.1.xcpng8.3
iperf3: 3.9-13.xcpng8.3
kernel: 4.19.19-8.0.37.1.xcpng8.3
kexec-tools: 2.0.15-20.1.xcpng8.3
netdata: 1.44.3-1.2.xcpng8.3
slang: 2.3.2-11.xcpng8.3
sm: 3.2.3-1.13.xcpng8.3
xapi: 24.19.2-1.9.xcpng8.3
xo-lite: 0.6.0-1.xcpng8.3

Optional packages:

kernel-alt: 4.19.322+1-1.xcpng8.3
socat: 1.7.4.1-6.xcpng8.3
traceroute: 2.1.5-2.xcpng8.3
Alternate drivers:
- broadcom-bnxt-en-alt: 1.10.3_231.0.162.0-1.xcpng8.3
- intel-i40e-alt: 2.26.8-1.xcpng8.3

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

flakpyro

@gduperrey installed on 2 test machines

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Both machines installed fine and all VMs came up without issue after.

I ran a backup job after to test snapshot coalesce, no issues there.

ravenet

@gduperrey

Tested on Multiple systems. Ryzen 1700x andThreadripper 5975. fine so far

Andrew

@gduperrey I have several hosts updated and running. I'm happy to see 8.3 updates on parity with 8.2.

ravenet

@gduperrey
Tested on 4 systems in production

Ryzen 1700x, on asrock rack mb w radeon pro GPU pass through
Threadripper 5975wx on asrock rack mb w radeon pro GPU pass through
Epyc 9224 on Asus
Epyc 7313P on Asus

XCP-ng-JustGreat

Applied latest candidate test updates to 3 x Dell OptiPlex 7040 (i7-6700, 48GB, 10Gbps-attached TrueNAS shared-storage) pool. Update process was error-free and successful. Everything appears to be working normally following the update.

gskger

@gduperrey Update some Dell R720s with GPUs and a Dell R730. Update worked without any problem and VMs operate as expected. Will update this post if that changes during day-to-day operation. Great work!

gduperrey

New update candidates for you to test!

In addition to the previous updates, available for testing, here are some new, non-urgent ones, expected to be released in a few days. Below are the details about these.

blktap: Add an option to use backup footer when vhd-util query is called. This will be used in a future storage driver.
intel-igc: Update to version 5.10.226.
xcp-ng-deps: Added vim-minimal as a dependency, so it is always present on XCP-ng systems.

For XOSTOR users:

drbd: Prevent a dead-lock in some situations, plus other improvements

(Reminder: XOSTOR is still in beta stage on XCP-ng 8.3)

Test on XCP-ng 8.3

From an up-to-date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

blktap: 3.54.9-1.2.xcpng8.3
intel-igc: 5.10.226-1.xcpng8.3
kmod-drbd: 9.2.11-1.1.xcpng8.3
xcp-ng-deps: 8.3-13

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~ 2 days