Updates announcements and testing

gduperrey

New Security Update Candidates (Xen)

Xen is being updated to mitigate some vulnerabilities:

XSA-427: "Guests running in shadow mode and being subject to migration or snapshotting may be able to cause Denial of Service and other problems, including escalation of privilege". This vulnerability concerns old platforms (Nehalem/Bulldozer families and older) which do not have Hardware Assisted Paging facilitie (EPT/NPT), or modern platforms where this extension is disabled by the firmware or the system software. This also concerns PV guests, which are not officially supported anymore in XCP-ng.
XSA-428: "Entities controlling HVM guests can run the host out of resources or stall execution of a physical CPU for effectively unbounded periods of time, resulting in a Denial of Servis (DoS) affecting the entire host. Crashes, information leaks, or elevation of privilege cannot be ruled out".
On the platforms managed by XCP-ng software, with regard of this vulnerability, we would rather talk of "reduction in defence in depth", as the only entity controlling HVM guests is a trusted software (QEMU) running in a trusted domain (dom0).
XSA-429: The patch completes the original Spectre/Meltdown mitigation work(XSA-254). A malicious PV guest might be able to infer the contents of arbitrary host memory, including memory assigned to other guests. Only AMD and Hygon CPUs which offer SMEP/SMAP facilities are affected. Although PV guests are not officially supported in XCP-ng, we also included a fix for this vulnerability.

Components are also updated to add bugfixes and enhancements:

Xen
- Update to Xen 4.13.5
- Initial Sapphire Rapids support
- Fix memory corruption issues in the Ocaml bindings.
- On xenstored live update, validate the config file before launching into the new xenstored

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.5-9.30.3.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days.

Andrew

@gduperrey Installed on 8.2 systems and running ok on home lab and other secondary machines. No issues before or after update. No Sapphire Rapids CPUs. Ran same update on 8.3 by mistake on a test machine but it's running ok too.

maxcuttins

@stormi

what is the kernel version of the latest XCP-ng?

stormi

Looks like some of the testers who used to test the update candidates moved their test hosts to the 8.3 alpha release. Thanks @Andrew for staying true to the job

stormi

@maxcuttins Is the question related to the testing of update candidates?

JeffBerntsen

Latest updates seem to be working fine in my test lab as well.

gduperrey

Update published. Thanks for the tests!

https://xcp-ng.org/blog/2023/03/23/march-2023-security-update/

maxcuttins

@stormi not really I would say.
I'm stick to version 8.0 and I'm planning to upgrade.
So I'm trying to understand which kernel I'll find in the next release.

PS: Upgrade is a pain because I need to remember that CEPH NBD share storage are not preserved during upgrade and so, those config file will be erased, I'll need to restore in order to have back my VDIs.

stormi

@maxcuttins Ok. Then let's discuss this in another thread and leave the current one for testing update candidates.

Andrew

@gduperrey XO (current source) rolling pool update did its job.

stormi

Hello here! I hope you are ready, because we'll have a train of update candidates for you to test shortly

stormi

New update candidates for you to test!

As you may know, we group non-urgent updates together for a collective release, in order not to cause unnecessary maintenance for our users.

The moment to release such a batch has come, so here they are, ready for user tests before the final release.

xcp-ng-release*:
- Updated web page on hosts to remove dependencies to Fontawesome Pro and Jquery.
- The XOA quick deploy script now uses HTTPS by default.
- Updated repository definitions in /etc/yum.repos.d/xcpng.repo, to add more testing repositories (disabled by default. More about this below). Warning: If you have any local changes to this file, it won't be overwritten. In this case, look for /etc/yum.repos.d/xcpng.repo.rpmnew after applying the update, and move it over xcpng.repo.
xen-*: sync with Citrix Hypervisor hotfix XS82ECU1030:
- Hardware support fixes, among which "Cope booting for x2APIC mode on AMD systems without XT mode."
- Improve loading of AMD microcode on all logical processors.
- (The hotfix from Citrix Hypervisor also includes fixes for the latest Xen Security Advisories, which we already published in a previous update)
AMD microcode (linux-firmware) and Intel microcode (microcode_ctl). AMD and Intel did not detail what they fix, but everyone is supposed to update. This is the frustrating situation with binary blobs in firmware.
XAPI and related components:
- Instead of a 403 error on HTTP requests to the host's web page, redirect to HTTPS instead.
- Fix spurious "not enough memory" error message in /var/log/xcp-rrdd-plugins.log.
- Sync with Citrix Hypervisor hotfix XS82ECU1027: various fixes.
qemu: sync with Citrix Hypervisor hotfix XS82ECU1031. Fixes for specific issues.
sm (Storage Manager): sync with Citrix hypervisor hotfix XS82ECU1022. Various fixes.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing forkexecd gpumon linux-firmware message-switch microcode_ctl qemu rrdd-plugins sm sm-rawhba varstored-guard xapi-core xapi-tests xapi-xe xcp-networkd xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xcp-rrdd xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenopsd xenopsd-cli xenopsd-xc
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

About the new testing repositories

Until recently, we would just have one testing repository: xcp-ng-testing. We decided to split it.

It had a lot of different uses:

Making updates available to testers for them to provide feedback, before pushing them to everyone. This use will remain and will now be the only role of this repository.
Storing updates to components we don't intend to push as official updates. For example newer zstd or GlusterFS releases. These now live in a new repository: xcp-ng-lab.
Providing temporary builds just to test a patch, before embedding it in a real update if tests are successful. There are several places where we can make them available to you when needed, depending on the situation: per-person developer repositories, scratch builds in koji, or . We'll tell you where to pull from each time we need you to test.

We also added two new repositories for our internal needs. You usually won't need to pull from them, even for tests: xcp-ng-incoming and xcp-ng-ci. xcp-ng-incoming is where we build updates first. When a consistent set of changes is ready, it moves to xcp-ng-ci and undergoes automated testing. Once the tests pass,
updates move to xcp-ng-testing for you to test.

Shortly before publishing to everyone, updates will be moved the new xcp-ng-candidates. Why are there both xcp-ng-testing and xcp-ng-candidates? Because not all updates move on at the same pace. Some can wait for weeks before we publish them in what we call internally "an update train", because we group non-urgent updates together. Some need to be published as soon as possible, notably security fixes. So while there may already be updates in xcp-ng-testing, sometimes we need to build, test and publish updates directly without any interferences from what's currently in xcp-ng-testing. What it means for you as testers is that sometimes we'll ask you to pull update candidates from xcp-ng-testing, sometimes from xcp-ng-candidates. In any case we'll always specify it in our testing instructions.

Test window before official release of the updates

~1 week.

stormi

We had some feedback on 8.3, but I'm also counting on you for XCP-ng 8.2

Andrew

@stormi I updated active 8.2.1 servers and it's running normally (24 hours), HP G8 and current 11th Gen i7. I updated other machines (older AMD and Intel) and they are ok too, but just used for testing. Normal update/reboot worked fine.

Active servers run: Windows, Linux (many versions), FreeBSD, hot migrations, CR, Delta S3 backup, NFS SR/ISO, VxLAN, etc...

JeffBerntsen

@stormi So far so good in my test lab and one minor production server.

Danp

Looks like these updates were released earlier today -- https://xcp-ng.org/blog/2023/05/26/may-2023-maintenance-update/

gduperrey

New update candidates for you to test!

Shortly after we released the previous batch of non-urgent updates, XenServer released several updates for Citrix Hypervisor 8.2 CU1. We prepared new update candidates based on these, as well as a specific update of xcp-ng-xapi-plugins.

There's no date for their release yet, but they're ready for your tests and feedback already.

xcp-ng-xapi-plugins: the updater plugin, used by Xen Orchestra to apply updates, can now also install new packages (this will be used to deploy XOSTOR from Xen Orchestra).
kernel: as explained in the hotfix from XenServer XS82ECU1028 "ACPI processor-related data is being reported incorrectly to the hypervisor, affecting Intel - Xeon 84xx/64xx/54xx/44xx/34xx - Sapphire Rapids and possibly other models."
grub: bugfix
lldpad:
- The FCoE service can have a memory leak that could use up dom0 memory
- A resource leak in the FCoE service can crash the service
- When trying to create an LACP bond using Cisco Nexus switches, host could have intermittent connection problems
- XenServer hotfix: XS82ECU1032
xen: Correct a flaw for VMx under Red Hat Enterprise 7 (and derivatives) with a large number of CPUs, that can cause migration failures when trying to migrate to AMD hosts.
- XenServer hotfix: XS82ECU1034

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing xen-* grub* lldpad kernel xcp-ng-xapi-plugins
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

kernel-4.19.19-7.0.16.1.xcpng8.2
grub-2.02-3.2.0.xcpng8.2
lldpad-1.0.1-10.xcpng8.2
xen-*4.13.5-9.32.1.xcpng8.2
xcp-ng-xapi-plugins-1.8.0-1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

Andrew

@gduperrey I've been running the update on all my active 8.2 machines. I did not run into any of the bugs before. After the update everything is still normal.

JeffBerntsen

@gduperrey I'm running the updates on both lab and production machines and all seems well so far.

NielsH

Yesterday Zenbleed (https://news.ycombinator.com/item?id=36848680) was announced. It mentions there that AMD may have already released microcode patches to fix this.
Is this perhaps the mysterious microcode update form may 16th?

AMD microcode (linux-firmware) and Intel microcode (microcode_ctl). AMD and Intel did not detail what they fix, but everyone is supposed to update. This is the frustrating situation with binary blobs in firmware.

Or can we expect another update to resolve the zenbleed vulnerability in the coming days?