Updates announcements and testing

stormi

New update candidates for you to test!

As you may know, we group non-urgent updates together for a collective release, in order not to cause unnecessary maintenance for our users.

The moment to release such a batch has come, so here they are, ready for user tests before the final release.

xcp-ng-release*:
- Updated web page on hosts to remove dependencies to Fontawesome Pro and Jquery.
- The XOA quick deploy script now uses HTTPS by default.
- Updated repository definitions in /etc/yum.repos.d/xcpng.repo, to add more testing repositories (disabled by default. More about this below). Warning: If you have any local changes to this file, it won't be overwritten. In this case, look for /etc/yum.repos.d/xcpng.repo.rpmnew after applying the update, and move it over xcpng.repo.
xen-*: sync with Citrix Hypervisor hotfix XS82ECU1030:
- Hardware support fixes, among which "Cope booting for x2APIC mode on AMD systems without XT mode."
- Improve loading of AMD microcode on all logical processors.
- (The hotfix from Citrix Hypervisor also includes fixes for the latest Xen Security Advisories, which we already published in a previous update)
AMD microcode (linux-firmware) and Intel microcode (microcode_ctl). AMD and Intel did not detail what they fix, but everyone is supposed to update. This is the frustrating situation with binary blobs in firmware.
XAPI and related components:
- Instead of a 403 error on HTTP requests to the host's web page, redirect to HTTPS instead.
- Fix spurious "not enough memory" error message in /var/log/xcp-rrdd-plugins.log.
- Sync with Citrix Hypervisor hotfix XS82ECU1027: various fixes.
qemu: sync with Citrix Hypervisor hotfix XS82ECU1031. Fixes for specific issues.
sm (Storage Manager): sync with Citrix hypervisor hotfix XS82ECU1022. Various fixes.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing forkexecd gpumon linux-firmware message-switch microcode_ctl qemu rrdd-plugins sm sm-rawhba varstored-guard xapi-core xapi-tests xapi-xe xcp-networkd xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xcp-rrdd xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools xenopsd xenopsd-cli xenopsd-xc
reboot

The usual update rules apply: pool coordinator first, etc.

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

About the new testing repositories

Until recently, we would just have one testing repository: xcp-ng-testing. We decided to split it.

It had a lot of different uses:

Making updates available to testers for them to provide feedback, before pushing them to everyone. This use will remain and will now be the only role of this repository.
Storing updates to components we don't intend to push as official updates. For example newer zstd or GlusterFS releases. These now live in a new repository: xcp-ng-lab.
Providing temporary builds just to test a patch, before embedding it in a real update if tests are successful. There are several places where we can make them available to you when needed, depending on the situation: per-person developer repositories, scratch builds in koji, or . We'll tell you where to pull from each time we need you to test.

We also added two new repositories for our internal needs. You usually won't need to pull from them, even for tests: xcp-ng-incoming and xcp-ng-ci. xcp-ng-incoming is where we build updates first. When a consistent set of changes is ready, it moves to xcp-ng-ci and undergoes automated testing. Once the tests pass,
updates move to xcp-ng-testing for you to test.

Shortly before publishing to everyone, updates will be moved the new xcp-ng-candidates. Why are there both xcp-ng-testing and xcp-ng-candidates? Because not all updates move on at the same pace. Some can wait for weeks before we publish them in what we call internally "an update train", because we group non-urgent updates together. Some need to be published as soon as possible, notably security fixes. So while there may already be updates in xcp-ng-testing, sometimes we need to build, test and publish updates directly without any interferences from what's currently in xcp-ng-testing. What it means for you as testers is that sometimes we'll ask you to pull update candidates from xcp-ng-testing, sometimes from xcp-ng-candidates. In any case we'll always specify it in our testing instructions.

Test window before official release of the updates

~1 week.

stormi

We had some feedback on 8.3, but I'm also counting on you for XCP-ng 8.2

Andrew

@stormi I updated active 8.2.1 servers and it's running normally (24 hours), HP G8 and current 11th Gen i7. I updated other machines (older AMD and Intel) and they are ok too, but just used for testing. Normal update/reboot worked fine.

Active servers run: Windows, Linux (many versions), FreeBSD, hot migrations, CR, Delta S3 backup, NFS SR/ISO, VxLAN, etc...

JeffBerntsen

@stormi So far so good in my test lab and one minor production server.

Danp

Looks like these updates were released earlier today -- https://xcp-ng.org/blog/2023/05/26/may-2023-maintenance-update/

gduperrey

New update candidates for you to test!

Shortly after we released the previous batch of non-urgent updates, XenServer released several updates for Citrix Hypervisor 8.2 CU1. We prepared new update candidates based on these, as well as a specific update of xcp-ng-xapi-plugins.

There's no date for their release yet, but they're ready for your tests and feedback already.

xcp-ng-xapi-plugins: the updater plugin, used by Xen Orchestra to apply updates, can now also install new packages (this will be used to deploy XOSTOR from Xen Orchestra).
kernel: as explained in the hotfix from XenServer XS82ECU1028 "ACPI processor-related data is being reported incorrectly to the hypervisor, affecting Intel - Xeon 84xx/64xx/54xx/44xx/34xx - Sapphire Rapids and possibly other models."
grub: bugfix
lldpad:
- The FCoE service can have a memory leak that could use up dom0 memory
- A resource leak in the FCoE service can crash the service
- When trying to create an LACP bond using Cisco Nexus switches, host could have intermittent connection problems
- XenServer hotfix: XS82ECU1032
xen: Correct a flaw for VMx under Red Hat Enterprise 7 (and derivatives) with a large number of CPUs, that can cause migration failures when trying to migrate to AMD hosts.
- XenServer hotfix: XS82ECU1034

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing xen-* grub* lldpad kernel xcp-ng-xapi-plugins
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

kernel-4.19.19-7.0.16.1.xcpng8.2
grub-2.02-3.2.0.xcpng8.2
lldpad-1.0.1-10.xcpng8.2
xen-*4.13.5-9.32.1.xcpng8.2
xcp-ng-xapi-plugins-1.8.0-1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

Andrew

@gduperrey I've been running the update on all my active 8.2 machines. I did not run into any of the bugs before. After the update everything is still normal.

JeffBerntsen

@gduperrey I'm running the updates on both lab and production machines and all seems well so far.

NielsH

Yesterday Zenbleed (https://news.ycombinator.com/item?id=36848680) was announced. It mentions there that AMD may have already released microcode patches to fix this.
Is this perhaps the mysterious microcode update form may 16th?

AMD microcode (linux-firmware) and Intel microcode (microcode_ctl). AMD and Intel did not detail what they fix, but everyone is supposed to update. This is the frustrating situation with binary blobs in firmware.

Or can we expect another update to resolve the zenbleed vulnerability in the coming days?

gduperrey

We began to work on the patch yesterday evening. We will publish it for testers later today, and if everything is fine, for everyone after two days (and success in our tests, of course).

gduperrey

New Security Update Candidates (Xen and AMD CPUs)

Xen is being updated to mitigate hardware vulnerabilities in AMD CPUs.

Upstream (Xen project) advisory: XSA-433
- Citrix Hypervisor Security Update for CVE-2023-20593

This issue affects systems running AMD Zen 2 CPUs. Under specific microarchitectural circumstances, it may allow an attacker to potentially access sensitive information.

Components are also updated to add bugfixes and enhancements:

Xen:
- Now, MPX feature is disabled by default. Cross-pool migration and upgrade will be simplified as VMs can migrate more easily from pools with Intel SkyLake, CascadeLake, or CooperLake hardware to pools with later Intel hardware (such as IceLake).
  A reboot is necessary after updating to benefit from this feature.
- Improvements to latency with a limit on the scheduler loadbalancing. This improves performance on large systems with high CPU utilization.

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" linux-firmware --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.5-9.34.1.xcpng8.2
linux-firmware: 20190314-8.1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days.

Andrew

@gduperrey Updated multiple servers and seems to be fine. But, none are Zen2 systems.

gduperrey

Update published. Thanks for the tests!

https://xcp-ng.org/blog/2023/07/27/july-2023-security-update-zenbleed/

TodorPetkov

Hello,

I saw there is new announcement on Xenbits regarding Zenbleed (https://xenbits.xen.org/xsa/advisory-433.html) - will there be new patch for XCP?

Thanks in advance.

bleader

@TodorPetkov Yes, for now we do not know when this update will be released on XenServer side yet, but it will be published on XCP-ng side too.

What was released for now is suffering from the same issue as described in your link.

If I'm not mistaken:

the linux-firmware update fixes the issues with zenbleed
the kernel patch is working around the case where the updated firmware is not used by disabling features via the control register, and there were too much disabled in the previous patch.
if you're using the updated firmware, this workaround will not be used, and therefore the updated patch is not critical.

You can check you're running the right microcode version via:

journalctl -k --grep=microcode

Without the -k you should be able to see previous boots and ensure the patch_level= has changed. I'm unsure which version to expect there as we do not have zen2 at hand for testing this.

We will indeed provide an update later, likely not in a dedicated update, but with other fixes.

I hope that answers properly your question!

gduperrey

New Security Update Candidates (Xen)

Xen is being updated to correct a flaw in the latest patch (XSA-433) for Zenbleed and AMD CPUs.

Upstream (Xen project) advisory: XSA-433

The patch provided with earlier versions was buggy by unintentionally disabling more bits than expected in the control register due to bad integer variable truncation.

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" --enablerepo=xcp-ng-testing
reboot

Version:

xen-*: 4.13.5-9.35.1.xcpng8.2

If you didn't already applied the previous updates, we invite you to also update linux-firmware.

yum update linux-firmware
reboot

Version:

linux-firmware: 20190314-8.1.xcpng8.2

One reboot for the two updates is enough.

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~1 days. We'll release before the WE if our internal tests are fine.

gduperrey

Update published. Thanks for the tests!

https://xcp-ng.org/blog/2023/08/04/erratum-july-2023-security-update-zenbleed/

NielsH

@gduperrey Thanks!

I see that shortly after this some more security related patched have become available:
XSA-432: https://www.openwall.com/lists/oss-security/2023/08/08/3
XSA-434: https://www.openwall.com/lists/oss-security/2023/08/08/4
XSA-435: https://www.openwall.com/lists/oss-security/2023/08/08/5

Some of this also seems pretty serious. We were currently busy running patches on our systems. This takes us about 1-2 weeks to complete. Do you think these 3 patches will also become available for XCP-ng? In that case we'd rather wait for those to become available so we don't have to spend another 1-2 weeks immediately after completing the current set of patches.

Thank you!

gduperrey

Hello,

Yes, these patches will become available in XCP-ng. We're working on it to release as soon as possible. We'd like to release them this week, so we do everything we can for that.

There will be a post here for the tests and for the final release.

olivierlambert

Hello @NielsH I don't want to sound moralistic, but if you are using XCP-ng in production without any subscription, and being worried about patches coming fast enough for your production system, you should really think about getting support for it That's exactly what the subscription money is made for! (well, in part but absolutely used for that).

I mean, you are free to not doing it, and even if we do our best to treat everyone fairly (paying or not) for our patches, we won't be against more support so the project can continue to grow