XCP-ng 8.2 updates announcements and testing

CJ

@NielsH Kind of off topic but figured I'd mention it as I only recently discovered this.

Not sure what VMs you're running, but if they can survive being off for a short time (redundancy of services or planned outage) you can reboot the host using Smart Reboot under the Advanced tab. While it incurs some downtime, it allows for a much faster reboot time than migrating the VMs to another server and back.

I use local storage as well and it's been a game changer for dealing with pool patches.

bleader

The update has been published, thanks for the feedback and tests.

https://xcp-ng.org/blog/2024/01/26/january-2024-security-update/

NielsH

@CJ said in Updates announcements and testing:

@NielsH Kind of off topic but figured I'd mention it as I only recently discovered this.

Not sure what VMs you're running, but if they can survive being off for a short time (redundancy of services or planned outage) you can reboot the host using Smart Reboot under the Advanced tab. While it incurs some downtime, it allows for a much faster reboot time than migrating the VMs to another server and back.

I use local storage as well and it's been a game changer for dealing with pool patches.

Cheers, thanks for the suggestion. In our case we actually are phashing out xcp-ng and are in the process of migrating to Proxmox since we can migrate with 30-35Gbit/s there. The disk performance is so much faster there we can perform all the updates in a single day instead of 2 weeks

Another issue we had was that VM migrations of very large VMs (usually 8cores+) are quite impactful. Because we want to use VMs with 24-48 cores and 128GB RAM as well it simply was not usable enough for us. There's several seconds, or sometimes even minutes of downtime during the last phase of the migration with the large VMs.

With Proxmox we have seen very little downtime (<1s) which we are very happy about.

bleader

New security update candidates (xen)

Two new XSAs were published on 30th of January.

XSA-449 impacts PCI passthrough users.
XSA-450 is only impacting the case where Xen is compiled without HVM support, that is not the case in XCP-ng. We therefore chose not to include this fix yet (will likely be included in future versions, maybe not part of a critical security update).

SECURITY UPDATES

xen-*:
* Fix XSA-449 - pci: phantom functions assigned to incorrect contexts. A malicious VM assigned with a PCI device could in some cases access data of a guest previously using the same PCI device. This requires PCI passthrough on a device using phantom functions and reassigning the same device to a new VM to be exploitable.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.13.5-9.38.2.xcpng8.2

What to test

Normal use and anything else you want to test, if you are using PCI passthrough devices that's even better, but we also would be glad to have confirmation from others that their normal use case still works as intended.

Test window before official release of the updates
2 day because of security updates.

JeffBerntsen

This seems to be working fine on my two test systems but I don't do PCI passthrough.

Andrew

@bleader I installed it on a bunch of busy hosts. All are fine, but none used PCI passthrough. The Rolling Pool Reboot in XO was very helpful.

bleader

The update has been published, thanks for testing.

https://xcp-ng.org/blog/2024/02/02/february-2024-security-update/

bleader

New security update candidate (xen, microcode_ctl)

Two new XSAs were published on 12th of March, in cunjunction with microcode updates from Intel.

XSA-452 The mitigation is currently off by default as it impacts only Atom CPUs, but can be enabled on Xen command line.
XSA-453 This is a variation of Spectre-v1, which impacts a large panel of recent CPUs and architectures. This seems to not really be exploitable on Xen without specific changes and is not considered an emergency.

SECURITY UPDATES

xen-*:
* Fix XSA-452 - x86: Register File Data Sampling. Data from floating point, vector and integer register could be infered by an attacker on Atom processors, including data from a privileged context.
* Fix XSA-453 - GhostRace: Speculative Race Conditions. As mentioned, this is a Spectre-v1 variation that can allow an attacker to infer memory accross host and guests through a Use-After-Free flaw.
microcode_ctl: Security updates from intel:
INTEL-SA-INTEL-SA-00972
INTEL-SA-INTEL-SA-00982
INTEL-SA-INTEL-SA-00898
INTEL-SA-INTEL-SA-00960
INTEL-SA-INTEL-SA-01045

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" microcode_ctl --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.13.5-9.39.1.xcpng8.2
microcode_ctl: 2.1-26.xs28.1.xcpng8.2

What to test

Normal use and anything else you want to test.

Test window before official release of the update

2 days because of security updates.

JeffBerntsen

This is installed and working on my two test systems but they're both AMD so I'm not able to test the updated microcode.

Andrew

@bleader Updates running on several old and new intel machines (including microcode update). Working fine so far. Rolling Pool Reboot is a helpful feature.

bleader

The update has been published, thank you for testing it out.

https://xcp-ng.org/blog/2024/03/15/march-2024-security-update/

gduperrey

New update candidates for you to test!

As you may know, we group non-urgent updates together for a collective release, in order not to cause unnecessary maintenance for our users.

The moment to release such a batch has come, so here they are, ready for user tests before the final release.

openvswitch:
- CVE-2023-1668: Correct a flaw when processing an IP packet with protocol 0.
- CVE-2023-5366: Apply the patch for OpenFlow and neightbor discovery target with IPv6
- CVE-2023-3966: Correct a vulnerabity with "crafted Geneve packets causing invalid memory accesses and potential denial of service".
blktap:
- Synced with XS82ECU1056:
  - Bugfix for time out on NFS tasks which can sometimes exceed the configured value.
  - Improve the error handling for some lost iSCSI connection.
sm:
- Support NFS servers which only offer NFSv4. The discovery process for such servers differs from that of servers which offer also NFSv3, so the SR driver had to be improved.
- Synced with XS82ECU1056: bugfix on the path checker for DELL EqualLogic with iSCSI protocol
- Synced with XS82ECU1060: bugfix for when a host is unable to log into all iSCSI portals because there are separate independent Target Portal Groups inside the IQN.
util-linux: preparatory steps to support 4k-only disks.
xapi: Bugfix in a testing framework.
xcp-ng-pv-tools: Small fixes regarding VM stats reporting.
xcp-ng-xapi-plugins: Add check_installed function in updater plugin to test installed packages. This is a prerequisite for the upcoming XOSTOR release.

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing blktap openvswitch sm-* util-linux xapi-* xcp-ng-pv-tools xcp-ng-xapi-plugins
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

blktap: 3.37.4-3.1.xcpng8.2
openvswitch: 2.5.3-2.3.12.2.xcpng8.2
sm: 2.30.8-10.1.xcpng8.2
util-linux: 2.23.2-52.1.xcpng8.2
xapi: 1.249.32-2.2.xcpng8.2
xcp-ng-pv-tools: 8.2.0-12.xcpng8.2
xcp-ng-xapi-plugins: 1.10.0-1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~1 week.

Andrew

@gduperrey Updates installed and running.

JeffBerntsen

They're running without problems for me on my test systems

olivierlambert

Tested and working here

JamesG

@gduperrey Applicable to 8.3?

gskger

@gduperrey Succesfully updated my two host pool. Let's see how the weekend goes with some tests.

stormi

@JamesG No, it's just for XCP-ng 8.2

gduperrey

The updates have been published; thank you for testing them out.

https://xcp-ng.org/blog/2024/03/29/march-2024-maintenance-update/

bleader

New security update candidate (xen)

Three new XSAs were published on 9th of April.

Notes:

XSA-456 was published on various public mailing list but its entry is not yet on the xenbits page, hence the different link for this one.
XSAs description to be completed later, early posting to provide more chances to run tests before final release.

XSA-454 impacts all host running HVM or PVH guests on x86_64, therefore all supported architectures on XCP-ng.
XSA-455 relates to XSA-407 (Branch Type Confusion) having a logical error, check its VULNERABLE SYSTEMS section for impacted systems.
XSA-456 should only impact Intel CPU as it is understood at this time.

SECURITY UPDATES

xen-*:
- Fix XSA-454 - x86 HVM hypercalls may trigger Xen bug check. HVM and PVH guests can DoS a host in some cases calling 32-bit-mode hypercalls with parameters that will lead the hypercall sanity checks to trigger a crash.
- Fix XSA-455 - x86: Incorrect logic for BTC/SRSO mitigations. Fix for XSA-407 was not properly used, meaning an attacker could be able to infer memory from host or other guests. All versions since 4.13.4-9.24.1 are vulnerable.
- Fix XSA-456 - x86: Native Branch History Injection. An attacker could infer memory of host or other guests by using the Native Branch History Ijnection flaw. This is an evolution of Spectre-BHB which was previously considered not to be a risk for Xen.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: 4.13.5-9.40.1.xcpng8.2

What to test

Normal use and anything else you want to test.

Test window before official release of the updates

~1 days because of security updates.