XCP-ng 8.2 updates announcements and testing

NielsH

@olivierlambert said in Updates announcements and testing:

Hello @NielsH I don't want to sound moralistic, but if you are using XCP-ng in production without any subscription, and being worried about patches coming fast enough for your production system, you should really think about getting support for it That's exactly what the subscription money is made for! (well, in part but absolutely used for that).

I mean, you are free to not doing it, and even if we do our best to treat everyone fairly (paying or not) for our patches, we won't be against more support so the project can continue to grow

Hey,

No worries, I understand your point.
For us specifically, we use local storage and the time it takes to migrate all VMs to a different host, reboot, next host, etc... is about 1-1.5 weeks during working hours. This is why, if another batch of updates is coming soon, we would rather wait and do everything at once instead of update twice in a row. I'm aware of the Rolling Pool Upgrade feature but it is not compatible with local storage.

Regarding XCP-ng Pro support, currently the pricing is a bit too steep for us to be feasable due to the amount of hosts we have. We do however pay for XOA Premium which is more affordable and we do find it valuable to support the developers.

I believe I did see some of your posts around regarding a possible all-in pricing plan which may be interesting for us, so once more details regarding that are available it is something for us to look into

olivierlambert

No worries, and indeed, new pricing might be a good fit for you

gduperrey

New Security Update Candidates (kernel, Xen, linux-firmware, microcode_ctl, XAPI...)

Xen is being updated to mitigate some vulnerabilities:

XSA-432: CVE-2023-34319. Under Linux, a buffer overrun in netback can be triggered due to unusual packets. This behavior was due to the fix of the XSA-423 which didn't account an extreme case of an entire packet being split into as many pieces as permitted by the protocol and still being smaller than the area that's dealt with to keep all headers together. It is possible to crash a host from a vm, with malicious and privileged code.
XSA-434: CVE-2023-20569. Researchers from ETH Zurich have extended their prior research (XSA-422, Branch Type Confusion, a.k.a Retbleed) and have discovered INCEPTION, also known as RAS (Return Address Stack) Poisoning, and Speculative Return Stack Overflow. An attacker might be able to infer the contents of memory belonging to other guests.
XSA-435: CVE-2022-40982. A security issue in certain Intel CPUs may allow an attacker to infer data from different contexts on the same core.

Components are also updated to add bugfixes and enhancements:

guest-templates-json: Added Debian 12 Bookworm
XAPI:
- Several hotfixes and improvements from XS82ECU1033
- From XS82ECU1045 Significant performance improvements on a set of CPU features for servers with Cascade Lake or later Intel CPUs.
microcode_ctl: Update to IPU 2023.3
linux-firmware: Expose additional features for Intel CPUs, especially for Cascade Lake or later Intel CPUs. Updated to latest AMD firmware for processor family 19h.
Xen: Expose MSR_ARCH_CAPS to guests on all Intel hardware by default.
blktap, nbd: An update of the packages for Xostor.

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" microcode_ctl linux-firmware kernel forkexecd gpumon message-switch "ocaml-*" rrd2csv rrdd-plugins sm-cli squeezed varstored-guard vhd-tool wsproxy "xapi-*" xcp-networkd xcp-rrdd "xenopsd*" xs-opam-repo "guest-templates-*" blktap xcp-ng-linstor nbd tzdata grub* lldpad xcp-ng-xapi-plugins --enablerepo=xcp-ng-testing
reboot

Version:

forkexecd: 1.18.3-2.1.xcpng8.2
gpumon: 0.18.0-10.1.xcpng8.2
kernel: 4.19.19-7.0.17.1.xcpng8.2
linux-firmware: 20190314-9.1.xcpng8.2
message-switch: 1.23.2-9.1.xcpng8.2
microcode_ctl: 2.1-26.xs26.1.xcpng8.2
ocaml-rrd-transport: 1.16.1-7.1.xcpng8.2
ocaml-rrdd-plugin: 1.9.1-7.1.xcpng8.2
ocaml-tapctl: 1.5.1-7.1.xcpng8.2
ocaml-xcp-idl: 1.96.5-1.1.xcpng8.2
ocaml-xen-api-client: 1.9.0-10.1.xcpng8.2
ocaml-xen-api-libs-transitional: 2.25.5-4.1.xcpng8.2
rrd2csv: 1.2.6-7.1.xcpng8.2
rrdd-plugins: 1.10.9-4.1.xcpng8.2
sm-cli: 0.23.0-53.1.xcpng8.2
squeezed-0.27.0-10.1.xcpng8.2
varstored-guard: 0.6.2-7.xcpng8.2
vhd-tool: 0.43.0-10.1.xcpng8.2
wsproxy: 1.12.0-11.xcpng8.2
xapi: 1.249.32-1.1.xcpng8.2
xapi-nbd: 1.11.0-9.1.xcpng8.2
xapi-storage: 11.19.0_sxm2-9.xcpng8.2
xapi-storage-script: 0.34.1-8.1.xcpng8.2
xcp-networkd: 0.56.2-7.xcpng8.2
xcp-rrdd: 1.33.2-6.1.xcpng8.2
xen: 4.13.5-9.36.1.xcpng8.2
xenopsd: 0.150.17-1.1.xcpng8.2
xs-opam-repo: 6.35.11-1.xcpng8.2
guest-templates-json: 1.9.6-1.3.xcpng8.2
blktap-3.37.4-1.0.2.xcpng8.2
tzdata-2022a-1.el7
xcp-ng-linstor-1.1-3.xcpng8.2
nbd-3.24-1.xcpng8.2
grub-2.02-3.2.0.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days.

olivierlambert

42 packages and 147M worth of updates

Installed here and worked on my HPE EPYC host

gskger

@gduperrey Updated my two host cluster (HP ProDesk 600 G6) and no issues so far.

gduperrey

Update published. Thanks for the tests!

https://xcp-ng.org/blog/2023/08/14/august-2023-security-update/

gduperrey

New Security Update Candidates (Xen)

Xen is being updated to mitigate some vulnerabilities:

XSA-439: CVE-2023-20588. On AMD Zen1 CPUs, "an attacker might be able to infer data from a different execution context on the same CPU core."

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" --enablerepo=xcp-ng-testing
reboot

Version:

xen: 4.13.5-9.36.2.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days.

stormi

We will soon publish the security update. Few hours left for community feedback before the release.

Andrew

@stormi I don't have a Zen 1 system... But I do have the update running on several systems for a day.

olivierlambert

Tested on my Zen2 system, no issues.

JeffBerntsen

Tested in my test lab and no problems but no Zen CPUs either.

stormi

Thanks for the tests! The update was published: https://xcp-ng.org/blog/2023/09/29/september-2023-security-update/

stormi

New security and maintenance updates

Xen and the linux kernel in the controller domain are updated to fix several vulnerabilities.

We also publish several maintenance updates which were ready and waiting for the next push.

Security updates

xen-*:
- Fix XSA-440 - xenstored: A transaction conflict can crash C Xenstored. XCP-ng uses the ocaml version of xenstored by default, so the issue only concerns users who deliberately switched to C Xenstored.
- Fix XSA-442 - x86/AMD: missing IOMMU TLB flushing. Privilege escalation, DoS, information leaks, on some AMD systems, from VMs with PCI passthrough.
- Fix XSA-443 - Multiple vulnerabilities in libfsimage disk handling. Privilege escalation from PV guests through flaws in libfsimage. Remember that PV guests are not security-supported in XCP-ng 8.2. Despite this, this fix is provided for users who still have PV guests, but we still urge them to convert their VMs to HVM.
- Fix XSA-444 - x86/AMD: Debug Mask handling. DoS provoked from a guest.
kernel:
- Fix XSA-441 - Possible deadlock in Linux kernel event handling. As we understand it, the denial of service would not be possible in XCP-ng's default configuration, but we provide the patched kernel as defence in depth.

Other updates

sm (Storage Manager): improvements around the handling of user customizations on multipath configuration
- Do not overwrite multipath.conf if users made changes
- Add warning to multipath.conf to prevent future modifications (for users which haven't modified it yet, that is, the vast majority)
- Add /etc/multipath/conf.d/custom.conf for user customization
guest-templates-json*: sync with latest Citrix Hypervisor hotfixes. We already had templates for Debian 11, Debian 12, Rocky Linux 9 and CentOS Stream 9 in XCP-ng 8.2. The only new template is Ubuntu 2204.
irqbalance: backport of hotfix XS82ECU1048. "Enable interrupt balancing for Fibre Channel (FC) PCI devices. This improves performance on fast FC HBA SRs, especially if multipathing is used."

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" kernel sm sm-rawhba irqbalance "guest-templates-json*" --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

If you are a user of XOSTOR, change the last parameter to --enablerepo=xcp-ng-testing,xcp-ng-linstor-testing to pick the linstor-compatible version of the sm update.

Versions

xen-*: 4.13.5-9.36.3.xcpng8.2
kernel: 4.19.19-7.0.17.2.xcpng8.2
sm: 2.30.8-2.3.xcpng8.2
irqbalance: 1.0.7-16.xcpng8.2
guest-templates-json*: 1.10.6-1.1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days due to the security updates.

olivierlambert

Tested on my EPYC host, no issue at all

Danp

Installed yesterday on my R620; no issues experienced thus far.

Andrew

@stormi I have had the updates running for a day on a few machines.

JeffBerntsen

I have it running on my test servers, an HP MicroServer and a ProLiant DL165 G7. No problems so far.

stormi

Thanks for your tests!

Update published: https://xcp-ng.org/blog/2023/10/12/october-2023-security-update/

olivierlambert

\o/ well done, thanks everyone!

stormi

New security update candidates

As promised in the previous security update, here's a new one which goes further, for better defence-in-depth.

Security updates

Xen, XAPI and all its components, sm, libfuse, e2fsprogs: various patches to either deprivilege any component which uses libfsimage, or remove its use altogether. The previous security update fix the urgent flaws described in XSA-443, and this update takes further measures as defence-in-depth.
xs-openssl: rebased on a more recent release from RHEL/CentOS, which fixes various CVEs
kernel: nothing to declare. We just rebuilt it after syncing with XenServer's latest hotfix, but there are no source code changes.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

If you are a user of XOSTOR, do not install this update candidate. A linstor-compatible version of the sm update is still being prepared.

Versions

blktap: 3.37.4-2.1.xcpng8.2
device-mapper-multipath*: 0.4.9-136.xcpng8.2
e2fsprogs*: 1.47.0-1.1.xcpng8.2
forkexecd: 1.18.3-3.1.xcpng8.2
fuse-libs (added): 2.9.2-10.xcpng8.2
gpumon: 0.18.0-11.1.xcpng8.2
kernel: 4.19.19-7.0.18.1.xcpng8.2
message-switch: 1.23.2-10.1.xcpng8.2
rrd2csv: 1.2.6-8.1.xcpng8.2
rrdd-plugins: 1.10.9-5.1.xcpng8.2
sm: 2.30.8-7.1.xcpng8.2
sm-cli: 0.23.0-54.1.xcpng8.2
sm-rawhba: 2.30.8-7.1.xcpng8.2
squeezed: 0.27.0-11.1.xcpng8.2
varstored-guard: 0.6.2-8.xcpng8.2
vhd-tool: 0.43.0-11.1.xcpng8.2
wsproxy: 1.12.0-12.xcpng8.2
xapi-core: 1.249.32-2.1.xcpng8.2
xapi-nbd: 1.11.0-10.1.xcpng8.2
xapi-storage: 11.19.0_sxm2-10.xcpng8.2
xapi-storage-script: 0.34.1-9.1.xcpng8.2
xapi-tests: 1.249.32-2.1.xcpng8.2
xapi-xe: 1.249.32-2.1.xcpng8.2
xcp-networkd: 0.56.2-8.xcpng8.2
xcp-rrdd: 1.33.2-7.1.xcpng8.2
xen-*: 4.13.5-9.37.1.xcpng8.2
xenopsd*: 0.150.17-2.1.xcpng8.2
xs-openssl-libs: 1.1.1k-9.1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~3 days