XCP-ng 8.2 updates announcements and testing

stormi

@normanghenderson They do

normanghenderson

@stormi Doh. How did I miss that?

stormi

New maintenance update candidate (xs-openssl, xcp-ng-xapi-plugins, blktap, sm)

Several package updates that we had queued for a future update are ready for you to test them.

xs-openssl:
- was rebuilt without compression support. Although compression was not offered by default and the clients that connect to port 443 of XCP-ng hosts don't enable compression by default, it's better security-wise not to support it at all (due to CRIME), and this will make security scanners happier.
- received a patch from RHEL 8's openssl which fixes a potential denial of service: "CVE-2022-0778 openssl: Infinite loop in BN_mod_sqrt() reachable when parsing certificates"
xcp-ng-xapi-plugins received a few fixes:
- Avoid accidentally installing updates from repositories that users may have enabled on XCP-ng (even if they should never do this), when using the updater plugin (Xen Orchestra uses it to install updates).
- In the updater plugin again, error handling was broken: whenever an error would occur (such as a network issue preventing from installing the updates), another error would be raised from the error handler, and thus mask the actual reason for the initial error. That's what happens when you write command with 3 m .
blktap: received a fix backported from one if Citrix Hypervisor's hotfixes, which addresses a possible segmentation fault if you create a lot of snapshots at the same time.
sm ("Storage Manager", responsible for the SMAPIv1 storage management layer) received a few fixes:
- We fixed an issue with local ISO SRs and mountpoints: creating a local ISO SR on a directory that is a mountpoint for another filesystem would unmount it. The patch was not accepted upstream because it touches legacy code that Citrix won't support, according to the developer who answered, but we considered it safe and useful enough to apply it to XCP-ng anyway.
- The (experimental) MooseFS driver will now default to creating a subdirectory in the mounted directory, to avoid collision between several SRs using the same share.
- The update also includes the followings fix from one of Citrix Hypervisor's hotfixes: CA-352880: when deleting an HBA SR remove the kernel devices
- Two other fixes which are hard to explain in user terms but typically don't affect the majority of users.

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update blktap sm sm-rawhba xcp-ng-xapi-plugins xs-openssl-libs --enablerepo=xcp-ng-testing
xe-toolstack-restart

Toolstack restart should be enough but I haven't checked yet that it does restart the TLS tunnel, so a reboot may be safer to ensure you actually test the updated packages.

Versions:

xs-openssl-libs: 1.1.1k-6.1.xcpng8.2
xcp-ng-xapi-plugins: 1.7.2-1.xcpng8.2
blktap: 3.37.4-1.0.1.xcpng8.2
sm, sm-rawhba: ~~2.30.7-1.2.xcpng8.2~~ updated 2022-07-08 to fix a regression: 2.30.7-1.3.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

No precise ETA, but the sooner the feedback the better.

olivierlambert

Tested with success here

gskger

@stormi All good with updating my playlab.

JeffBerntsen

Seems to be working fine for me as well.

stormi

We found and fixed a regression in the update candidate for sm regarding shared thick-provisioned drivers based on LVM (lvmoiscsi and others).

Please update again (mirrors will receive the within 15 minutes) using the command given above. This will give you sm and sm-rawhba in version-release 2.30.7-1.3.xcpng8.2.

gskger

@stormi Can not help with the sm and sm-rawhba regression, but did the update anyway. Worked well and kicking some VMs around works as expected. Let's see how my playlab does during the weekend.

AlexD2006

@stormi
is there some information about that regression?
maybe i can help to test.
have a lab pool here with 3 hosts on a iscsi SR.

busthead

Hello,

This thread appears to be for testing update candidate releases. Where can I find the release notes for production ready updates?

Thx

olivierlambert

Any functional change will be explained in our blog. Other updates are mostly security (see the security tag on the blog: https://xcp-ng.org/blog/tag/security)

Also https://xcp-ng.org/blog/tag/update

gduperrey

New security update (xen, Intel and AMD CPUs)

Xen is being updated to mitigate hardware vulnerabilities in Intel and AMD CPUs.

Upstream (Xen project) advisory: XSA-407
Citrix Hypervisor Security Bulletin: https://support.citrix.com/article/CTX461397/citrix-hypervisor-security-bulletin-for-cve202223816-and-cve202223825

Impact of the vulnerabilities - RETbleed is a speculative execution attack on x86-64 processors, including some recent Intel and AMD chips. You can read the original paper from Computer Security Group at this address: https://comsec.ethz.ch/research/microarch/retbleed/

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.4-9.24.1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days.

olivierlambert

Tested in my lab, so far so good.

More testers means faster release

JeffBerntsen

@gduperrey This update breaks the XOSTOR setup in my test lab. Would you like any information from me for troubleshooting? If not, what's the easiest way to revert the test updates?

olivierlambert

Hmm no reason for that, did you have other updates coming with it? (ie replacing some other unrelated packages).

olivierlambert

Adding @ronan-a in the loop.

When you said "break", can you be more specific @JeffBerntsen ?

JeffBerntsen

@olivierlambert said in Updates announcements and testing:

Adding @ronan-a in the loop.

When you said "break", can you be more specific @JeffBerntsen ?

@olivierlambert said in Updates announcements and testing:

Hmm no reason for that, did you have other updates coming with it? (ie replacing some other unrelated packages).

I have 3 servers in my lab's test pool, vh97, vh98, and vh99. Server vh97 was the pool master. I placed it into maintenance mode and shifted the pool master to vh98 in the process, installed the updates, and rebooted.

When vh97 came back up, it could no longer attach to the XOSTOR SR. Attempting to replug the PBD gives me errors indicating that vh97 can no longer see the SR. It appears that there are some related errors in the SMlog file (I've captured a copy for future examination).

The updates installed were only the ones from this most recent set. All of the other test updates from this thread from the last two weeks were already installed and running without problems on all 3 servers in the pool.

olivierlambert

If you test XOSTOR, don't test other updates, because this might break the rest.

Also, never switch the master after updates, the master HAS to reboot first, whatever happens.

JeffBerntsen

@olivierlambert said in Updates announcements and testing:

If you test XOSTOR, don't test other updates, because this might break the rest.

Also, never switch the master after updates, the master HAS to reboot first, whatever happens.

I know that XOSTOR might break during an update test. I thought whether it did or didn't might be something you would be interested in as part of testing and that's part of why I test with it in place and running.

ronan-a

@JeffBerntsen What's your sm version? I suppose, you updated your hosts and you don't have the right one.

Please send me the output of: rpm -qa | grep sm-.