XCP-ng 8.2 updates announcements and testing

Danp

@stormi I've updated my R620 running XCP-ng 8.2, and only issue I've observed thus far is the network slowdown that we've been discussing elsewhere.

I need to rebuild the kernel again without the one patch. Has there been any feedback / response from the Xen group?

stormi

@danp I haven't contacted them globally though I asked a developer individually for guidance on how to move on. My issue is: we do know that for some users there's a performance impact and it's related to the kernel updates, but the results of the tests don't reveal yet the exact updates or patches that are involved, or not for everyone (though the one you found looks definitely a good candidate to explain at least parts of the slowdown). And I want to see where @fohdeesha's attempt at helping everyone get better perfs will lead.

(but let's continue on the dedicated thread)

stormi

Thanks for all the tests. The security updates have been pushed: https://xcp-ng.org/blog/2020/12/18/december-2020-security-updates/

stormi

New security update for XCP-ng 8.2 only

See https://xcp-ng.org/forum/post/35658

The interesting story about this one is that it was first reported and debugged on this forum, on the thread linked just above, without us knowing that it would be identified as a security (DoS) issue by Xen developers when we'd report it to them.

Please install it, reboot, and as usual check for any obvious regressions.

gskger

@stormi Can not speak for the issue solved, but upgrading my three host playlab from XCP-ng 8.2 fully patched worked as well as create, live / storage migrate, copy, delete, snapshot (with/without ram), backup and restore of Linux and Windows 10 VMs. Nice upstream, downstream and forum work .

stormi

@gskger Thanks, you're always there for the tests!

olivierlambert

Indeed, that's really useful! Thanks @gskger

stormi

Update pushed to the update repository, blog post to be published on Monday.

stormi

Another update in testing: ca-certificates, for both XCP-ng 8.1 and 8.2.

The one we have (from Citrix Hypervisor) is 2 years old and it's good to refresh the list of root certificates from time to time. Not much impact on XCP-ng actually because it's mostly used when you wget or curl an external URL AFAIK. And probably for yum too.

Install with

yum clean metadata --enablerepo=xcp-ng-testing
yum update ca-certificates --enablerepo=xcp-ng-testing

Then usual checks that nothing looks unexpectedly broken.

HeMaN

@stormi
I did not test the updates in the test-repository this time because I wanted to test the Rolling Pool Update function from XO. I recently created a pool of 2 hosts.

I noticed the patches were available according to XO on the recently installed host, but not on the host I am using already for a longer time (and used to install the test-patches before). This host is also the pool master.
Only after a yum clean all and a yum update the updates were visible on the first host and thus the pool.

After fixing that, the Rolling Pool Update went very smooth. I like this feature!

HeMaN

@stormi said in Updates announcements and testing:

yum clean metadata --enablerepo=xcp-ng-testing
yum update ca-certificates

After the rolling pool update of the released production patches, I wanted to test the ca-certificates from the testing repository as well.
Maybe I was to fast but I got no updates on both hosts

[18:30 xenserver-2 ~]# yum clean metadata --enablerepo=xcp-ng-testing
Loaded plugins: fastestmirror
Cleaning repos: xcp-ng-base xcp-ng-testing xcp-ng-updates
9 metadata files removed
8 sqlite files removed
0 metadata files removed
[18:30 xenserver-2 ~]# yum update ca-certificates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
xcp-ng-base/signature                                                 |  473 B  00:00:00
xcp-ng-base/signature                                                 | 3.0 kB  00:00:00 !!!
xcp-ng-updates/signature                                              |  473 B  00:00:00
xcp-ng-updates/signature                                              | 3.0 kB  00:00:00 !!!
(1/2): xcp-ng-updates/primary_db                                      |  46 kB  00:00:00
(2/2): xcp-ng-base/primary_db                                         | 1.2 MB  00:00:01
No packages marked for update

[18:34 xenserver-3 ~]# yum list installed ca-certificates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Installed Packages
ca-certificates.noarch                                                                2018.2.22-70.0.el7_5                                                                @install/$releasever

stormi

@heman You're right, I had forgotten an --enablerepo switch in my post. I edited it to add it.

HeMaN

@stormi said in Updates announcements and testing:

@heman You're right, I had forgotten an --enablerepo switch in my post. I edited it to add it.

I am not at my best today I noticed, I should have seen that

Anyway, installed without issue. No strange behaviour afterwards

olivierlambert

@heman said in Updates announcements and testing:

I noticed the patches were available according to XO on the recently installed host, but not on the host I am using already for a longer time (and used to install the test-patches before). This host is also the pool master.
Only after a yum clean all and a yum update the updates were visible on the first host and thus the pool.

After fixing that, the Rolling Pool Update went very smooth. I like this feature!

Thanks for the feedback. I think we must add a feature to do that from the plugin ("force refresh updates"). Pinging @nraynaud about this.

jmccoy555

@stormi Applied ca-certificates along with the security patch and all is good in my pool.

stormi

The blog post, as promised: https://xcp-ng.org/blog/2021/01/25/january-2021-security-update/

stormi

A new update of the Xen packages which is not a security update this time is available for tests. It fixes crash analysis with xen-crashdump-analyser (this runs automatically when the host crashes and puts results in /var/crash).

When the Release tag (e.g. 9.8.2.xcpng8.2) of the RPM was longer than a certain number of characters (last digit of Xen version + - + release tag <= 16 chars), it was truncated in the xen_extra field (xl info xen_extra) which caused xen-crashdump-analyser to not find some required files from /boot to conduct its crash analysis). The updated packages fix that by removing the .xcpng8.2 or .xcpng8.1 suffix from the filenames in /boot and from the xen_extra value.

Installation:

yum clean metadata --enablerepo=xcp-ng-testing
yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing

Main objective of the tests: as usual, detect obvious regressions.

If you want to test the fixed behaviour in case of crash, see https://github.com/xcp-ng/xcp/issues/476 (basically, provoke a crash with the command I gave in the comments, then check that the xen-crashdump-analyser.log file in /var/crash is OK).

ghost created this issue in xcp-ng/xcp

closed Kdump not generating proper Xen Crashdump Analyzer file on XCP-ng 8.2 #476

stormi

I've promoted the sudo (https://xcp-ng.org/blog/2021/01/28/security-issue-in-sudo/) and ca-certificate update candidates to official updates.

The Xen update is on hold until it's been sufficiently tested.

jmccoy555

A bit late to the party....... Updated my pool and no oddities to report.

Arraylist

Is a host reboot really necessary for the sudo and ca-certificate updates (as noted in the blog post)?
On an ordinary linux system I wouldn't see a need to restart after updating these packages.