XCP-ng 8.2 updates announcements and testing

NielsH

@stormi said in Updates announcements and testing:

@stormi said in Updates announcements and testing:

A big thank you to those who tested, and last call for the others! Publish time is close.

If you are wondering "how close", I delayed them a bit to potentially group them with another update.

Hi @stormi

Any idea how long "a bit" is?

Cheers!
Niels

stormi

The updates I wanted to group them with did not come, so it all got delayed (and retrospectively I could have released them earlier).

Are you waiting for a specific fix?

NielsH

@stormi said in Updates announcements and testing:

The updates I wanted to group them with did not come, so it all got delayed (and retrospectively I could have released them earlier).

Are you waiting for a specific fix?

Not specifically, but we have not yet installed the previous patches. We were planning on combining them with the upcoming patches. Since it always takes a while to update / reboot all hypervisors we wanted to avoid double work if we were to install the current available updates today and for example tomorrow new ones are released

stormi

The best I can say is: real close... probably...

stormi

The long awaiting (and awaited?) train of updates was finally published (including the XAPI update)!

https://xcp-ng.org/blog/2021/12/08/december-2021-xcp-ng-updates/

A big thanks for your feedback everyone, and get ready for the next ones!

NielsH

@stormi said in Updates announcements and testing:

The long awaiting train of updates was finally published (including the XAPI update)!

https://xcp-ng.org/blog/2021/12/08/december-2021-xcp-ng-updates/

A big thanks for your feedback everyone, and get ready for the next ones!

Cheers!

We don't see them yet, but I assume it may take a few hours for the mirrors to update?

[13:59 xen32 ~]# yum  update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

stormi

In theory, mirrorbits should always redirect you to a mirror that has the latest files you are requesting. However there's also a local cache for yum metadata. Try to clean it: yum clean all.

NielsH

@stormi said in Updates announcements and testing:

I theory, mirrorbits should always redirect you to a mirror that has the latest files you are requesting. However there's also a local cache for yum metadata. Try to clean it: yum clean all.

Ah yes, after that I see 22 packages. Just googling for this but appears to be based on the metadata_expire option in /etc/yum.conf. The man page says the default is 6 hours, but that config file has a commented out 90m. Either way, I'll wait a few hours/until tomorrow and then it should show all packages to update in XOA on all servers

stormi

New security updates (xen, kernel)

Citrix security bulletin: https://support.citrix.com/article/CTX335432

Impact: privileged code in a guest VM may crash a host or make it unresponsive.

Note: although PV guests are unsupported since XCP-ng 8.1, this very update actually breaks PV 32 bit compatibility (on purpose, for a performance gain on dom0).

However, you really should not have any remaining PV guest at this stage (especially 32 bit), for security reasons. There are unfixed vulnerabilities, and they won't be fixed.

🦺 Our support will be ready to help migrate to HVM (better), or other solutions.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update kernel linux-firmware xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.4-9.18.1.xcpng8.2
kernel: 4.19.19-7.0.14.1.xcpng8.2
linux-firmware: linux-firmware-20190314-2.xcpng8.2

What to test

Normal use and anything else you want test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days. I plan to release the updates on Monday morning, CET.

gskger

@stormi Updated my two host playlab and tested Debian 11 and Windows 10 VMs (create, live migrate with guest tools, start/stop/reboot, online-/offline storage migration from/to shared and local SR, snapshot with/-out RAM and revert) and imported an Ubuntu VM for testing as well. All seems to be good.

Made some full copies of the Debian VM while at it to see what performance I get in my temporary 10G setup with a simultaneous live migration between hosts, which worked out nicely (knowing that this is not a realworld test and just for fun).

olivierlambert

We should expect some perf bump in dom0, around 5 to 10% which is nice (thanks to PV32 removal)

olivierlambert

Tested on my HPE ProLiant DL385 Gen10 (EPYC), everything is working too

Andrew

You mean xen 4.13.4, right? That's what's I installed.

I updated a few active machines and no problems so far:
HP G5 (E5450), HP G8 (E5-2680), AMD (Opt 6380)

stormi

@andrew said in Updates announcements and testing:

You mean xen 4.13.4, right? That's what's I installed.

Indeed. I fixed it in the message, thanks.

stormi

Thanks @gskger and @Andrew (and of course Olivier) for the tests.

We released the update on monday: https://xcp-ng.org/blog/2022/01/17/january-2022-security-update/

If for any reason you still had (unsupported since 8.1) 32 bit PV guests, read carefully.

stormi

New security updates (Intel microcode, Xen)

Impact: Vulnerabilities in Xen -> privileged code in a guest VM may crash a host under some conditions (PV guests / PCI-passthrough). Vulnerabilities in Intel CPUs have consequences... That I'm not able yet to describe precisely in the context of XCP-ng and virtualization in general. Looks bad enough: privilege escalation and DoS.

See https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00528.html and https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00532.html

Citrix' security bulletin: https://support.citrix.com/article/CTX337526

Test on XCP-ng 8.2

You can test even if your host was updated with the test packages for XCP-ng 8.2.1.

yum clean metadata --enablerepo=xcp-ng-testing
yum update microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.4-9.19.1.xcpng8.2
microcode_ctl: 2.1-26.xs19.xcpng8.2

What to test

Normal use and anything else you want test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days. Maybe less.

HeMaN

@stormi

I get an error on the xcp-ng-testing repository

[14:14 xcp-ng-01 ~]# yum clean metadata --enablerepo=xcp-ng-testing
Loaded plugins: fastestmirror
Cleaning repos: dell-system-update_dependent dell-system-update_independent xcp-ng-base xcp-ng-testing xcp-ng-updates
10 metadata files removed
11 sqlite files removed
0 metadata files removed
[14:16 xcp-ng-01 ~]# yum update microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-testing: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
dell-system-update_dependent                                                                                                                                           | 2.3 kB  00:00:00
dell-system-update_independent                                                                                                                                         | 2.3 kB  00:00:00
xcp-ng-testing/signature                                                                                                                                               |  473 B  00:00:00
xcp-ng-testing/signature                                                                                                                                               | 3.0 kB  00:00:00 !!!
xcp-ng-testing/primary_db      FAILED
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found-:--:-- ETA
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

(1/3): dell-system-update_dependent/7/x86_64/primary_db                                                                                                                |  25 kB  00:00:00
(2/3): dell-system-update_independent/primary_db                                                                                                                       | 132 kB  00:00:01
xcp-ng-testing/primary_db      FAILED
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found-:--:-- ETA
Trying other mirror.
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.


 One of the configured repositories failed (XCP-ng Testing Repository),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=xcp-ng-testing ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable xcp-ng-testing
        or
            subscription-manager repos --disable=xcp-ng-testing

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=xcp-ng-testing.skip_if_unavailable=true

failure: repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2 from xcp-ng-testing: [Errno 256] No more mirrors to try.
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found

stormi

@heman said in Updates announcements and testing:

yum clean metadata --enablerepo=xcp-ng-testing

Looks like there's a small delay during which mirrorbits hasn't noticed the changed metadata files, and thus redirects you to old filenames. I had taken measures against this (I make it rescan the local repository just after it is updated), but there seems to be situations where it doesn't work.

I restarted the service and now it's good.

HeMaN

@stormi

Installation now went well, no issues noted after the reboot.
Will keep an eye on my system and report back if anything unusual is happening

Andrew

@stormi Are these updates going to staging and be part of 8.2.1 or is this a February 8.2.0 security update before 8.2.1 ?