Updates announcements and testing

HeMaN

@stormi said in Updates announcements and testing:

Have a installation without any of the fixed bugs, but Installed the updates on my system and did the toolstack restart.

No issues to report this far.

stormi

There are at least two blocking issues with this update candidate, so we'll retain it until they are fixed:

HTTP 403 errors on port 443. Easily reproduced: just load XCP-ng's web page over HTTPS, most images and scripts don't load. We debugged it and reported it to the XAPI project: https://github.com/xapi-project/xen-api/issues/4517
HA Lizard users reported issues connecting to XAPI. I don't know yet whether the fix belongs in XAPI or in HA Lizard itself. Wait and see.

stormi created this issue in xapi-project/xen-api

closed Bogus HTTP 403 errors after hotfix XS82E031 for valid HTTPS requests #4517

stormi

New security updates (xen)

Citrix security bulletin: https://support.citrix.com/article/CTX325319

Impact: privileged code in a guest VM may crash or compromise a host.

Test on XCP-ng 8.2

yum clean metadata --enablerepo=xcp-ng-testing
yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Version for xen packages: 4.13.1-9.12.1.xcpng8.2

What to test

The main goal is to avoid obvious regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

24h.

JeffBerntsen

@stormi Running well for me. I've tested startup, shutdown, and migration of Windows and Linux VMs with no obvious regressions.

gskger

@stormi No issue with updating my two host playlab. Just run some simple tests with Debian VMs (create, live migrate with/-out 7.20.0-9 guest tools, start/stop/reboot, snapshot with/-out RAM and revert, online-/offline storage migrate from/to shared and local SR, restore from backup). Restored a Windows 10 VM from backup as well and moved it around a bit. Everything works as expected.

cg

Any info about support of Windows Server 2022. RTM (final) version has been released ~2 weeks ago.

cnaumer

@stormi Also installed on our test-lab and booting, migrating of RockyLinux and Windows VMs works as expected.

stormi

@cg I've not heard of it yet on Citrix side. Meanwhile the template for 2019 should be enough. IIRC @Darkbeldin has tested Windows Server 2022 regularly during the pre-release phase.

cg

@stormi sounds good. I was guessing that 2019 might do the job. Tempted to use 2022 on a fresh installation and not use the "old" 2019. (Thanks!)

HeMaN

@stormi Installed on my testlab, no issues

stormi

The security update is now live. Thank you everyone for the prompt feedback in this short timeframe!

https://xcp-ng.org/blog/2021/09/10/september-2021-security-update/

HeMaN

@stormi said in Updates announcements and testing:

There are at least two blocking issues with this update candidate, so we'll retain it until they are fixed:

HTTP 403 errors on port 443. Easily reproduced: just load XCP-ng's web page over HTTPS, most images and scripts don't load. We debugged it and reported it to the XAPI project: https://github.com/xapi-project/xen-api/issues/4517

I am using my own self signed certificate on my servers and did not notice the page looking any different, all pictures and text are there.
I do notice the "deploy xoa" script is not working. The buttons seems to be non-functional and do nothing when clicked.
All other buttons and links are functional.

On a larger screen the alignment of the page content is different (topics and pictures side by side in the old page, but in a single colomn in the new page, just like it is with the old page on smaller windows).

Another thing I think is cosmetical but annoying:
when not typing https explicitly in the browser bar (so using http) I get an "403 forbidden" message instead of being forwarded to https

Edit: browser is firefox version 91.0.2 64 bits on windows 10

gskger

@stormi Nice work and the joint effort by the community for tests gets better every time . Keep up the good work . Updated my (semi production) three host homelab and it works - as usual. Now it is time to tear down my playlab for some 10G testing .

jmccoy555

Hi @stormi is there any easy way to rollback the September updates??? I'm guessing not but I have a strange issue which I think coincides with my last reboot after applying the updates so want to confirm or eliminate as the cause.

Thanks.

stormi

@jmccoy555 Have a look at yum history. You should find the update listed and be able to rollback.

Another way is to use yum downgrade package-1version-release package2-version-release on every package involved in the update (list found in yum history or /var/log/yum.log).

stormi

Note (for you or anyone coming here later): rollback/downgrade is not officially supported, because it's not tested, and it is not always possible to ensure that it really brings the exact previous state. You wouldn't rollback a XAPI update that modified the structure of the XAPI db, for example. However for many packages it's safe to attempt it. The September update, that only contains xen packages IIRC, is one of the updates that should be easy and safe to revert.

jmccoy555

@stormi thanks as always. If I do find the issue I'll let you know..... if I break everything then I'll just get a

jmccoy555

It appears that the rollback worked, but doesn't appear to have an impact on my issue, which is good news in a way.

stormi

Let's test the next train of updates

I have various updates ready and tested internally that are eager to be pushed officially. All they need is a bit of user feedback, and that's why we're all on this thread right?

What changes

Updated ca-certificates removes an expired root certificate that was used by Let's Encrypt, in order to workaround a limitation of the old version of openssl included in XCP-ng when the chain of trust contains an expired certificate, even when another path would allow to verify the certificate. Basically, this just means that wget would fail on most HTTPS URLs that use a Let's Encrypt certificate, and now it won't fail anymore.
Updated kernel (bugfix update, already detailed above and tested by some of you)
Update (2021-10-27): new patches synced from new [Citrix hotfix](https://support.citrix.com/article/CTX330706). Removes spurious kernel warnings and supposedly increases the "resiliency" of the kernel (ie, bugs were fixed).
Updated grub fixes a booting issue with buggy UEFI firmware that only wants to boot from EFI/BOOT/BOOTX64.EFI... Or worse, firmware that doesn't really boot from this file but won't boot if the file doesn't exist...
Updated xcp-featured fixes a bug that made the Pool Secret Rotation feature (something you rarely need - as no one reported the issue - but can be useful sometimes) unavailable.
Updated guest-templates-json* packages add a VM template for Rocky Linux. It's not really different from the template for CentOS 8, but should please Rocky users.
Updated xcp-ng-release* packages bring small fixes to the XOA deploy feature on host landing web pages, and update jquery to fix an XSS vulnerability in this library.

How to update

yum clean metadata --enablerepo=xcp-ng-testing
yum update ca-certificates grub grub-efi grub-tools guest-templates-json guest-templates-json-data-linux guest-templates-json-data-other guest-templates-json-data-windows kernel xcp-featured xcp-ng-release xcp-ng-release-config xcp-ng-release-presets --enablerepo=xcp-ng-testing

Then reboot.

What to test

The same as usual: installation of the update, normal use, check that you find no obvious regressions... This is the most important.

And optionnally the changes described above if you're in a situation that allows it.

Test window before release

A few days.

What's not included in this update train

The XAPI update is not included yet due to a regression found during the tests: our landing web page was completely broken when loaded in HTTPS (which becomes the only way as HTTP is disabled with this update). We identified the issue and contributed a fix to the XAPI project. There's still some work to do internally before we can release it confidently.

ug1556

@jcastang

Hello are you using i40e driver for your network card?