XCP-ng 8.2 updates announcements and testing

hecki

@stormi I read the warning about problems with the rolling-pool-update, so after updating the pool with xen-orchestra, i wanted to live-migrate the Vms manually with xen-center. But live-migration fails, an the VMs crashed. I had to stop the other vms on the master and then reboot to get them back online.
After that I shut down the other VMs to savely reboot the other nodes.

So I think, the warning is not fully correct, also without using the rolling-pool-update feature you can get into trouble. Maybe the mentioned toolstack-restart after the update could have helped me, but there was no info about that in the blog post warning ...

stormi

@hecki Thanks for the feedback. I agree that we could try to make it even more obvious how to avoid the issue, but the blog post does already point at the documentation for updating, as always when we publish an update, and the documentation clearly states you need to restart the toolstack (and it's not new... It's just that many people just overlook it and fortunately for them in most cases this is without consequences... Hence bad habits taken): https://xcp-ng.org/docs/updates.html#from-xen-orchestra

AFAIK the next release of Xen Orchestra will either restart the toolstack or remind you to do it when you apply patches.

hecki

@stormi I alwas used the rolling pool update since its available so i did not read the docs since then any more. If I understand you correct, until now the rolling pool update does not do a toolstack restart during the node restarts although it is so important?

stormi

@hecki In the case of Rolling Pool Update it's not a matter of toolstack restart anymore but rather of updating and rebooting hosts one by one rather than all at once.

hecki

@stormi ok, so "Install pool patches" and "Rolling pool update" internally behave completely different. This is new for me, until now, i thought, the Rolling Pool Update is a "Install poool patches" with following evacuation and reboot of the nodes, so thats what I have done manually today. Thanks for the explanation ...

stormi

@hecki That's what it was (with extra steps also such as putting hosts in maintenance mode), but this will change in the next release of Xen Orchestra as Rolling Pool Update will now evacuate, update then reboot every host one by one

NielsH

Looks like yet another security patch from Xenserver available now

https://xenbits.xen.org/xsa/advisory-398.html

stormi

@NielsH No, they just updated the existing advisory. See "UPDATES IN VERSION 2".

NielsH

@stormi Ah phew, good to hear we won't have to update everything so quickly again

stormi

New security updates (xen, Intel microcode)

Impact: host crash by exploiting PCI passthrough from a VM

Citrix' security bulletin: https://support.citrix.com/article/CTX390511

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.4-9.21.1.xcpng8.2

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days.

olivierlambert

Tested here on one home lab machine, it works

Andrew

@stormi Post update, things are working as normal on my hosts.

HeMaN

updated and all fine

JeffBerntsen

@stormi Seems to be working well for me too.

stormi

Thanks for your tests. Xen developers have found an issue in the patches so I will provide a new test package when ready. The issue is rather specific so I doubt your hosts will be affected, but if you prefer to revert you can with: yum downgrade "xen-*".

stormi

So, new packages are now available for this security update, which now have fixes for the regressions the previous fixes caused.

Note that you were really unlikely to be directly affected by those regressions. They have been detected by code analysis tools, but all other CI tests did already pass with the previous update (Xen's CI, Citrix' CI, XCP-ng's CI).

Install the update with:

yum clean metadata --enablerepo=xcp-ng-testing
yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Version for xen-*: 4.13.4-9.21.1.xcpng8.2

If I could get feedback on it within 24h, this would be wonderful.

olivierlambert

Tested and OK here

Andrew

@stormi Updates ran fine. Systems seem to be working.

theAeon

just joining the crowd of "seems fine"

stormi

Thank you everyone.

The update is published: https://xcp-ng.org/blog/2022/04/11/april-2022-security-update/