XCP-ng 8.2 updates announcements and testing

Andrew

@olivierlambert Install patches from from XO source (current master) and 8.2.1. It works, but I an error.

Host install:

pool.installPatches
{
  "hosts": [
    "64b9bf4a-a991-438d-973f-60179f3d0868"
  ]
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

Pool install:

pool.installPatches
{
  "pool": "98ab0129-f9d1-ba08-c1c0-aa93c38d7fec"
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:152:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

gskger

@olivierlambert @stormi I started the rolling update from XO and left it running alone since it usually takes some time to complete. This xenopsd internal error is from the XO log.

2022-02-28T15_21_32.891Z - XO.log.txt

An hour later I realized that the updates had been applied, but the hosts did not reboot. I tried to reboot the master from XO first which might have resulted in this XO log.

2022-02-28T16_35_04.612Z - XO.log.txt

After that all VMs including XO became unresponsive and I continued to investigate with XCP-ng Center. In the end, I decided to reboot the hosts from CLI (master first and than all slaves). That resolved the issue and XO came up without any further problems.

I have copies of the logs @stormi suggested from the master and one slave if they are needed. XO is XO from source by 3rd party script. I did an XO update after resolving the issue, so if you need the XO version, I can restore from backup and check.

Hope that helps.

olivierlambert

@Andrew I believe this is a problem on XO, unrelated to the XCP-ng update itself.

@gskger so we had the same issue. However, I did something differently: after restarting the toolstack, I could migrate VMs around without any problem, then reboot manually each evacuated host.

gskger

@olivierlambert Yes, restarting the toolstack first would have been a better approach. Is this something that needs additional actions on the hosts?

olivierlambert

No, that should be enough

stormi

We added a warning on the blog post https://xcp-ng.org/blog/2022/02/28/xcp-ng-8-2-1-update/ and will work on a fix in XO.

HeMaN

@stormi
now realise I had the same behaviour after applying the latest 8.2.0 security updates a few weeks ago.
At that time thought it was because I patched the master manually already (test repo release) and updated the entire pool only after the official release 2 days later (roling pool update).

So it is not (only) related to the 8.2.1 release

stormi

@HeMaN I think it's more likely to happen with the 8.2.1 update because more components are updated. Else I agree.

stormi

New security updates (xen, Intel microcode)

Impact: newly disclosed security vulnerabilities may allow attackers from within a guest to access memory of the host or other VMs. Only AMD CPUs are affected. To know if your specific model is affected, consult the vendor publications. Xen has been patched to mitigate the hardware issue.

Additionnally but unrelated to the security issue, firmware is updated for some models.

Citrix' security bulletin: https://support.citrix.com/article/CTX341586

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update linux-firmware microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.4-9.20.1.xcpng8.2
microcode_ctl: 2.1-26.xs20.xcpng8.2
linux-firmware-20190314-3.xcpng8.2

What to test

Normal use and anything else you want test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days. Maybe less.

Andrew

@stormi Manual updates done on all my 8.2.1 machines and things are working correctly so far.

olivierlambert

Same thing on my EPYC machine.

NielsH

Thanks!

I see there's also another XSA available; XSA-396 (https://xenbits.xen.org/xsa/ )
Will these be combined with the microcode patches ?

Cheers!

stormi

@NielsH XSA-396 does not affect XCP-ng as the device back-ends are in dom0 directly, not in driver domains.

stormi

Security update released. Thanks @Andrew for testing!

https://xcp-ng.org/blog/2022/03/11/march-2022-security-update/

hecki

@stormi I read the warning about problems with the rolling-pool-update, so after updating the pool with xen-orchestra, i wanted to live-migrate the Vms manually with xen-center. But live-migration fails, an the VMs crashed. I had to stop the other vms on the master and then reboot to get them back online.
After that I shut down the other VMs to savely reboot the other nodes.

So I think, the warning is not fully correct, also without using the rolling-pool-update feature you can get into trouble. Maybe the mentioned toolstack-restart after the update could have helped me, but there was no info about that in the blog post warning ...

stormi

@hecki Thanks for the feedback. I agree that we could try to make it even more obvious how to avoid the issue, but the blog post does already point at the documentation for updating, as always when we publish an update, and the documentation clearly states you need to restart the toolstack (and it's not new... It's just that many people just overlook it and fortunately for them in most cases this is without consequences... Hence bad habits taken): https://xcp-ng.org/docs/updates.html#from-xen-orchestra

AFAIK the next release of Xen Orchestra will either restart the toolstack or remind you to do it when you apply patches.

hecki

@stormi I alwas used the rolling pool update since its available so i did not read the docs since then any more. If I understand you correct, until now the rolling pool update does not do a toolstack restart during the node restarts although it is so important?

stormi

@hecki In the case of Rolling Pool Update it's not a matter of toolstack restart anymore but rather of updating and rebooting hosts one by one rather than all at once.

hecki

@stormi ok, so "Install pool patches" and "Rolling pool update" internally behave completely different. This is new for me, until now, i thought, the Rolling Pool Update is a "Install poool patches" with following evacuation and reboot of the nodes, so thats what I have done manually today. Thanks for the explanation ...

stormi

@hecki That's what it was (with extra steps also such as putting hosts in maintenance mode), but this will change in the next release of Xen Orchestra as Rolling Pool Update will now evacuate, update then reboot every host one by one