XCP-ng 8.2 updates announcements and testing

gskger

stormi Updating my homelab (3x R210II, 2x R720) to 8.2.1 worked without a problem (apart from the fact that XO produced some hiccups with the rolling update). Nice

olivierlambert

gskger I think we might have the same. Do you have an error log?

gskger

olivierlambert I haven't looked for error logs (yet), but if you tell me which logs are interesting or what to look for, I'll be happy to give it a try.

stormi

gskger mainly xensource.log, possibly daemon.log.

olivierlambert

I thought about XO Logs (in Settings/Logs).

gskger

olivierlambert stormi I will send you what I have tomorrow if it is not that urgent, ok?

olivierlambert

No rush at all Thanks a lot already for giving us that valuable feedback.

Andrew

olivierlambert Install patches from from XO source (current master) and 8.2.1. It works, but I an error.

Host install:

pool.installPatches
{
  "hosts": [
    "64b9bf4a-a991-438d-973f-60179f3d0868"
  ]
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

Pool install:

pool.installPatches
{
  "pool": "98ab0129-f9d1-ba08-c1c0-aa93c38d7fec"
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:152:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

gskger

olivierlambert stormi I started the rolling update from XO and left it running alone since it usually takes some time to complete. This xenopsd internal error is from the XO log.

2022-02-28T15_21_32.891Z - XO.log.txt

An hour later I realized that the updates had been applied, but the hosts did not reboot. I tried to reboot the master from XO first which might have resulted in this XO log.

2022-02-28T16_35_04.612Z - XO.log.txt

After that all VMs including XO became unresponsive and I continued to investigate with XCP-ng Center. In the end, I decided to reboot the hosts from CLI (master first and than all slaves). That resolved the issue and XO came up without any further problems.

I have copies of the logs stormi suggested from the master and one slave if they are needed. XO is XO from source by 3rd party script. I did an XO update after resolving the issue, so if you need the XO version, I can restore from backup and check.

Hope that helps.

olivierlambert

Andrew I believe this is a problem on XO, unrelated to the XCP-ng update itself.

gskger so we had the same issue. However, I did something differently: after restarting the toolstack, I could migrate VMs around without any problem, then reboot manually each evacuated host.

gskger

olivierlambert Yes, restarting the toolstack first would have been a better approach. Is this something that needs additional actions on the hosts?

olivierlambert

No, that should be enough

stormi

We added a warning on the blog post https://xcp-ng.org/blog/2022/02/28/xcp-ng-8-2-1-update/ and will work on a fix in XO.

HeMaN

stormi
now realise I had the same behaviour after applying the latest 8.2.0 security updates a few weeks ago.
At that time thought it was because I patched the master manually already (test repo release) and updated the entire pool only after the official release 2 days later (roling pool update).

So it is not (only) related to the 8.2.1 release

stormi

HeMaN I think it's more likely to happen with the 8.2.1 update because more components are updated. Else I agree.

stormi

New security updates (xen, Intel microcode)

Impact: newly disclosed security vulnerabilities may allow attackers from within a guest to access memory of the host or other VMs. Only AMD CPUs are affected. To know if your specific model is affected, consult the vendor publications. Xen has been patched to mitigate the hardware issue.

Additionnally but unrelated to the security issue, firmware is updated for some models.

Citrix' security bulletin: https://support.citrix.com/article/CTX341586

Test on XCP-ng 8.2

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update linux-firmware microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.4-9.20.1.xcpng8.2
microcode_ctl: 2.1-26.xs20.xcpng8.2
linux-firmware-20190314-3.xcpng8.2

What to test

Normal use and anything else you want test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days. Maybe less.

Andrew

stormi Manual updates done on all my 8.2.1 machines and things are working correctly so far.

olivierlambert

Same thing on my EPYC machine.

NielsH

Thanks!

I see there's also another XSA available; XSA-396 (https://xenbits.xen.org/xsa/ )
Will these be combined with the microcode patches ?

Cheers!

stormi

NielsH XSA-396 does not affect XCP-ng as the device back-ends are in dom0 directly, not in driver domains.