XCP-ng 8.2 updates announcements and testing

Andrew

You mean xen 4.13.4, right? That's what's I installed.

I updated a few active machines and no problems so far:
HP G5 (E5450), HP G8 (E5-2680), AMD (Opt 6380)

stormi

@andrew said in Updates announcements and testing:

You mean xen 4.13.4, right? That's what's I installed.

Indeed. I fixed it in the message, thanks.

stormi

Thanks @gskger and @Andrew (and of course Olivier) for the tests.

We released the update on monday: https://xcp-ng.org/blog/2022/01/17/january-2022-security-update/

If for any reason you still had (unsupported since 8.1) 32 bit PV guests, read carefully.

stormi

New security updates (Intel microcode, Xen)

Impact: Vulnerabilities in Xen -> privileged code in a guest VM may crash a host under some conditions (PV guests / PCI-passthrough). Vulnerabilities in Intel CPUs have consequences... That I'm not able yet to describe precisely in the context of XCP-ng and virtualization in general. Looks bad enough: privilege escalation and DoS.

See https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00528.html and https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00532.html

Citrix' security bulletin: https://support.citrix.com/article/CTX337526

Test on XCP-ng 8.2

You can test even if your host was updated with the test packages for XCP-ng 8.2.1.

yum clean metadata --enablerepo=xcp-ng-testing
yum update microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: 4.13.4-9.19.1.xcpng8.2
microcode_ctl: 2.1-26.xs19.xcpng8.2

What to test

Normal use and anything else you want test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~2 days. Maybe less.

HeMaN

@stormi

I get an error on the xcp-ng-testing repository

[14:14 xcp-ng-01 ~]# yum clean metadata --enablerepo=xcp-ng-testing
Loaded plugins: fastestmirror
Cleaning repos: dell-system-update_dependent dell-system-update_independent xcp-ng-base xcp-ng-testing xcp-ng-updates
10 metadata files removed
11 sqlite files removed
0 metadata files removed
[14:16 xcp-ng-01 ~]# yum update microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-testing: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
dell-system-update_dependent                                                                                                                                           | 2.3 kB  00:00:00
dell-system-update_independent                                                                                                                                         | 2.3 kB  00:00:00
xcp-ng-testing/signature                                                                                                                                               |  473 B  00:00:00
xcp-ng-testing/signature                                                                                                                                               | 3.0 kB  00:00:00 !!!
xcp-ng-testing/primary_db      FAILED
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found-:--:-- ETA
Trying other mirror.
To address this issue please refer to the below wiki article

https://wiki.centos.org/yum-errors

If above article doesn't help to resolve this issue please use https://bugs.centos.org/.

(1/3): dell-system-update_dependent/7/x86_64/primary_db                                                                                                                |  25 kB  00:00:00
(2/3): dell-system-update_independent/primary_db                                                                                                                       | 132 kB  00:00:01
xcp-ng-testing/primary_db      FAILED
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found-:--:-- ETA
Trying other mirror.
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.


 One of the configured repositories failed (XCP-ng Testing Repository),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo=xcp-ng-testing ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable xcp-ng-testing
        or
            subscription-manager repos --disable=xcp-ng-testing

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=xcp-ng-testing.skip_if_unavailable=true

failure: repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2 from xcp-ng-testing: [Errno 256] No more mirrors to try.
http://mirrors.xcp-ng.org/8/8.2/testing/x86_64/repodata/0359c514489ed1674f98d30cf8b500598cae60b2bdc7fb7971b005aae0c691b6-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found

stormi

@heman said in Updates announcements and testing:

yum clean metadata --enablerepo=xcp-ng-testing

Looks like there's a small delay during which mirrorbits hasn't noticed the changed metadata files, and thus redirects you to old filenames. I had taken measures against this (I make it rescan the local repository just after it is updated), but there seems to be situations where it doesn't work.

I restarted the service and now it's good.

HeMaN

@stormi

Installation now went well, no issues noted after the reboot.
Will keep an eye on my system and report back if anything unusual is happening

Andrew

@stormi Are these updates going to staging and be part of 8.2.1 or is this a February 8.2.0 security update before 8.2.1 ?

stormi

@andrew There will be a security update before 8.2.1.

stormi

Security update published. Thanks for testing! https://xcp-ng.org/blog/2022/02/11/february-2022-security-update/

stormi

The update that we named XCP-ng 8.2.1 is now published!

https://xcp-ng.org/blog/2022/02/28/xcp-ng-8-2-1-update/

gskger

@stormi Updating my homelab (3x R210II, 2x R720) to 8.2.1 worked without a problem (apart from the fact that XO produced some hiccups with the rolling update). Nice

olivierlambert

@gskger I think we might have the same. Do you have an error log?

gskger

@olivierlambert I haven't looked for error logs (yet), but if you tell me which logs are interesting or what to look for, I'll be happy to give it a try.

stormi

@gskger mainly xensource.log, possibly daemon.log.

olivierlambert

I thought about XO Logs (in Settings/Logs).

gskger

@olivierlambert @stormi I will send you what I have tomorrow if it is not that urgent, ok?

olivierlambert

No rush at all Thanks a lot already for giving us that valuable feedback.

Andrew

@olivierlambert Install patches from from XO source (current master) and 8.2.1. It works, but I an error.

Host install:

pool.installPatches
{
  "hosts": [
    "64b9bf4a-a991-438d-973f-60179f3d0868"
  ]
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

Pool install:

pool.installPatches
{
  "pool": "98ab0129-f9d1-ba08-c1c0-aa93c38d7fec"
}
{
  "message": "",
  "name": "Error",
  "stack": "Error: 
    at Xapi._xcpUpdate (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xapi/mixins/patching.mjs:323:15)
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processImmediate (node:internal/timers:437:9)
    at process.topLevelDomainCallback (node:domain:152:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)
    at Object.installPatches (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/api/pool.mjs:115:3)
    at Api.callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202202281818/packages/xo-server/src/xo-mixins/api.mjs:307:20)"
}

gskger

@olivierlambert @stormi I started the rolling update from XO and left it running alone since it usually takes some time to complete. This xenopsd internal error is from the XO log.

2022-02-28T15_21_32.891Z - XO.log.txt

An hour later I realized that the updates had been applied, but the hosts did not reboot. I tried to reboot the master from XO first which might have resulted in this XO log.

2022-02-28T16_35_04.612Z - XO.log.txt

After that all VMs including XO became unresponsive and I continued to investigate with XCP-ng Center. In the end, I decided to reboot the hosts from CLI (master first and than all slaves). That resolved the issue and XO came up without any further problems.

I have copies of the logs @stormi suggested from the master and one slave if they are needed. XO is XO from source by 3rd party script. I did an XO update after resolving the issue, so if you need the XO version, I can restore from backup and check.

Hope that helps.