XCP-ng 8.2.1 (maintenance update) - ready for testing

stormi

[Message updated on January 31st]

What is XCP-ng 8.2.1?

It's just an update to XCP-ng 8.2. It's bigger than previous security or bugfix updates we released, but it's the same principle. If you're already running XCP-ng 8.2, you'll just update the same way as usual.

Then why is it called XCP-ng 8.2.1?

Two reasons:

Among other changes, we included those from Citrix Hypervisor CU1, and the official version number changed to 8.2.1 there. You will see 8.2.1 instead of 8.2.0 as the minor version number in xsconsole, for example. But for me it's just an up to date 8.2.
At the same time as we release updates for XCP-ng 8.2, we release new installation ISOs that contain all the updated packages up to now. And also use it as an opportunity to fix a few bugs in the installer (anyone got stuck on the blue installation screen after choosing software RAID with 8.2 installation ISOs? ). So in a way, it's also a real release.

Is it an optional update, like Citrix Hypervisor 8.2 CU1 is?

No. We don't maintain two separate branches of XCP-ng 8.2. After sufficient internal and community testing, it will be the XCP-ng 8.2.

Are there new fancy features?

Well, it's a maintenance release so don't expect too much.

However:

Secure boot for VMs is now supported (full documentation here). Some of you already tested this feature months ago. Now it's time for it to reach everyone. There's a but: XCP-ng's guest tools are not signed with a recent enough certificate and Microsoft's user support is so bad that we haven't been able to get a new signing certificate in months (there are issues with their own website that they have trouble finding a workaround for)! So for now the basic rule is: if you want to enable Secure Boot on a Windows VM, use guest tools from Citrix.
A few components like qemu are updated to pave the way to future vTPM support. I said future. It's not available yet.
Rocket Lake processors are now supported.

Other notable changes

Guest template for Windows Server 2022 added.
Log rotation. Log files should now be automatically rotated if they reach a size of 100M, without waiting for the daily log rotation. This should better handle the situations where a single log file grows up very fast to the point of filling the log partition.
Updated default drivers on the system:
- avago-mpt3sas-33.100.00.01
- intel-ice-1.6.4 (new RPM. We were previously using the built-in driver from kernel 4.19)
- intel-ixgbe-5.9.4
- qlogic-fastlinq-8.42.10.0
- qlogic-netxtreme2-7.14.76
The default console menu, xsconsole, was updated and includes an improvement that we had contributed upstream: when the XAPI service is unreachable on the host, xsconsole will try to display a useful error message, rather than displaying a misleading message saying that no network was configured.
A bug that we discovered and reported upstream regarding the handling of web pages over HTTPS on the host when HTTP support was forbidden has been fixed, so I could finally enforce HTTPS for the host's web page. Any request to get the web page on port 80 will reply with a 403 error.
samba and openssl were updated, which fixes various CVEs. The update to the samba packages pulled several new dependencies such as gnutls, nettle, python-tdb, ...
xenserver-transfer-vm was removed by Citrix.

How to test

Either install or upgrade a host using the test installation ISO found at mirrors.xcp-ng.org/tmp/xcp-ng-8.2.1-test1.iso. Oh, don't try the netinstall yet .

Or update an existing XCP-ng 8.2 host:

create a file named xcp-ng-staging.repo.

[xcp-ng-staging]
name=XCP-ng Staging Repository
baseurl=http://mirrors.xcp-ng.org/8/8.2/staging/x86_64/ http://updates.xcp-ng.org/8/8.2/staging/x86_64/
enabled=0
gpgcheck=1
repo_gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-xcpng

Update: yum update --enablerepo=xcp-ng-staging
Usual instructions from https://xcp-ng.org/docs/updates.html still apply

In both cases, be ready for future updates (yum update --enablerepo=xcp-ng-staging). I've got already at least two minor fixes to do.

What to test

As usual, anything that you need XCP-ng for.

We also would like you to give special focus to the following items:

UEFI VMs, without Secure Boot
UEFI VMs, with Secure Boot (check the docs. There's a manual command to run once on the pool, to download and install the certificates from Microsoft.)
On Windows installed from a not too recent image (otherwise the test is impossible), installation of update KB4535680, which updates the list of revocated certificates for Secure Boot. Should work without Secure Boot on, but we had reports of failures in this situation so I'm interested in finding a way to reproduce. Should also work with Secure Boot on.
Log rotation if you have a way to trigger very verbose logs.
The installer (installation, upgrade, backup restore...).

Ready... Test!

And of course, ask anything.

JeffBerntsen

@stormi
Is it safe to install this on a machine where I'm already testing XOSTOR?

Andrew

@stormi I installed staging updates on a few running machines and it's good so far. No errors or strange issue. Running VMs, doing replication, etc.

I did a new install from the new ISO to an external USB SSD (as a test) and it's working. The volume name of the install ISO needs to be updated to 8.2.1 (from 8.2.0).

I see you are using "http" for the repo, what about "https" for better security?

stormi

@jeffberntsen probably not. We'll need to rebuild some packages, like sm on top of the latest versions else you will lose needed specific patches that are not merged in the main branch yet.

CC @ronan-a

stormi

@andrew said in XCP-ng 8.2.1 (maintenance update) - ready for testing:

@stormi I installed staging updates on a few running machines and it's good so far. No errors or strange issue. Running VMs, doing replication, etc.

I did a new install from the new ISO to an external USB SSD (as a test) and it's working. The volume name of the install ISO needs to be updated to 8.2.1 (from 8.2.0).

I see you are using "http" for the repo, what about "https" for better security?

The security relies on GPG signatures of metadata and RPMs so HTTPS is not strictly required. See https://xcp-ng.org/docs/mirrors.html#security. We might switch to HTTPS by default for RPM repositories at some point, but this will exclude a few mirrors.

gskger

@stormi Reinstalled my playlab from the test installation ISO and will do some testing tomorrow. Poking around with some imported VMs showed now errors so far, but real tests are pending for the weekend . Most likely much to early but looking good so far. Keep fingers crossed for the weekend

gskger

@stormi I ran some more thorough tests today to complement yesterday's initial tests. So far I tested upgrading fully patched 8.2.0 and fresh install hosts to 8.2.1 from ISO, create, live migrate with/-out guest tools (7.20.0-9), start/stop/reboot, snapshot with/-out RAM and revert, storage migrate from/to shared and local SR, backup and restore as well as import several Linux and Windows VMs. Still looking good .

Andrew

@stormi I upgraded my normal pool from 8.2.0 to 8.2.1 (staging) using yum. It took some work because of the version change my pool master got unhappy with the order I did it. My mistake with the process... I ended up upgrading and rebooting all pool members and then things were good. I abused the upgrade process and things still worked out in the end. No trouble, stuck, damaged, or lost VMs (or other resources). Things are working as they should including shared SR on NFS, ISO on NFS, VxLAN, migration, replication, and S3 delta backups. I'm not testing USB/GPU/pass-thru.

JeffBerntsen

@stormi said in XCP-ng 8.2.1 (maintenance update) - ready for testing:

@jeffberntsen probably not. We'll need to rebuild some packages, like sm on top of the latest versions else you will lose needed specific patches that are not merged in the main branch yet.

CC @ronan-a

That's what I thought but figured it wouldn't hurt to ask.

stormi

I updated the announcement above with details about the changes and on what to focus tests on if possible.

Andrew

@stormi I'm not sure what I was doing at the time....

host.isHyperThreadingEnabled
{
  "id": "b9aaf368-7be4-4b5f-ae9d-867e7e83d1e3"
}
{
  "code": "-1",
  "params": [
    "'module' object has no attribute 'run'",
    "",
    "Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 98, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/hyperthreading.py\", line 14, in get_hyperthreading
    result = run_command(['xl', 'info', 'threads_per_core'])
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 67, in run_command
    res = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
AttributeError: 'module' object has no attribute 'run'
"
  ],
  "call": {
    "method": "host.call_plugin",
    "params": [
      "OpaqueRef:6d554a61-ec51-49b0-b58d-e002ea93ce54",
      "hyperthreading.py",
      "get_hyperthreading",
      {}
    ]
  },
  "message": "-1('module' object has no attribute 'run', , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 98, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/hyperthreading.py\", line 14, in get_hyperthreading
    result = run_command(['xl', 'info', 'threads_per_core'])
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 67, in run_command
    res = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
AttributeError: 'module' object has no attribute 'run'
)",
  "name": "XapiError",
  "stack": "XapiError: -1('module' object has no attribute 'run', , Traceback (most recent call last):
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 98, in wrapper
    return func(*args, **kwds)
  File \"/etc/xapi.d/plugins/hyperthreading.py\", line 14, in get_hyperthreading
    result = run_command(['xl', 'info', 'threads_per_core'])
  File \"/etc/xapi.d/plugins/xcpngutils/__init__.py\", line 67, in run_command
    res = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)
AttributeError: 'module' object has no attribute 'run'
)
    at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202201310821/packages/xen-api/src/_XapiError.js:16:12)
    at /opt/xo/xo-builds/xen-orchestra-202201310821/packages/xen-api/src/transports/json-rpc.js:41:27
    at AsyncResource.runInAsyncScope (node:async_hooks:199:9)
    at cb (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/util.js:355:42)
    at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/promise.js:649:10)
    at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/promise.js:729:18)
    at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/async.js:93:12)
    at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/async.js:86:9)
    at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/async.js:102:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202201310821/node_modules/bluebird/js/release/async.js:15:14)
    at processImmediate (node:internal/timers:464:21)
    at process.topLevelDomainCallback (node:domain:152:15)
    at process.callbackTrampoline (node:internal/async_hooks:128:24)"
}

stormi

@andrew Thanks. We'll investigate.

Andrew

@stormi I did a fresh install from the new ISO... after reboot I get an error reported in user.log.

Feb  2 18:19:39 xcp4 kdump: Loaded crash kernel
Feb  2 18:19:43 xcp4 fcoe_driver INFO: eth0 is FCoE capable
Feb  2 18:19:43 xcp4 fcoe_driver INFO: eth1 is FCoE capable
Feb  2 18:19:43 xcp4 fcoe_driver CRITICAL:
Feb  2 18:19:43 xcp4 fcoe_driver CRITICAL: ['Traceback (most recent call last):\n', '  File "/opt/xensource/libexec/fcoe_driver", line 34, in execute\n    output = subprocess.check_output(cmd)\n', '  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output\n    raise CalledProcessError(retcode, cmd, output=output)\n', "CalledProcessError: Command '['fcoeadm', '-i']' returned non-zero exit status 2\n"]
Feb  2 18:19:43 xcp4 fcoe_driver INFO: Applying config on interface: eth0
Feb  2 18:19:44 xcp4 fcoe_driver INFO: Applying config on interface: eth1

Andrew

This post is deleted!

Andrew

@stormi Here's more info from the xensource.log

I found this error happens when you use XO, click on a HOST and then the ADVANCED tab.

Feb  2 18:40:32 xcp4 xapi: [debug||741 HTTPS 192.168.1.131->:::80|host.get_sched_gran R:cdd533230ce9|audit] Host.get_sched_gran: host='a87516dc-1363-450d-8384-10e9e4a131b4 (xcp4)'
Feb  2 18:40:32 xcp4 xapi: [debug||741 HTTPS 192.168.1.131->:::80|host.get_sched_gran R:cdd533230ce9|helpers] about to call script: /opt/xensource/libexec/xen-cmdline
Feb  2 18:40:32 xcp4 xapi: [debug||742 HTTPS 192.168.1.131->:::80|host.call_plugin R:4f64bd0de6ba|audit] Host.call_plugin host = 'a87516dc-1363-450d-8384-10e9e4a131b4 (xcp4)'; plugin = 'hyperthre
ading.py'; fn = 'get_hyperthreading' args = [ 'hidden' ]
Feb  2 18:40:32 xcp4 xapi: [ warn||740 HTTPS 192.168.1.131->:::80|event.from D:b61f8cdc98d8|xapi_message] get_since_for_events: no in_memory_cache!
Feb  2 18:40:32 xcp4 xapi: [debug||741 HTTPS 192.168.1.131->:::80|host.get_sched_gran R:cdd533230ce9|helpers] /opt/xensource/libexec/xen-cmdline --get-xen sched-gran succeeded [ output = '\x0A' ]
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] host.call_plugin R:4f64bd0de6ba failed with exception Server_error(-1, [ 'module' object has no attribute 'run'; ; Traceback (most recent call last):\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 98, in wrapper\x0A    return func(*args, **kwds)\x0A  File "/etc/xapi.d/plugins/hyperthreading.py", line 14, in get_hyperthreading\x0A    result = run_command(['xl', 'info', 'threads_per_core'])\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 67, in run_command\x0A    res = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)\x0AAttributeError: 'module' object has no attribute 'run'\x0A ])
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] Raised Server_error(-1, [ 'module' object has no attribute 'run'; ; Traceback (most recent call last):\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 98, in wrapper\x0A    return func(*args, **kwds)\x0A  File "/etc/xapi.d/plugins/hyperthreading.py", line 14, in get_hyperthreading\x0A    result = run_command(['xl', 'info', 'threads_per_core'])\x0A  File "/etc/xapi.d/plugins/xcpngutils/__init__.py", line 67, in run_command\x0A    res = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=True)\x0AAttributeError: 'module' object has no attribute 'run'\x0A ])
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 1/6 xapi Raised at file ocaml/xapi/rbac.ml, line 231
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 2/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 103
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 3/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 121
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 4/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 5/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace] 6/6 xapi Called from file lib/backtrace.ml, line 177
Feb  2 18:40:32 xcp4 xapi: [error||742 :::80||backtrace]
Feb  2 18:40:32 xcp4 xapi: [ warn||743 HTTPS 19.168.1.131->:::80|event.from D:6e6288e090db|xapi_message] get_since_for_events: no in_memory_cache!
Feb  2 18:40:32 xcp4 xapi: [ warn||744 HTTPS 192.168.1.131->:::80|event.from D:6c41ed917a6a|xapi_message] get_since_for_events: no in_memory_cache!

stormi

@andrew said in XCP-ng 8.2.1 (maintenance update) - ready for testing:

@stormi I did a fresh install from the new ISO... after reboot I get an error reported in user.log.

Feb  2 18:19:39 xcp4 kdump: Loaded crash kernel
Feb  2 18:19:43 xcp4 fcoe_driver INFO: eth0 is FCoE capable
Feb  2 18:19:43 xcp4 fcoe_driver INFO: eth1 is FCoE capable
Feb  2 18:19:43 xcp4 fcoe_driver CRITICAL:
Feb  2 18:19:43 xcp4 fcoe_driver CRITICAL: ['Traceback (most recent call last):\n', '  File "/opt/xensource/libexec/fcoe_driver", line 34, in execute\n    output = subprocess.check_output(cmd)\n', '  File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output\n    raise CalledProcessError(retcode, cmd, output=output)\n', "CalledProcessError: Command '['fcoeadm', '-i']' returned non-zero exit status 2\n"]
Feb  2 18:19:43 xcp4 fcoe_driver INFO: Applying config on interface: eth0
Feb  2 18:19:44 xcp4 fcoe_driver INFO: Applying config on interface: eth1

Is this something you'd reproduce with XCP-ng 8.2 at first boot?

Andrew

@stormi No, I did not see it before. Also it's not an error on hosts upgraded from 8.2.0 to 8.2.1.

stormi

@andrew said in XCP-ng 8.2.1 (maintenance update) - ready for testing:

@stormi Here's more info from the xensource.log

I found this error happens when you use XO, click on a HOST and then the ADVANCED tab.

We reproduced and will fix this one. Thanks!

stormi

@andrew said in XCP-ng 8.2.1 (maintenance update) - ready for testing:

@stormi No, I did not see it before. Also it's not an error on hosts upgraded from 8.2.0 to 8.2.1.

At first I don't see what could have changed here. Does the error appear to have consequences?

Andrew

@stormi I'll say no consequences because I don't use FCoE. It looks like a timing issue with exactly when the script runs and the status of the ethernet interfaces. As I was looking at the code and the error vanished so now runs correctly (without changes).