XCP-ng 8.2 updates announcements and testing

olivierlambert

Thanks for your feedback @gskger , as usual. For all your prompt help/feedback you always gave here, I really need to do what I said: we'll send you some XCP-ng/XO swag, @Marc-pezin will deal with that soon (he'll contact you in private chat).

@XCP-ng-JustGreat thanks also for your feedback. Indeed, it took us one year of @beshleman work + @stormi integration and @Darkbeldin tests/QA to get something that works

DHowett

I was testing with a main focus on uefistored and the Secure Boot support. I'm happy to report that my one secureboot VM¹ started up with full signature checking and everything. This is with a custom/in-house PK.

Additional test cases:

Export UEFI secureboot VM to OVA and re-importing it: SUCCESS
Copying a secureboot VM within the same pool: SUCCESS

In both cases, the new VM successfully verified the bootloader.

¹ I had loaded my PK, KEK and db and enabled secureboot before the uefistored update, as I was already experimenting with secureboot.

stormi

New security updates (xen + microcode)

These security updates have higher priority than the update train above. You can install them if you had already installed the previous update candidates, or install them without installing the previous update candidates.

Citrix security bulletin: https://support.citrix.com/article/CTX316324

There's a new attack related to speculative code execution, ~~that's why there is updated microcode (both for Intel and AMD)~~ Updated: actually, the microcode update is only for Intel and is not related to this specific attack. Whether your hardware is vulnerable or not depends on various things (model, Xen's strategy against previous vulnerabilities, which may or may not protect you already from the new vulnerability, depending of the hardware...).

Test on XCP-ng 8.2

yum update microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing

Version for microcode_ctl: 2.1-26.xs15.xcpng8.2
Version for xen packages: 4.13.1-9.11.1.xcpng8.2

What to test

The main goal is to avoid obvious regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

Between 24h and 36h.

JeffBerntsen

@stormi No regressions so far on my test pool with both sets of test updates installed.

jmccoy555

@stormi A bit late to the party again (must try harder ) as I have been moving my rack and my test host was not set up and main pool down to running on 2 hosts..... taking one more offline would make Ceph very unhappy!!!....

Anyway, both test updates applied to my test host and I haven't managed to break anything yet!!! So looks good from my point of view.

stormi

Many thanks for the prompt feedback on the security updates everyone!

I've pushed the release button (well, actually I ran koji move v8.2-testing v8.2-updates xen-4.13.1-9.11.1.xcpng8.2 microcode_ctl-2.1-26.xs15.xcpng8.2. Don't try this at home.), and the security updates will be available within 5 minutes, identical to what you have tested.

I have not released the rest of the update train that is being tested (see this post), so let the testing continue!

stormi

Security and bugfix updates to be released soon, please test!

So, previously some of you had tested a batch of updates (see https://xcp-ng.org/forum/post/39925).

It is going to be released very soon, along with a new security update and some other changes.

What changed sinced the last tests

QEMU was updated to fix security issues (cf. https://support.citrix.com/article/CTX316325)
The guest tools ISO was updated to fix a small display issue (replaced @BRAND_GUEST@ with Virtual Machine in the initscript metadata)
sm was updated again to fix a regression in the previous update candidate

You can still test the new Secure Boot support, but it won't be released in the next batch of updates. We still have work to do to fix some issues, the main one being that the XCP-ng guest drivers for Windows need to be re-signed so that Windows accepts them when SB is enabled. Without this, SB + XCP-ng guest tools = unbootable VM. If you don't enable secure boot on them, the uefistored update is not supposed to change anything for your UEFI VMs.

Reminder of the previous unreleased changes

The updated uefistored brings guest SecureBoot support. The number one priority it to check that UEFI VMs still work well in various situations (including backups, restore from older backups, fresh install after the update was installed...). For SecureBoot support itself, usage detailed at https://github.com/xcp-ng/xcp-ng-org/pull/85/files currently until we merge the instructions to the official docs. We'll create a dedicated thread to discuss this feature.
Updated XAPI brings the latest fixes from Citrix hotfix XS82E020.
Updated storage manager (sm) brings the latest fixes from Citrix hotfix XS82E023 as well as a fix for a minor regression this hotfix brought (detected by @ronan-a and reported upstream with a patch proposal), an experimental MooseFS driver contributed by the MooseFS developers (not enabled by default), and a fix for NFS SR creation with some QNAP devices (contributed upstream, to the sm project, but still waiting for review after ~4 months).
Updated xsconsole fixes DNS settings management: when changed from the text UI, DNS settings were not saved to the XAPI and were thus lost after a reboot (not contributed upstream], because there's no public git repository for XSConsole unfortunately).
Updated blktap fixes a rare crash in specific situations.
Updated guest tools ISO brings support for new OSes and versions, such as CentoS 8.3+ & Stream, AlmaLinux, Rocky Linux, fixed installation on FreePBX... Those have already been tested in this thread but more tests are always welcome.

How to update (XCP-ng 8.2 only)

yum update blktap forkexecd message-switch qemu rrdd-plugins sm sm-rawhba xapi-core xapi-tests xapi-xe xcp-ng-pv-tools xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xenopsd xenopsd-cli xenopsd-xc xsconsole --enablerepo=xcp-ng-testing

If you also want to test the secure boot support (won't be released with the rest yet):

yum update uefistored --enablerepo=xcp-ng-testing

What to test

The main goal of the testing phase is to avoid regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

If interested, you can also test that what we claim we have fixed is actually fixed, and have a go at guest Secure Boot.

Test window before official release of the updates

Official release due on monday.

beshleman opened this pull request in xcp-ng/xcp-ng-org

closed Guest secure boot #85

gskger

@stormi Again more a regression test for the basic things. Tested Debian and Ubuntu VMs (create, live migrate with/-out guest tools (now at 7.20.0-9), start/stop/reboot, snapshot with/-out RAM and revert, storage migrate from/to shared and local SR). Imported Centos and Ubuntu VMs and restored a Windows 10 and a Debian VM from backup. No issues so far. Nice work.

gskger

@stormi what is the best strategy to revert a host with testing updates installed back to the standard, non-testing status? Just wondering, if there is a more simple approach than doing a clean install / revert to backup.

stormi

@gskger the yum history command can be handy to rollback to earlier versions of RPMs that are still available on repositories.

yum downgrade same-list-of-RPMs-that-you-updated can also work most of the time.

Be aware though that RPM transactions are not always meant to be reversible. Replacing files is one thing, but the scriptlets that run after an update are almost never tested backwards. I don't foresee any specific issue, it's a general warning.

AlexD2006

@stormi
Did some Testing over the Weekend too.
Setup with 2 Hosts in a Pool and shared iSCSI-LMV Storage with multipath 8 paths per LUN.
Anything seems to work fine (migrate/import/cross-pool-migrate/snapshots/backups).

Even our longtime Problem (snapshots taking much too long) is getting much better (still not good, but much better).

stormi

The update was released yesterday: https://xcp-ng.org/blog/2021/06/28/summer-security-and-bugfix-updates/

Again, a lot of thanks for the feedback.

As I said earlier, the update for uefistored, which brings guest secure boot support, was not included yet.

stormi

New installation ISO for XCP-ng 8.2

I opened a dedicated thread. Meet you there for the tests.

stormi

A bugfix kernel update available for testing

Based on Citrix's hotfix XS82E030, here's a bugfix kernel update. I don't think it will change much for most hosts, except in some specific cases.

What changed

Previous kernel updates (that fixed network performance issues for FreeBSD and sometimes other VMs), may have reduced the performance in some situation according to Citrix. Based on the patches, it looks like it's related to IRQ affinity and cross-domain networking. Here's the patch: https://github.com/xcp-ng-rpms/kernel/blob/master/SOURCES/0001-xen-events-fix-setting-irq-affinity.patch
Tools that need to make the ioperm syscall were crashing on dom0. For example Supermicro Update Manager (SUM). This should fix it.
An additional dependency was added to the perf RPM (not installed by default) to make it able to do backtraces when you try to run it on binaries in dom0.
A patch fixes CVE-2021-29154 was added. It's not considered a security update because it does not fix an exploitable vulnerability. It's extra defence in depth.

How to update (XCP-ng 8.2 only)

yum update kernel --enablerepo=xcp-ng-testing

Version that should be installed: 4.19.19-7.0.12.1.xcpng8.2

What to test

Installation of the update, normal use, no obvious regressions...

Plus the changes described above if you're in a situation that allows it.

Test window before release

None defined at the moment. As it's not a security update, I'll wait for more updates to be ready before I push the next train officially. But feedback is always useful as soon as it can be provided.

gskger

@stormi A lazy sunday morning, some coffee, a notebook and time for testing. Updated my two host playlab (DELL Optiplex 9010, 8.2 fully patched) and tested Debian VMs and Windows 10 VMs (create, live migrate with/-out guest tools, start/stop/reboot, snapshot with/-out RAM and revert, online-/offline storage migrate from/to shared and local SR). Restored a Windows 10 and a Debian VM from backup. All good so far for this regression test.

JCastang

Hello, @stormi I did a quick test because we encounter in some cases "bad' network perfs on our pools (with Intel 10GB network cards) such as :

VXLAN network are getting max 2.5Gbps of bandwidth
VM to VM network with a BSD firewall in the middle getting max 2.5Gbps of bandwidth

But VM to VM (on same LAN) are getting near 10Gbps.

Testing this new kernel does not change anything on BP for theses cases.
BP still stuck at 2,5Gbps.

stormi

@jcastang AFAIK there's still room for improvement regarding network perfs in *BSD VMs, but I doubt the update was meant to address this. If you want to discuss it further, please create a new thread.

stormi

A bugfix update for USB passthrough

A fix was contributed by @jeremfg to the XAPI project, so that when XAPI calls usb_reset.py with the -r switch it does not fail anymore. This fixes passing through both a PCI device and an USB device to a VM, and could also fix other USB passthrough issues that raised the same error: "usage: usb_reset.py attach [-h] -d DOMID -p PID [-r RESET_ONLY] device\nusb_reset.py attach: error: argument -r: expected one argument\n".

Related thread: https://xcp-ng.org/forum/topic/3594/pci-passthrough-usb-passthrough-does-not-work-together

Related issues:

I have built patched XAPI packages for XCP-ng 8.2 that are available for testing.

How to update (XCP-ng 8.2 only)

yum update xapi-core xapi-tests xapi-xe --enablerepo=xcp-ng-testing
xe-toolstack-restart

You should get version-release 1.249.9-1.2.xcpng8.2

What to test

USB passthrough:

What used to work should still work
What did not work, if the error message in the logs is the one quoted above, should now work... Or maybe fail further in cases that haven't been tested yet?

Test window before release

None defined at the moment. As it's not a security update, I'll wait for more updates to be ready before I push the next train officially. But feedback is always useful as soon as it can be provided

AdrianFretwell created this issue in xcp-ng/xcp

closed Starting a VM with USB passthrough results in Internal Server Error #108

jeremfg created this issue in xcp-ng/xcp

closed USB+PCI Passthrough combined, vm boot fails #511

stormi

I created a dedicated thread for the testing of Guest Secure Boot support: https://xcp-ng.org/forum/post/41541

See you there.

And a reminder about the current update candidates that are in testing in the current thread:

... with my interest being geared mainly towards regression testing.

stormi

New toolstack (XAPI) update candidate for 8.2

Based on Citrix's hotfix XS82E031. Also includes the USB passthrough bugfix mentioned above in this thread.

What changed

Fixes

USB passthrough fix
Reduced log verbosity for stunnel, to prevent xapi-ssl.log from growing so fast that it could fill the /var/log partition entirely in some cases (users from HA-lizard had reported such behaviour and we were waiting for this upstream fix from the XAPI project to be included in a hotfix).
Fix xe vm-reset-powerstate for VMs with GPU passthrough that are stuck in paused state.
Other fixes for rare issues (see hotfix description).

Features

The hotfix from Citrix also comes with a few improvements. Quoting (the highlight in bold characters is mine):

Adds a default boot order for VMs.

Improved error messaging for when the graphics card is not configured for SRIOV.

Remove HTTP access to the management network static web page. This web page can now only be accessed through HTTPS.

Adds additional snapshots of glocktop data to the bugtool output.

How to update (XCP-ng 8.2 only)

yum clean metadata --enablerepo=xcp-ng-testing
yum update forkexecd message-switch xapi-core xapi-tests xapi-xe xcp-rrdd xenopsd xenopsd-cli xenopsd-xc --enablerepo=xcp-ng-testing
xe-toolstack-restart # or reboot to be 100% sure everything restarted

What to test

No obvious regressions, and if possible the changes described above.

Test window before release

A few weeks, but the quicker the better, as this kind of message tends to be forgotten over time.

Other update candidates still in testing phase

The kernel bugfix update is still in testing phase. Many thanks to those who already tested it, and I'm still eager for feedback from others: https://xcp-ng.org/forum/post/41241