XCP-ng 8.2 updates announcements and testing

gskger

@olivierlambert as always, it is a pleasure to help and also easy to do and to contribute. Oh and @BenjiReis - sorry for not replying to your initial post.

BenjiReis

@gskger Thanks for the report!

BenjiReis

Experimental feature: select a network to evacuate an host
A new feature is available in our testing repo: select a network for host evacuation, this would allow to evacuate an host on any (faster) given network instead of the management one.

To access the feature, on all your hosts (always starting with master when in a pool):
yum update --enablerepo=xcp-ng-testing xapi-core-1.249.5-1.1.0.evacnet.1.xcpng8.2.x86_64 xapi-xe-1.249.5-1.1.0.evacnet.1.xcpng8.2.x86_64 xapi-tests-1.249.5-1.1.0.evacnet.1.xcpng8.2.x86_64
And restart your hosts.

WHAT TO TEST
Host evacuation not on the management network (probably a 10G storage network to go faster!)
You can run xe host-evacuate host=<host_uuid> network-uuid=<network_uuid>
Or a XAPI client can call host.evacuate with a network ref parameter.
Host evacuation without the optionnal new parameter should behave as before the update.

Please report here if anything goes wrong (or right hopefully ) and if you spot a regression as well.

Thanks!

Edit: there are no plans for now to add this feature in 8.2 LTS, the package will probably stay in the testing repo for 8.2 and will be available in 8.3. It means that the package would be erased at next xapi update.

stormi

Up! Poor @BenjiReis needs some feedback

gskger

@benjireis Had some time for testing and installed the update on my two host playlab (DELL Optiplex 9010, XCP-ng 8.2.0 fully pathed). This are identical hosts with an onboard NIC (Intel 82579LM, eth0) and a dual port NIC (Intel 82571EB/82571GB, eth1 and eth2). Both are connected to a Synology as shared storage.

Starting with the master, both hosts updated and rebooted without an issue.
Live migration of Debian, Ubuntu, Centos and Windows 10 VMs works (with and without guest tools installed).
Host evacuation over eth0 (Management network) without the new network-uuid parameter works as before.
Host evacuation over eth1 (Storage network) with the new parametrer works at full gigabit speed.

Still working on 10G in my playlab, so i only have 1G available, but host evacuation over a non- management network works with the new network-uuid parameter.

BenjiReis

@gskger Thanks for the report! This is greatly appreciated!

stormi

New update candidates for 8.2, including guest SecureBoot support

Several update candidates are ready for testing.

Description of the changes

The updated uefistored brings guest SecureBoot support. The number one priority it to check that UEFI VMs still work well in various situations (including backups, restore from older backups, fresh install after the update was installed...). For SecureBoot support itself, usage detailed at https://github.com/xcp-ng/xcp-ng-org/pull/85/files currently until we merge the instructions to the official docs. We'll create a dedicated thread to discuss this feature.
Updated XAPI brings the latest fixes from Citrix hotfix XS82E020.
Updated storage manager (sm) brings the latest fixes from Citrix hotfix XS82E023 as well as a fix for a minor regression this hotfix brought (detected by @ronan-a and reported upstream with a patch proposal), an experimental MooseFS driver contributed by the MooseFS developers (not enabled by default), and a fix for NFS SR creation with some QNAP devices (contributed upstream, to the sm project, but still waiting for review after ~4 months).
Updated xsconsole fixes DNS settings management: when changed from the text UI, DNS settings were not saved to the XAPI and were thus lost after a reboot (not contributed upstream], because there's no public git repository for XSConsole unfortunately).
Updated blktap fixes a rare crash in specific situations.
Updated guest tools ISO brings support for new OSes and versions, such as CentoS 8.3+ & Stream, AlmaLinux, Rocky Linux, fixed installation on FreePBX... Those have already been tested in this thread but more tests are always welcome.

How to update (XCP-ng 8.2 only)

yum update blktap forkexecd message-switch rrdd-plugins sm sm-rawhba uefistored xapi-core xapi-tests xapi-xe xcp-ng-pv-tools xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xenopsd xenopsd-cli xenopsd-xc xsconsole --enablerepo=xcp-ng-testing

What to test

The main goal of the testing phase is to avoid regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

If interested, you can also test that what we claim we have fixed is actually fixed, and have a go at guest Secure Boot.

beshleman opened this pull request in xcp-ng/xcp-ng-org

closed Guest secure boot #85

vmpr

@benjireis today I tested it but didn't reboot the host, was working after a toolstack restart. please integrate it into 8.2 LTS, a very useful feature, and we want to stay at 8.2 LTS.

MartinB

@stormi can you please tell us more about this moosefs driver?

Regards,

Martin

stormi

@martinb Not much. I'm waiting for the developers of MooseFS to contribute documentation about how to use it.

gskger

@stormi Did the update on my two host playlab, which worked well. Do not use secureboot/UEFI or QNAP, so this is more a regression test for the usual stuff. I tested Debian, Centos and Ubuntu VMs (create, live migrate with/-out guest tools (now at 7.20.0-8), start/stop/reboot, snapshot with/-out RAM and revert, storage migrate from/to shared and local SR) and restored a Windows 10 and a Debian VM from backup. As for now, everthing is working. I will see how backup runs tonight.

XCP-ng-JustGreat

@stormi Installed all of the test updates on my three-host home-lab this weekend. Similar configuration to @gskger 3 x Dell OptiPlex 7040 SFF hosts and home-built FreeNAS server with separate physical 1Gb networks for management, storage and migration. I call it my "Tiny Cluster" due to its diminutive footprint. I use it for configuration prototyping. Intel VPRO AMT on Xen hosts and storage server enables headless console operation using MeshCommander (think poor man's iDRAC). All updates were installed without issue. Backups and restores seem to work just fine. Of special interest to me was the UEFI Secure Boot capabilities. Installed the x64 dbx.auth from uefi.org (I presume since XCP-ng is 64-bit that that was the correct choice. Probably should be made explicit in the instructions.) Seems to work perfectly. I tested with Windows 10-20H2 and Windows 10-21H1. Also tested with RHEL 8.4 which has built-in support for secure boot (Microsoft-signed bootloader shim) and that too "just works." The varstore-ls <VM-uuid> command shows PK, KEK, dbx and db in the store as expected. Stops unsigned bootloader as expected on unsupported OSes. Looks great! Thank you for all of the work you've put into it. I suspect designing and building emulated system firmware is not for the faint of heart . . . Very impressive!

olivierlambert

Thanks for your feedback @gskger , as usual. For all your prompt help/feedback you always gave here, I really need to do what I said: we'll send you some XCP-ng/XO swag, @Marc-pezin will deal with that soon (he'll contact you in private chat).

@XCP-ng-JustGreat thanks also for your feedback. Indeed, it took us one year of @beshleman work + @stormi integration and @Darkbeldin tests/QA to get something that works

DHowett

I was testing with a main focus on uefistored and the Secure Boot support. I'm happy to report that my one secureboot VM¹ started up with full signature checking and everything. This is with a custom/in-house PK.

Additional test cases:

Export UEFI secureboot VM to OVA and re-importing it: SUCCESS
Copying a secureboot VM within the same pool: SUCCESS

In both cases, the new VM successfully verified the bootloader.

¹ I had loaded my PK, KEK and db and enabled secureboot before the uefistored update, as I was already experimenting with secureboot.

stormi

New security updates (xen + microcode)

These security updates have higher priority than the update train above. You can install them if you had already installed the previous update candidates, or install them without installing the previous update candidates.

Citrix security bulletin: https://support.citrix.com/article/CTX316324

There's a new attack related to speculative code execution, ~~that's why there is updated microcode (both for Intel and AMD)~~ Updated: actually, the microcode update is only for Intel and is not related to this specific attack. Whether your hardware is vulnerable or not depends on various things (model, Xen's strategy against previous vulnerabilities, which may or may not protect you already from the new vulnerability, depending of the hardware...).

Test on XCP-ng 8.2

yum update microcode_ctl xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing

Version for microcode_ctl: 2.1-26.xs15.xcpng8.2
Version for xen packages: 4.13.1-9.11.1.xcpng8.2

What to test

The main goal is to avoid obvious regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

Between 24h and 36h.

JeffBerntsen

@stormi No regressions so far on my test pool with both sets of test updates installed.

jmccoy555

@stormi A bit late to the party again (must try harder ) as I have been moving my rack and my test host was not set up and main pool down to running on 2 hosts..... taking one more offline would make Ceph very unhappy!!!....

Anyway, both test updates applied to my test host and I haven't managed to break anything yet!!! So looks good from my point of view.

stormi

Many thanks for the prompt feedback on the security updates everyone!

I've pushed the release button (well, actually I ran koji move v8.2-testing v8.2-updates xen-4.13.1-9.11.1.xcpng8.2 microcode_ctl-2.1-26.xs15.xcpng8.2. Don't try this at home.), and the security updates will be available within 5 minutes, identical to what you have tested.

I have not released the rest of the update train that is being tested (see this post), so let the testing continue!

stormi

Security and bugfix updates to be released soon, please test!

So, previously some of you had tested a batch of updates (see https://xcp-ng.org/forum/post/39925).

It is going to be released very soon, along with a new security update and some other changes.

What changed sinced the last tests

QEMU was updated to fix security issues (cf. https://support.citrix.com/article/CTX316325)
The guest tools ISO was updated to fix a small display issue (replaced @BRAND_GUEST@ with Virtual Machine in the initscript metadata)
sm was updated again to fix a regression in the previous update candidate

You can still test the new Secure Boot support, but it won't be released in the next batch of updates. We still have work to do to fix some issues, the main one being that the XCP-ng guest drivers for Windows need to be re-signed so that Windows accepts them when SB is enabled. Without this, SB + XCP-ng guest tools = unbootable VM. If you don't enable secure boot on them, the uefistored update is not supposed to change anything for your UEFI VMs.

Reminder of the previous unreleased changes

The updated uefistored brings guest SecureBoot support. The number one priority it to check that UEFI VMs still work well in various situations (including backups, restore from older backups, fresh install after the update was installed...). For SecureBoot support itself, usage detailed at https://github.com/xcp-ng/xcp-ng-org/pull/85/files currently until we merge the instructions to the official docs. We'll create a dedicated thread to discuss this feature.
Updated XAPI brings the latest fixes from Citrix hotfix XS82E020.
Updated storage manager (sm) brings the latest fixes from Citrix hotfix XS82E023 as well as a fix for a minor regression this hotfix brought (detected by @ronan-a and reported upstream with a patch proposal), an experimental MooseFS driver contributed by the MooseFS developers (not enabled by default), and a fix for NFS SR creation with some QNAP devices (contributed upstream, to the sm project, but still waiting for review after ~4 months).
Updated xsconsole fixes DNS settings management: when changed from the text UI, DNS settings were not saved to the XAPI and were thus lost after a reboot (not contributed upstream], because there's no public git repository for XSConsole unfortunately).
Updated blktap fixes a rare crash in specific situations.
Updated guest tools ISO brings support for new OSes and versions, such as CentoS 8.3+ & Stream, AlmaLinux, Rocky Linux, fixed installation on FreePBX... Those have already been tested in this thread but more tests are always welcome.

How to update (XCP-ng 8.2 only)

yum update blktap forkexecd message-switch qemu rrdd-plugins sm sm-rawhba xapi-core xapi-tests xapi-xe xcp-ng-pv-tools xcp-ng-release xcp-ng-release-config xcp-ng-release-presets xenopsd xenopsd-cli xenopsd-xc xsconsole --enablerepo=xcp-ng-testing

If you also want to test the secure boot support (won't be released with the rest yet):

yum update uefistored --enablerepo=xcp-ng-testing

What to test

The main goal of the testing phase is to avoid regressions, so test whatever you want. The closer to your actual use of XCP-ng, the better.

If interested, you can also test that what we claim we have fixed is actually fixed, and have a go at guest Secure Boot.

Test window before official release of the updates

Official release due on monday.

beshleman opened this pull request in xcp-ng/xcp-ng-org

closed Guest secure boot #85

gskger

@stormi Again more a regression test for the basic things. Tested Debian and Ubuntu VMs (create, live migrate with/-out guest tools (now at 7.20.0-9), start/stop/reboot, snapshot with/-out RAM and revert, storage migrate from/to shared and local SR). Imported Centos and Ubuntu VMs and restored a Windows 10 and a Debian VM from backup. No issues so far. Nice work.