Guest UEFI Secure Boot on XCP-ng

stormi

I'm opening this thread dedicated to the upcoming guest secure boot feature on XCP-ng, that some of you already started testing here.

We will soon enter the final testing stages before release. I will post an answer to this thread when it's read, with all the details.

stormi

The new updates that add Guest Secure Boot support are now available for testing.

It brings SB support and also fixes the installation of update "KB4535680" on Windows server 2019.

How to install on XCP-ng 8.2

Install with:

yum update uefistored varstored-tools --enablerepo=xcp-ng-testing

No reboot required.

Revert if needed with:

yum downgrade uefistored varstored-tools

Documentation

The feature is documented here: https://xcp-ng.org/docs/guides.html#guest-uefi-secure-boot

Feedback is welcome on the docs too: is it clear how to enable SB for VMs on XCP-ng?

What you can test

Check the docs for how to do concretely.

Pool setup for SB
Existing UEFI VMs still running well, no regressions
Enabling SB for a new VM
Enabling SB for an existing VM
Manipulating the certs for a given VM

Linux and Secure Boot

Due to certificate revocations last year and this year (caused by exploits in grub and/or shim), the current situation with Linux and SB is complicated. With the latest dbx (revocated certs) file from uefi.org, you won't be able to boot most distros with SB on.

To make it work (which makes little sense until distros can sign new binaries again and publish updates, but is useful to us as far as testing is concerned), you can either install an older dbx, which leaves your VMs vulnerable to the latest grub/shim vulnerabilities. Or to use no dbx at all, which is even worse.

I suggest to install the latest dbx pool-wide, and to modify the dbx variable on linux VMs that you want to enable SB on:

download an older dbx: https://uefi.org/revocationlistfile/archive

install it for a VM

# Run this on an XCP-ng host.
# "d719b2cb-3d3a-4596-a3bc-dad00e67656f dbx" is the identifier of the variable we'll set and 0x27 the attributes
varstore-set {VM-UUID} d719b2cb-3d3a-4596-a3bc-dad00e67656f dbx 0x27 {name-of-dbx-file}

jhester7097

@stormi

I am not able to find the uefistored package. Any reason it wouldn't be showing up?

[19:40 xcphost ~]$ yum search uefistored --enablerepo=xcp-ng-testing
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-testing: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Warning: No matches found for: uefistored
No matches found

olivierlambert

Are you using XCP-ng 8.2, right?

apz

@stormi Does this system change or break backups, either in XO or xe vm-export?

apz

@stormi said in Guest UEFI Secure Boot on XCP-ng:

Feedback is welcome on the docs too: is it clear how to enable SB for VMs on XCP-ng?

Some test feedback:

Documentation was clear enough to get it right on the first try.

Tested on a WS 2019 virtual that has the KB4535680 issue, after enabling SB the updates installed without problems and a test clone of the virtual is now running isolated for a longer period test.

olivierlambert

@apz said in Guest UEFI Secure Boot on XCP-ng:

@stormi Does this system change or break backups, either in XO or xe vm-export?

No it's not related at all.

ASUSEagle

@stormi

I'm having issues with a Windows Server 2019 VM. I am running XCP-ng 8.2 with Xen Orchestra. After enabling secure boot the VM boots to Windows Recovery and will not boot to Windows Server. The VM will boot fine with secure boot off. These are the steps I followed is there something I am missing?

# yum update uefistored varstored-tools --enablerepo=xcp-ng-testing

# secureboot-certs install default default default latest

# varstore-sb-state d4960d10-e6dc-4bf7-daf4-5684c66cdb9e setup

I then enabled secure boot through Xen Orchestra and started the VM.

Update:
There appears to be some problem with a driver signatures. The VM would not boot into safe mode but I was able to get it to boot by disabling driver signature enforcement. I've run sigverif.exe and it show all drivers are signed so I'm not sure what to do from here.

Also I forgot to mention earlier, the VM is set for UEFI firmware and is running the XCP guest tools. This is an existing VM that was setup with secure boot off I'm trying to turn it on so I can install KB4535680.

stormi

@asuseagle I realize that I forgot to mention it at the beginning of this thread: the guest drivers from XCP-ng are indeed signed in a way that does not please Windows when driver signature enforcement is on, which happens automatically when SB is enabled.

How did you disable driver signature enforcement? This would be a useful trick until we can provide new signed drivers (in progress).

This is an existing VM that was setup with secure boot off I'm trying to turn it on so I can install KB4535680.

I don't think you need to enable SB in order to be able to install KB4535680. In any case, it should not fail anymore.

ASUSEagle

@stormi

I disabled it through advanced boot options. It does not persist through a restart so it has to be set every time the VM boots. The VM also took significantly longer to restart. It sits at the firmware splash screen for 30 minutes to an hour then boots the rest of the way in about a minute like normal. While sitting at the splash screen CPU0 utilization is hover around 90% while CPU1-3 are at 0%, memory is showing max at the 16GB I've assigned, and disk throughput is at 2-3 MiB(r) after an initial spike of 28 MiB(r).

These are the steps I took to boot with driver signature enforcement disabled:

Once secure boot has been enabled, I start the VM. The VM fails to start Windows and boots to Windows Recovery.
On the Windows Recovery screen I select "Troubleshoot".
Then I select "Startup Settings", this brings up the "Advanced Boot Options" screen.
I then key down and select "Disable Driver Signature Enforcement"

Once I've selected "Disable Driver Signature Enforcement" the VM restarts and hangs at the firmware splash screen as I described above before finally booting to Windows.

I didn't think I would have needed secure boot to install KB4535680 either but for some reason it would fail to install until I turned secure boot on. Thanks for the info on the guest drivers, I'll have to keep an eye out for when the signed drivers are released.

JurgenDM

I confirm that installing this patch alone is not enough, also tried rebooting both the OS and XCP-NG.
Even after the the update the patch failes to install with error code 0x800f0922.
I've not tried enabling Secure Boot since it will probably cause issues with the installed guest drivers.

onyxfire

@stormi Do we have any timing or plans on getting the XCP guest drivers signed properly? This would be essential to ever being able to break fully away from citrix tools. Is there another thread tracking that progress and technical roadblocks?

olivierlambert

We are making progress on getting our EV certificates.

XCP-ng-JustGreat

@olivierlambert @stormi Here's a good link for persistent disablement of driver signature checking on Windows using bcdedit https://blog.pcrisk.com/windows/12194-how-to-disable-driver-signature-enforcement that may help those above wanting to use the XCP-ng drivers. If for some reason that doesn't work, they can, of course, use the signed Citrix drivers as a stopgap measure.

beshleman

@xcp-ng-justgreat The issue I ran into was that bcdedit can't modify testsigning when secure boot is enabled, and setting testsigning before enabling secure boot resulted in an issue I can't quite recall, but I think it was a broken boot.

noship

@beshleman Have Vates fixed the problem with Secure Boot issue Yet

stormi

@noship Which issue are you talking about exactly?

rjt

@stormi Installation of MS KB4535680 is failing for us as well as many others. To be clear, we have not downloaded the latest patches mentioned in XOA. Will simply installing the latest ca-certificates (dated Sept 14, 2021) and updated grub-efi (dated June 29, 2021) along with the other updates such as xcp-ng-release-config 8.2.0-8 fix this or do we have to manually make changes mentioned above as well?

I expect having Windows reboot into UEFI and configuring the UEFI to attempt secure boot will be necessary no matter what. Meeting with a software vendor later today and they may ask why MS KB4535680 is not installed.

stormi

@rjt You still need varstored-tools and uefistored from the testing repository.

rjt

Guest UEFI Secure Boot looks like a great guide. Everyone needs to read the "Boothole and fallouts" section.