secureboot-certs install fails

apz

This is a fresh 8.2 system with the secure boot features added per XCP-ng documentation. The secureboot-certs install however fails:

# secureboot-certs install

No arguments provided to command install, default arguments will be used:
- PK: default
- KEK: default
- db: default
- dbx: latest

Downloading https://www.microsoft.com/pkiops/certs/MicCorKEKCA2011_2011-06-24.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicCorUEFCA2011_2011-06-27.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicWinProPCA2011_2011-10-19.crt...
Downloading https://uefi.org/sites/default/files/resources/dbxupdate_x64.bin...
error: unable to retrieve certificate from URL: https://uefi.org/sites/default/files/resources/dbxupdate_x64.bin. Error message: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>.

If the failure can't be fixed at the network configuration level, consider downloading the certificates manually and then loading one or more of them with secureboot-certs install <PK-filename>|default <KEK-filename>|default <db-filename>|default <dbx-filename>|latest. Check secureboot-certs install -h for usage details as well as a list of the download links used by secureboot-certs install.

The system's clock is correct and the uefi.org certificate seems fine to me:

* Server certificate:
*       subject: CN=uefi.org
*       start date: Oct 19 13:50:03 2021 GMT
*       expire date: Jan 17 13:50:02 2022 GMT
*       common name: uefi.org
*       issuer: CN=R3,O=Let's Encrypt,C=US

wget on the same system says the certificate is expired. A desktop's browser was fine with it and allowed me to download the file. Is there something I missed here? I did the same kind of installation couple of months back and had no issues with that.

stormi

It's related to the expiry of a root certificate and sub-optimal behaviour of the version of openssl we have in XCP-ng.

You can fix it by updating the ca-certificates RPM (it's been validated internally already and will be released to everyone with our next updates).

yum update ca-certificates --enablerepo=xcp-ng-testing

planedrop

@stormi I know this is quite an old topic, but this is still happening on XCP-ng 8.2.1. Do we need to update the RPM manually still or should this have been dealt with in the newer versions?

stormi

@planedrop Are you sure it's the same error? There's another known issue that is a 403 forbidden error received from microsoft's download site. You will find a workaround for this in another thread that mentions it. We're working on an update but it's not an easy situation: we don't choose which user agents Microsoft will decide to block.

antonseitz

@stormi
Hi, I just made a quick and dirty workaround by editing

/usr/sbin/secureboot-certs

I replaced this line:

req.add_header("User-Agent", "Mozilla/5.0")

by this line:

req.add_header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36")

Afterwards install went through without 403 error:

[18:10 xs ~]# secureboot-certs install default default default latest
Downloading https://www.microsoft.com/pkiops/certs/MicCorKEKCA2011_2011-06-24.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicCorUEFCA2011_2011-06-27.crt...
Downloading https://www.microsoft.com/pkiops/certs/MicWinProPCA2011_2011-10-19.crt...
Downloading https://uefi.org/sites/default/files/resources/dbxupdate_x64.bin...
Successfully installed certificates to the XAPI DB for pool.
[18:26 xs ~]#

stormi

@antonseitz This is indeed the workaround that is mentioned in the other thread.

planedrop

@antonseitz said in secureboot-certs install fails:

req.add_header("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.67 Safari/537.36")

Thank you, this was the issue! Appreciate the help here!

planedrop

@stormi Figured I'd ask, is this something that is going to be fixed in a future release? Seems like it should be default in there if a modification is needed.

Makes me a little hesitant to get this going in a production environment, which I was planning on doing after some more lab testing.

stormi

@planedrop Yes, as I wrote above, we're working on an update.

stormi

@planedrop Why does it make you hesitant? Are you going to install certificates a lot? Usually one only needs to run it once, on the first host of the pool. If you're running Windows VMs, you will get certificate updates through Windows update for each VM anyway. The pool certs are just here to bootstrap the VMs (this holds true for linux VMs too).

By the way, we aren't the ones making Microsoft's policy which is "we prefer that users get their certificates through their server's firmware, or download them manually from our website". As a convenience, we provided secureboot-certs install (Citrix doesn't in Citrix hypervisor for example... They get the certs from the host. If it's in BIOS mode... Too bad for you, no secure boot in VMs...) to automate things for you, but you're not in a dead end if the automated way doesn't work.

Lastly, even if they would block downloads again:

the new secureboot-certs install will both have a different default user agent and an option to let you use the user agent you want
you can always install certificates after downloading them manually, if the script can't download for some reason.

planedrop

@stormi OK this is a good point. My thing is that I generally don't like to modify things from near default in a prod environment because some update comes along later and breaks it.

To be fair though, I've never had this happen with XCP-ng in my lab and I also doubt something related to certs and secure boot would cause that, so maybe I'm paranoid and incorrect here.

Also, my apologies, I missed the reply about working on a fix.

This all makes sense to me now though, and the fix is pretty easy to get the automated install to work.

Thanks for all the help!

stormi

Update candidate available to fix certificate download: https://xcp-ng.org/forum/post/49373