BitLocker Boot Recovery Key Requested After Latest 8.3 Updates

XCP-ng-JustGreat

Had a couple of Windows 11 VMs setup previously on XCP-ng Beta 8.3 with UEFI Secure Boot, vTPM and BitLocker encryption active. After applying the latest updates, unencrypted UEFI Secure Boot Windows VMs still boot fine, but BitLocker encrypted Windows VMs boot to blue screen and prompt for the BitLocker recovery key. Normally with Windows, an OEM firmware update will trigger this behavior if BitLocker is not suspended prior to flashing the system firmware. As a result, OEM firmware installers generally check for active encryption and suspend it automatically prior to flashing the firmware. Not sure which of the latest updates changed the VM firmware state values, but this could potentially be a huge issue for a production system. In my case, these were just test VMs so no damage was incurred.

olivierlambert

Might be related to EDK, let me ping @stormi when he's back

stormi

Thanks for the notice. I'm not sure what was detected as a firmware update, but since edk2 implements the UEFI firmware, its update is probably what caused this.

Would this also happen if the VM is shut down before the update?

john.c

@stormi said in BitLocker Boot Recovery Key Requested After Latest 8.3 Updates:

Thanks for the notice. I'm not sure what was detected as a firmware update, but since edk2 implements the UEFI firmware, its update is probably what caused this.

Would this also happen if the VM is shut down before the update?

Yes, it would still happen if the VM is shut down before the update. As the PCRs wouldn't match what is required for the TPM to release the keys, when the VM next boots. This happens as edk2 which implements the UEFI has been updated, thus no longer the same.

The only time this won't happen is if BitLocker is suspended before updating the firmware (edk2) as it will refresh the PCRs as part of resuming BitLocker, due potentially have been PCR changing alterations during the suspension.

So to deal with this the windows xcp-ng agent would need to be notified if there's any uefi firmware (edk2) updates required, so it can automatically suspend BitLocker for the update and then resume BitLocker after has been completed.

john.c

@stormi Oh, I forgot to mention that the same will need to be done for packages, which implement Legacy BIOS firmware updates for VMs.

As the same will happen for them too when the firmware update changes the PCRs.

stormi

@john-c There's no support for vTPM with Legacy BIOS on XCP-ng. Can you still use Bitlocker despite this?

john.c

@stormi said in BitLocker Boot Recovery Key Requested After Latest 8.3 Updates:

@john-c There's no support for vTPM with Legacy BIOS on XCP-ng. Can you still use Bitlocker despite this?

You can by altering Group Policy, its more laborious to setup, as well as use and liable to being hacked. As it requires a credential key and/or passphrase of some kind to use. As you can supply the credentials, unknowingly when hacked allowing the system to fully boot. The TPM based method can help with this as a request for the recovery key, can potentially be considered a warning sign though there's other reasons for this to happen (e.g. admin implementing a uefi/bios firmware update, TPM cleared or TPM disabled).

The TPM based method will only release the cryptography keys when sealed to the OS for BitLocker to use, if the PCRs match those from when they were sealed.

So would be better to be corrected as vTPM support for BIOS based VMs as well.

stormi

So @ThierryEscande investigated the situation and made tests. There's no easy solution. We have no way to detect that bitlocker is in use, and our update process doesn't allow to block based on prerequisites either. We might be able to hack something in Xen Orchestra whenever we detect that updates contain edk2, but this would only covert part of the problem. System upgrades using the installation ISO would also break VM boot, as well as updates made directly with yum, VM import, snapshot restore, backup restore, and live migration to another XCP-ng where the version of edk2 is not strictly the same.

A long term solution would likely involve keeping compatibility with any previous version of the firmware we once shipped, and letting users manually upgrade the firmware VM after VM. But this might prevent us from doing necessary updates of edk2/OVMF when the other components around them require newer versions to operate.

We had a look at how XenServer handles the situation, and it turns out they simply stated that Bitlocker is not supported. Given the above, this is understandable. There are too many ways this can break. This will also be the situation in our case for now: we will document the risks, and users will need to be aware that, would they enable it, they can face such issues. Fortunately, the VMs seem to be easily recoverable provided you have the recovery key.

john.c

@stormi @ThierryEscande The BitLocker can be managed by command line so thus if you can run or launch that then Xen Orchestra (and/or XO Lite) can via Guest Tools suspend and resume it.

The Windows Guests will have an executable called "manage-bde.exe" you can use this to get the status of BitLocker Drive Encryption and also suspend and resume it.

In the case of ISO upgrading, maybe have some metadata about the BitLocker status associated with VMs on the hypervisor host or with network up the SRs previously configured. In other words (BitLocker - enabled, BitLocker - disabled, BitLocker - suspended and/or BitLocker - resumed").

The ISO based upgrader could scan for and/or check this metadata then if the status is "BitLocker - enabled" display a message warning or preventing the upgrade, while requesting that the user suspends BitLocker for the affected VMs before attempting the upgrade.

This metadata could also then be a part of Netbox plugin synchronisation as part of Xen Orchestra as well as made visible on that particular VM's information panel.

https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-7/dd875513(v=ws.10)

https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/manage-bde

stormi

We envisioned it, but we're not confident that the guest agent should be given such high powers. Reporting the status of bitlocker would be useful, but I think disabling it should be left to the user.

Also, we still have work to do before we can provide a signed installer and signed drivers for the guest tools.

john.c

@stormi I'm not expecting you to do it at the moment just something to consider in regards to your previous post.

Cause even if you don't enable the suspend or resume reporting the status during upgrade and update would help.

Plus there's not just dependencies for the packages, aren't there also pre-installation scripts tied to each package. If there are then a pre-installation or update script associated with edk2 could run which would communicate with XO Lite or Xen Orchestra to check the status of BDE. So it can warn of BitLocker's status of VMs so that users can apply the updated edk2 packages.

XCP-ng-JustGreat

@john-c @stormi @olivierlambert All, given the complexities involved in providing an automated solution for this issue, it may make the most sense at this time to mitigate any negative outcomes using an installation/update warning. For instance, throw a message dialog issued by both the ISO installer and the pool update feature in XO such as: WARNING: One or more of the pending updates may alter your VM's firmware. If you are using Microsoft BitLocker to encrypt virtual drives, you should exit now and suspend BitLocker prior to performing this update. (Provide customer with Exit or Continue buttons with Exit selected by default.)