Best posts made by XCP-ng-JustGreat

@stormi Installed all of the test updates on my three-host home-lab this weekend. Similar configuration to @gskger 3 x Dell OptiPlex 7040 SFF hosts and home-built FreeNAS server with separate physical 1Gb networks for management, storage and migration. I call it my "Tiny Cluster" due to its diminutive footprint. I use it for configuration prototyping. Intel VPRO AMT on Xen hosts and storage server enables headless console operation using MeshCommander (think poor man's iDRAC). All updates were installed without issue. Backups and restores seem to work just fine. Of special interest to me was the UEFI Secure Boot capabilities. Installed the x64 dbx.auth from uefi.org (I presume since XCP-ng is 64-bit that that was the correct choice. Probably should be made explicit in the instructions.) Seems to work perfectly. I tested with Windows 10-20H2 and Windows 10-21H1. Also tested with RHEL 8.4 which has built-in support for secure boot (Microsoft-signed bootloader shim) and that too "just works." The varstore-ls <VM-uuid> command shows PK, KEK, dbx and db in the store as expected. Stops unsigned bootloader as expected on unsupported OSes. Looks great! Thank you for all of the work you've put into it. I suspect designing and building emulated system firmware is not for the faint of heart . . . Very impressive!

@stormi All new patches applied fine. No apparent problems identified so far.

@beshleman I tried the latest testing update @stormi published with the updated SB support and it does indeed work properly including allowing installation of Windows Update KB4535680 on Server 2019 as previously cited. Also--a big thank you for adding the default parameter values for the improved secureboot-certs install command. Less is more. Very nice!

@beshleman So, after yum --enablerepo=base install python-requests on each of my hosts, secureboot-certs install default default default latest works perfectly. (Cool that it installs certs to each host in the pool with one invocation from any pool host.) Interesting that it doesn't install the three files to /var/lib/uefistored until you secure boot a vm on each host. I went looking for them and was initially confused because they were only written to the pool db. Serves me right for looking under the hood! Looks like XCP-ng secure boot is ready for prime time. Great job!

@stormi So far, I have tested a fresh install using software RAID mirror creation. Works fine. Also, noticed the new EFI boot kludge to correct missing bootloader on Dell and other faulty UEFI firmware. (I used to always add the /boot/efi/EFI/boot/bootx64.efi file to correct this since it also occurs on my ASUS-motherboard machine.) That works well. The newly refreshed secureboot-certs install default default default latest command is not working. The requests python module is not being found. BTW, I think the default option where the command is secureboot-certs install should be equivalent to adding default default default latest parameters @beshleman . I'll continue to test and report back later.

@noship Hello. The secure boot feature is currently available as pre-release code. My personal experience is that it works well for my use case. Some others are reporting boot issues after installing the updates so it continues to evolve and is not yet released for production. Search the forum for UEFI and you will find the relevant threads for obtaining and installing secure boot support. Here's one: https://xcp-ng.org/docs/guides.html#guest-uefi-secure-boot

@stormi @olivierlambert It looks like we now have the attention of Andrew Cooper at Citrix. For anyone interested in following or participating in the Xen developer list nested virtualization thread we originated, it begins here: https://lists.xenproject.org/archives/html/xen-devel/2021-07/msg01269.html (Just click Thread Next to go through it sequentially.) For the purposes of that list, I have become Xentrigued. Cooper readily admits that nested virtualization in Xen is "a disaster" and has suffered from neglect. With the upcoming launch of Windows 11 and Server 2022, nested virtualization of Hyper-V and, likely, vTPM 2.0 support will become "musts" for hypervisor certification by Microsoft so there are some strong tail-winds that may aid in pushing this forward beyond the XCP-ng community. I will try to be of some use toward that end.

@stormi Sounds good. We'll wait for the wizards at Vates to do their thing. With great admiration and appreciation for all that you do. XCP-ng just keeps getting better and better! We thank you!!

@olivierlambert @stormi Here's a good link for persistent disablement of driver signature checking on Windows using bcdedit https://blog.pcrisk.com/windows/12194-how-to-disable-driver-signature-enforcement that may help those above wanting to use the XCP-ng drivers. If for some reason that doesn't work, they can, of course, use the signed Citrix drivers as a stopgap measure.

@alexanderk @olivierlambert Sorry to have not responded sooner to your question. It has been a very long, slow slog so far and I haven't been able to devote as much time as I'd like to working on this. Here's what I've done so far: Based on Andrew Cooper's recommendation, I installed a fully patched Windows Server 2008 R2 VM to Xen. (Hyper-V was initially released with Server 2008 so this is almost as far back as you can go.) Using the current unmodified Xen source code, the VM will permit Hyper-V to be enabled in the Windows Server 2008 R2 guest, but--as with newer versions of Windows--once you perform the finishing reboot, Hyper-V is not actually active. Adding the two recommended source-code patches, recompiling and performing the same test causes the VM to hang following the enablement of Hyper-V. I know that I need to set up a serial console for the VM in order to view any logging that might provide a clue as to what's failing during the boot, but I haven't worked that out just yet.

I've also spent some considerable time reading through the Xen Dev email posts on the history of the development of nested virtualization in Xen. One very significant learning from that reading is that nested virtualization on Xen was initially developed by an AMD developer. Development of the NV feature-set for Intel came later after the AMD-focused design die had been cast. As far as I can tell given that I'm running Server 2008 R2, this never worked on Intel. (Maybe it did on an older Intel processor, but I am currently working with SkyLake i7-6700s so have no way to test older hardware.) Unfortunately, I also don't have appropriate AMD hardware on which to perform the same test to see whether or not it might work on AMD.

On the Microsoft Hyper-V side, it seems as though the opposite evolution happened. Nested virtualization was developed on Intel first, then (very recently) AMD. This makes me suspect that it doesn't work on AMD either. In other words, I don't know that nested virtualization of Windows on Xen ever worked such that Hyper-V was actually active in the guest. I would be delighted to have somebody prove me wrong.

@olivierlambert @AlexanderK Wow! Changing the platform:featureset=value as prescribed does indeed enable support for nested Hyper-V on XCP-ng!! This is huge!!! I will now need to begin testing some of the actual Hyper-V-dependent functions in the nested Windows guest. The msinfo32.exe command displays no apparent knowledge that the guest OS is running on a hypervisor. It simply indicates that all four prerequisites for running Hyper-V are enabled. It would appear that this platform modification effectively tells the XCP-ng VM to lie to the nested guest OS: "No, you are not running on a hypervisor." BTW, I still have all of the latest pre-release patches installed that we tested a couple of weeks ago for @stormi . Therefore, the tested Windows VM is also booting with secure boot support by @beshleman along with this additional major breakthrough. Because of nested virtualization's complexity, there will no doubt be additional hurdles we need to jump over to make the desired guest functions work, but we have made tremendous progress in a rather short span of time. Congratulations everybody! If it's possible, we may want to change the title of the post to Nested Virtualization of Windows Hyper-V on XCP-ng as that will provide an easier way for it to be found by others. I can't really adequately express my gratitude to all of you for the amazing level of support! (Pinging some others who have fought with this issue in the past and will no doubt appreciate hearing of its progress: @Noiden @Haribo112 )

After reading the latest XCP-ng blog entry, I thought that I'd give this a try. Is the newly rewritten secure boot code deployed to production? I am running XCP-ng 8.2 with all the latest fixes. Xen Orchestra from source is also completely up-to-date. VM, Advanced, Secure boot setting is toggled on. Entering command xe vm-param-get param-name=platform uuid=<MyVMuuid> shows secureboot: true. However, Tiano firmware settings show current secure boot state = Disabled (see picture). Also, the PowerShell command: Confirm-SecureUEFIBoot from Windows 10 (20H2) guest shows secure boot is off as does the msinfo32.exe command. Any ideas?

@noiden Assuming the setting preventing processor virtualization is not in UEFI firmware, then keep looking at those obscure new exploit protection settings. I know that disabling CFG solved the problem for us on a physical server where we needed to run Docker containers. I have personally used nested virtualization of a UEFI-booted XCP-ng guest, running on XCP-ng (very cool, it works!) and did not have any problems. Since Docker won't run without the Hyper-V virtualization engine running, I still believe the answer for you lies there.