Posts made by XCP-ng-JustGreat

@warriorcookie Your characterization is basically correct, but perhaps it should be "closer but no cigar." Masking the hypervisor's presence from the guest is required in all of the other hypervisors to successfully run a Windows guest with nested virtualization enabled. Prior to the discovery of the cited technique, nobody in the community knew how to do it on XenServer/XCP-ng using the xe API. However, the upstream Xen code itself and likely the guest drivers need more work in order for nested virtualization of a Windows guest to work reliably the way it does on ESXi, Hyper-V, etc. With the advent of Windows 11 and Server 2022, a virtualized TPM is also a required feature for full Windows compliance, so Xen has quite a bit on its "to do" list with respect to nested virtualization of Windows.

@thefrisianclause That's a good point. However, it's not here http://hcl.xenserver.org/gpus/?gpusupport__version=20&vendor=50 so NVIDIA and Citrix have no obligation to support it for their commercial customers. If you do get it to work, it's by the grace of Vates and/or other XCP-ng users here. Best of luck!

@thefrisianclause Sorry to hear that. NVIDIA does jealously guard its secret sauce from the world. Altering the CPUID hypervisor bit falls within the realm of unnatural acts. While the technique has proven useful in other use cases and is a good thing to know about, it may be that countermeasures have been added to the GPU drivers to expose the lie. Hard to know . . .

@thefrisianclause Yes, it should work for a Linux guest too. It alters the VM's apparent CPUID as presented to the guest OS--whatever that happens to be. I'm not one-hundred percent sure about AMD, but try the same technique. The same bit probably has the same purpose on an AMD CPU.

@thefrisianclause Hello, not sure if I missed something from the above thread, but did any of you try to turn off the CPUID "hypervisor present" bit on an Intel-based XCP-ng host VM using this technique from the thread referenced by @warriorcookie above? https://xcp-ng.org/forum/topic/4643/nested-virtualization-of-windows-hyper-v-on-xcp-ng/26

It is the equivalent of the ESXi Hypervisor.CPUID.v0="FALSE" vmx file configuration tweak. It configures the XCP-ng VM to, in effect, lie to the guest OS by saying, "you are not running on a hypervisor."

@stormi All new patches applied fine. No apparent problems identified so far.

@noship Hello. The secure boot feature is currently available as pre-release code. My personal experience is that it works well for my use case. Some others are reporting boot issues after installing the updates so it continues to evolve and is not yet released for production. Search the forum for UEFI and you will find the relevant threads for obtaining and installing secure boot support. Here's one: https://xcp-ng.org/docs/guides.html#guest-uefi-secure-boot

@alexanderk @olivierlambert Sorry to have not responded sooner to your question. It has been a very long, slow slog so far and I haven't been able to devote as much time as I'd like to working on this. Here's what I've done so far: Based on Andrew Cooper's recommendation, I installed a fully patched Windows Server 2008 R2 VM to Xen. (Hyper-V was initially released with Server 2008 so this is almost as far back as you can go.) Using the current unmodified Xen source code, the VM will permit Hyper-V to be enabled in the Windows Server 2008 R2 guest, but--as with newer versions of Windows--once you perform the finishing reboot, Hyper-V is not actually active. Adding the two recommended source-code patches, recompiling and performing the same test causes the VM to hang following the enablement of Hyper-V. I know that I need to set up a serial console for the VM in order to view any logging that might provide a clue as to what's failing during the boot, but I haven't worked that out just yet.

I've also spent some considerable time reading through the Xen Dev email posts on the history of the development of nested virtualization in Xen. One very significant learning from that reading is that nested virtualization on Xen was initially developed by an AMD developer. Development of the NV feature-set for Intel came later after the AMD-focused design die had been cast. As far as I can tell given that I'm running Server 2008 R2, this never worked on Intel. (Maybe it did on an older Intel processor, but I am currently working with SkyLake i7-6700s so have no way to test older hardware.) Unfortunately, I also don't have appropriate AMD hardware on which to perform the same test to see whether or not it might work on AMD.

On the Microsoft Hyper-V side, it seems as though the opposite evolution happened. Nested virtualization was developed on Intel first, then (very recently) AMD. This makes me suspect that it doesn't work on AMD either. In other words, I don't know that nested virtualization of Windows on Xen ever worked such that Hyper-V was actually active in the guest. I would be delighted to have somebody prove me wrong.

@olivierlambert @stormi Here's a good link for persistent disablement of driver signature checking on Windows using bcdedit https://blog.pcrisk.com/windows/12194-how-to-disable-driver-signature-enforcement that may help those above wanting to use the XCP-ng drivers. If for some reason that doesn't work, they can, of course, use the signed Citrix drivers as a stopgap measure.

@beshleman I tried the latest testing update @stormi published with the updated SB support and it does indeed work properly including allowing installation of Windows Update KB4535680 on Server 2019 as previously cited. Also--a big thank you for adding the default parameter values for the improved secureboot-certs install command. Less is more. Very nice!

@beshleman So, after yum --enablerepo=base install python-requests on each of my hosts, secureboot-certs install default default default latest works perfectly. (Cool that it installs certs to each host in the pool with one invocation from any pool host.) Interesting that it doesn't install the three files to /var/lib/uefistored until you secure boot a vm on each host. I went looking for them and was initially confused because they were only written to the pool db. Serves me right for looking under the hood! Looks like XCP-ng secure boot is ready for prime time. Great job!

@stormi @olivierlambert It looks like we now have the attention of Andrew Cooper at Citrix. For anyone interested in following or participating in the Xen developer list nested virtualization thread we originated, it begins here: https://lists.xenproject.org/archives/html/xen-devel/2021-07/msg01269.html (Just click Thread Next to go through it sequentially.) For the purposes of that list, I have become Xentrigued. Cooper readily admits that nested virtualization in Xen is "a disaster" and has suffered from neglect. With the upcoming launch of Windows 11 and Server 2022, nested virtualization of Hyper-V and, likely, vTPM 2.0 support will become "musts" for hypervisor certification by Microsoft so there are some strong tail-winds that may aid in pushing this forward beyond the XCP-ng community. I will try to be of some use toward that end.

@stormi So far, I have tested a fresh install using software RAID mirror creation. Works fine. Also, noticed the new EFI boot kludge to correct missing bootloader on Dell and other faulty UEFI firmware. (I used to always add the /boot/efi/EFI/boot/bootx64.efi file to correct this since it also occurs on my ASUS-motherboard machine.) That works well. The newly refreshed secureboot-certs install default default default latest command is not working. The requests python module is not being found. BTW, I think the default option where the command is secureboot-certs install should be equivalent to adding default default default latest parameters @beshleman . I'll continue to test and report back later.

@olivierlambert Incorporating suggested changes from @stormi above in bold italics.

@stormi Yes. That is better. I'll update it.

@olivierlambert @stormi OK. My draft to the Xen-devel ML follows. Feel free to critique if you think it will strengthen our case. Once finalized, I'll send it to the ML.

SUBJECT: Nested Virtualization of Hyper-V on Xen Not Working

RATIONALE: Features in recent versions of Windows now REQUIRE Hyper-V support to work. In particular, Windows Containers, Sandbox, Docker Desktop and the Windows Subsystem for Linux version 2 (WSL2). Running Windows in a VM as a development and test platform is currently a common requirement for various user segments and will likely become necessary for production in the future. Nested virtualization of Hyper-V currently works on VMware ESXi, Microsoft Hyper-V and KVM-based hypervisors. This puts Xen and its derivatives at a disadvantage when choosing a hypervisor.

WHAT IS NOT WORKING? Provided the requirements set forth in: https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen have been met, an hvm guest running Windows 10 PRO Version 21H1 x64 shows that all four requirements for running Hyper-V are available using the msinfo32.exe or systeminfo.exe commands. More granular knowledge of the CPU capabilities exposed to the guest can be observed using the Sysinternals Coreinfo64.exe command. CPUID flags present appear to mirror those on other working nested hypervisor configurations. Enabling Windows Features for Hyper-V, Virtual Machine Platform, etc. all appear to work without error. However, after the finishing reboot, Hyper-V is simply not active. This--despite the fact that vmcompute.exe (Hyper-V host compute service) is running and there are no errors in the logs. In addition, all four Hyper-V prerequisites continue to show as available.

By contrast, after the finishing reboot of an analogous Windows VM running on ESXi, the four prerequisites are reversed: hypervisor is now active; vmx, ept and urg (unrestricted guest) are all off as viewed with the Coreinfo64.exe –v command. Furthermore, all functions requiring Hyper-V are now active and working as expected.

This deficiency has been observed in two test setups running Xen 4.15 from source and XCP-ng 8.2, both running on Intel with all of the latest, generally available patches. We presume that the same behavior is present on Citrix Hypervisor 8.2 as well.

SUMMATION:
Clearly, much effort has already been expended to support the Viridian enlightenments that optimize running Windows on Xen. It also looks like a significant amount of effort has been put forth to advance nested virtualization in general.

Therefore, if it would be helpful, I am willing to perform testing and provide feedback and logs as appropriate in order to get this working. While my day job is managing a heterogeneous collection of systems running on various hypervisors, I have learned the rudiments of integrating patches and rebuilding Xen from source so could no doubt be useful in assisting you with this worthwhile endeavor.

@stormi OK. I'll put it together here first.

@olivierlambert @stormi It was quite an odyssey getting everything to run with pure vanilla Xen 4.15 compiled from source on Debian 10.10, but I finally accomplished it. (Learned a lot in the process too!) The final sticking point was that the Windows VM xl config file previously built and working on the older version of Xen packaged with Debian 10, wouldn't boot. Something wasn't working with the guest UEFI support so I switched to BIOS boot and that worked. The net result is that nested Hyper-V installs fine as before, but still won't activate on reboot. I also note that the x2apic CPU capability is now present in the guest as it is with VMware ESXi. That flag is missing when running nested Windows under XCP-ng 8.2 on my Intel i7-6700 processor-based system. Now that we know for sure it is still not working in the very latest Xen kernel, what next steps should we take for getting this issue to the attention of the Xen developers?

@stormi @olivierlambert All, something stormi mentioned yesterday made me double-check the version of vanilla Xen that was packaged with the Debian 10 test distro. Turns out, it's older than the version used in XCP-ng 8.2. We really do need to see whether or not nested Hyper-V works in Xen 4.15 (the latest) before bringing it to the attention of the Xen dev ML. Toward that end, I astonished myself last night by compiling Xen 4.15 from the source code! Most of the time was spent identifying and installing the many prerequisites--now documented--so subsequent builds will be quite fast. One packaging issue remains: the final make install command installed the xen kernel etc., but did not add the grub entry to boot it. What is the proper way to add the grub menu Debian with Xen boot choice? I considered doing it in a hacky fashion using the leftover grub menu entry from the packaged version in Debian 10. Can you tell me the right way? Please let me know and I'll give it a try this weekend. Thank you.

@olivierlambert Cool. Thank you!