UEFI Setting on VM for nested virtualization?
-
@olivierlambert Ok, and nested should work with UEFI? Or should I change to BIOS?
-
In theory, it's unrelated.
-
@noiden @olivierlambert There are settings in the Tiano UEFI firmware. Can't remember if there is one to turn on guest virtualization. I believe the hotkey to enter is F2. On Windows, if you click the power, restart option while holding down the SHIFT key, it should provide you with advanced startup options one of which is to access UEFI firmware settings.
-
@xcp-ng-justgreat I got in there, but there was no settings about that. I should troubleshoot some more, with Hyper-V itself.
-
@noiden Assuming the setting preventing processor virtualization is not in UEFI firmware, then keep looking at those obscure new exploit protection settings. I know that disabling CFG solved the problem for us on a physical server where we needed to run Docker containers. I have personally used nested virtualization of a UEFI-booted XCP-ng guest, running on XCP-ng (very cool, it works!) and did not have any problems. Since Docker won't run without the Hyper-V virtualization engine running, I still believe the answer for you lies there.
-
@xcp-ng-justgreat I have checked here, https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen
But I can't really figure out where to set the options,
hap=1 nestedhvm=1 cpuid = ['0x1:ecx=0xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx']I have set theese variables on the VM under vm-param platform and other-config. But that does not help.
I have also disabled the CFG on the VM, still no luck though.
Thanks.
-
I'm currently struggling with the same problem; running Windows Server 2019 on XCP-ng 8.1.0 . I changed the settings hinted at by @XCP-ng-JustGreat but it did not work.
-
@haribo112 @Noiden All, from within Windows, when you launch the task manager, performance tab, under CPU details, does it show you that virtualization is enabled? If yes, then the nested virtualization setting in XCP-ng appears to have worked. Try removing the Hyper-V roles and any related virtualization features and dependencies including your container support, reboot and then reinstall them and perform a finishing reboot. Coupled with disabling Control Flow Guard (CFG) for vmcompute.exe, I'm afraid that that's all I've got for you since that is what worked for us.
-
@xcp-ng-justgreat I will try that sequence. Obviously, Virtualization Extensions are detected by Windows, or else I wouldn't be able to install the Hyper-V role at all. The CPU performance tab of task manager does not show what you indicated, as it already says "Virtual machine: yes" in that location.
Honestly I'm considering rebuilding my homelab as a Hyper-V cluster, but that sucks too because realistically you need a domain controller for that to function properly.
-
@haribo112 @Noiden All, Looks like nested virtualization of Hyper-V within XCP-ng is a no go unless something changes in either or both XCP-ng and Hyper-V. Device driver for the Hyper-V virtual machine bus provider is the missing component that won't load. Seems this may be a problem only Microsoft can fix.
-
@XCP-ng-JustGreat Situation still unchanged
? -
@donileo Sadly, yes. No apparent forward movement to date. From the testing I was able to do and also from information passed along by Xen guru Andrew Cooper of Citrix, the problem lies partially with the Xen hypervisor code itself. It therefore requires the applied focus of an expert Xen developer in cooperation with, I think, the XenServer Windows Tools (drivers and management agent) developers. The guest would often hang with Xen drivers installed. The boot hang seemed to get worse with newer versions of Windows. It sometimes would boot and work in a flakey way with a really old Windows version e.g. Server 2008 SP2. This makes some sense intuitively since the Xen bus driver, Hyper-V bus driver and all the rest have to coexist and work together harmoniously. I simply don't have the skills to debug that. My sense is that there is a conflict among the various Windows guest drivers and also more work to be done on nested virtualization in Xen itself. I continue to hold out for a Xen hero that will bring nested-virtualization functional parity to Xen and its derivatives matching that of VMware, Hyper-V and KVM. The recent addition of nascent vTPM support in XCP-ng 8.3 gives me hope that the talent required to do this exists.
-
XenServer & Vates are both aware of those requirements. I even personally pushed for it during a meeting with the XS top execs. To be transparent, the thing that will prioritize it will be likely a big enough customer asking for it badly enough so the work will be higher in the backlog, or alternatively to be patient. I know it's not ideal, but it is what it is. Also, hopefully Vates continues to grow fast enough so we could assign new people on it
-
@olivierlambert You and the rest of the Vates team are already my Xen heros! I've been running XCP-ng 8.3 Beta on my home lab since August. I was impressed to see the new virtual TPM option in XO this weekend after pulling and compiling the latest source code. (BTW: Windows 11-23H2 BitLocker works flawlessly with the new vTPM support.) XCP-ng and XO truly just keep getting better and better! Hopefully, nested-virtualization of Windows Hyper-V on Xen will get solved before too long since a good variety of capable hypervisor options is important for a healthy and competitive virtualization ecosystem. This is especially true now as we all look to see what Broadcom will do as the new owner of VMware.
