Nested Virtualization of Windows Hyper-V on XCP-ng
-
By opening an issue on https://github.com/xcp-ng/xcp and trying to reproduce it on "vanilla" Xen (eg any Arch Linux or Debian with a recent version of Xen) with
xltoolstack. -
@olivierlambert @AlexanderK As Olivier requested, I set up a basic Xen Hypervisor configuration using Debian 10.9 (Buster) following the tutorial here https://wiki.xenproject.org/wiki/Xen_Project_Beginners_Guide in my home lab. It took a bit of work since the documentation is outdated and not entirely clear, but I was able to get a basic Windows 10 VM running on that. I then added the nested virtualization parameters from here https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen to the VM config file. [Next section edited retrospectively.] Nested Hyper-V installs and all the guest Hyper-V drivers load on the parent Xen platform, but Hyper-V does not activate following the finishing reboot. I have commented the working xl toolstack config file: windows.cfg detailing my findings if you would like me to upload that somewhere. Seems like we need to find out if it is possible to set the cpuid and hap parameters from xe vm-param-set platform:<param>=value in order to flip the appropriate switches in the underlying hypervisor. (There doesn't seem to be a documented way to fully activate the needed support right now from the xe toolstack.) It does look like all of the viridian parameters are available. Viridian exposes various interfaces and features that Hyper-V requires. The cpuid param appears to cause the guest OS to become unaware that it is running on a hypervisor. Please let me know if you'd like me to upload the working xl config file someplace. Nested virtualization was but an interesting parlor trick before, but the architecture of modern Windows is rapidly making it an essential feature for running fully functional Windows VMs.
-
First, thanks for taking time to test it
So did you check with the viridian param if it works on XCP-ng? Obviously, we'd love to document this behavior if it's just an extra param to add on the VM record!
About the config file, you might be able to paste it directly here? Otherwise, go for https://paste.vates.fr/
-
@xcp-ng-justgreat can you make a guide? what have you done and it is working? Will save time and will be really appreciated
-
@olivierlambert @AlexanderK Unfortunately, I could not make it work on XCP-ng. I tried entering the same name/key pairs under the VM platform category, but it does not look like the xe toolstack maps them correctly to the hypervisor. I didn't find documentation for applying them from xe. The success was only realized in Xen hypervisor.
-
Please share the exact key/values you used to make it work on "vanilla" Xen. It could be trivial to "port" it to XCP-ng, but for that we need that you share what you found
-
@olivierlambert @AlexanderK I pasted my xl config file here: https://paste.vates.fr/?449c18ba665cd704#GzDDKa4fui6jqbvG7ssT5TpH7uj8uBVrg7zojpGqYmu7
-
@olivierlambert @AlexanderK One final thing, here is the Xen reference manual link for the xl config file parameters (I learned a lot reading it.) https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html The outstanding issue appears to be whether or not there is a corresponding setting configurable from the xe toolstack to turn on all of the required parameter/value pairs as implemented by my test configuration on the pure Xen hypervisor. If so, we should then be able to make nested virtualization of Hyper-V work on XCP-ng.
-
As long as we know exactly what's missing, we can fix it
-
@olivierlambert said in Windows 10 Vm and Desktop Docker Issue:
As long as we know exactly what's missing, we can fix it
can't wait for the fix
-
It's still unclear what parameter is missing on XAPI vs libxl, waiting for more details from @XCP-ng-JustGreat
-
@AlexanderK can you share the VM record here please? (the one that's nested but doesn't work with Docker). A simple output of
xe vm-param-list uuid=<VM UUID>will do it.Just checked the
xlconfig posted by @XCP-ng-JustGreat, and I can already answer for some parameters, but I'd like to check if they are already used or not (egplatform:viridiancan be set totrueif it's not already the case) -
-
Can you try with a Windows 10 template from the start?
-
@olivierlambert ok will do it.
-
@alexanderk @olivierlambert Did a bit more testing by removing one of the four nested-virtualization parameters at a time in the windows.cfg file on pure vanilla Xen hypervisor. Omitting the cpuid setting is the one where the Hyper-V machine bus provider driver fails to load in the guest and, basically, nested Hyper-V breaks. From what I can tell, entering that key/value pair under the platform parameter in the XCP-ng VM definition has no effect after starting from a "stock" Windows 10 VM template.
Since the platform parameter would seem to be the logical place for that value, it looks like a developer needs to look at the code to see whether or not the mechanism for passing cpuid already exists and is not documented, or instead it needs to be implemented. Thanks again to @olivierlambert and the team at Vates for your continued innovation and enhancement of this remarkably useful software!
-
Okay so it's the only thing we don't have is
cpuidand that make the diff. I'll ask around to see how we could have this in XCP-ng/XAPI (xenopsin fact).Thanks for your precious feedback @XCP-ng-JustGreat !
-
@olivierlambert
here it is from the starthttps://paste.vates.fr/?1d9650dbc857af53#3STr5czhCrom7y4HvyNGBH6nZ1bghHaiXKuChhcDrbCK
-
@xcp-ng-justgreat
@olivierlambert how can we implement manually the cpuid that @XCP-ng-JustGreat has found at a running vm? -
So after discussing with my favorite Xen expert, here what could be do to get a similar result on the cpuid:
- Get the pool CPU ID:
xe pool-param-get uuid=<pool_uuid> param-name=cpu_info param-key=features_hvm. Should be something like1fcbfbff-f7fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000 - To compute the value you want to pass on the VM, you need to clear the hypervisor bit. So the previous string will become now
1fcbfbff-77fa3203-[...]-00000000 - See this value for your VM, which will override the default computed featureset for this specific VM:
xe vm-param-set uuid=<VM UUID> platform:featureset=1fcbfbff-77fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000 - Start the VM, and report.
Reference for bit positions in the bitmap: https://github.com/xen-project/xen/blob/master/xen/include/public/arch-x86/cpufeatureset.h
The hypervisor bit is the top one, so you need to apply (val & 0x7fffffff) on it (it might be different in your example).
Note: we do NOT understand why doing this will solve the issue, but at least let's try first.
- Get the pool CPU ID:
