XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Nested Virtualization of Windows Hyper-V on XCP-ng

    Scheduled Pinned Locked Moved Compute
    111 Posts 12 Posters 72.2k Views 12 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by olivierlambert

      So after discussing with my favorite Xen expert, here what could be do to get a similar result on the cpuid:

      1. Get the pool CPU ID: xe pool-param-get uuid=<pool_uuid> param-name=cpu_info param-key=features_hvm. Should be something like 1fcbfbff-f7fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000
      2. To compute the value you want to pass on the VM, you need to clear the hypervisor bit. So the previous string will become now 1fcbfbff-77fa3203-[...]-00000000
      3. See this value for your VM, which will override the default computed featureset for this specific VM: xe vm-param-set uuid=<VM UUID> platform:featureset=1fcbfbff-77fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000
      4. Start the VM, and report.

      Reference for bit positions in the bitmap: https://github.com/xen-project/xen/blob/master/xen/include/public/arch-x86/cpufeatureset.h

      The hypervisor bit is the top one, so you need to apply (val & 0x7fffffff) on it (it might be different in your example).

      Note: we do NOT understand why doing this will solve the issue, but at least let's try first.

      AlexanderKA X 2 Replies Last reply Reply Quote 0
      • AlexanderKA Offline
        AlexanderK @olivierlambert
        last edited by

        @olivierlambert

        i am trying to calculate for my vm...

        the cpu id on pool is
        1fcbfbff-80b82221-2993fbff-00000403-00000000-00000000-00000000-00000000-00001000-9c000000-00000000-00000000-00000000-00000000-00000000

        i must change the second "team" of 8 digits? I suppose that it is something in hex.

        You wrote that the hypervisor bit is the top one. What do you mean?
        I should add on 80b82221 the value 0x7fffffff?

        X 1 Reply Last reply Reply Quote 0
        • X Offline
          XCP-ng-JustGreat @AlexanderK
          last edited by

          @alexanderk I think @olivierlambert wants us to logically AND the bits there. Will give it a try later on my pool.

          1 Reply Last reply Reply Quote 0
          • X Offline
            XCP-ng-JustGreat @olivierlambert
            last edited by

            @olivierlambert @AlexanderK Wow! Changing the platform:featureset=value as prescribed does indeed enable support for nested Hyper-V on XCP-ng!! This is huge!!! I will now need to begin testing some of the actual Hyper-V-dependent functions in the nested Windows guest. The msinfo32.exe command displays no apparent knowledge that the guest OS is running on a hypervisor. It simply indicates that all four prerequisites for running Hyper-V are enabled. It would appear that this platform modification effectively tells the XCP-ng VM to lie to the nested guest OS: "No, you are not running on a hypervisor." BTW, I still have all of the latest pre-release patches installed that we tested a couple of weeks ago for @stormi . Therefore, the tested Windows VM is also booting with secure boot support by @beshleman along with this additional major breakthrough. Because of nested virtualization's complexity, there will no doubt be additional hurdles we need to jump over to make the desired guest functions work, but we have made tremendous progress in a rather short span of time. Congratulations everybody! If it's possible, we may want to change the title of the post to Nested Virtualization of Windows Hyper-V on XCP-ng as that will provide an easier way for it to be found by others. I can't really adequately express my gratitude to all of you for the amazing level of support! (Pinging some others who have fought with this issue in the past and will no doubt appreciate hearing of its progress: @Noiden @Haribo112 )

            AlexanderKA 1 Reply Last reply Reply Quote 1
            • AlexanderKA Offline
              AlexanderK @XCP-ng-JustGreat
              last edited by

              @xcp-ng-justgreat Title changed.

              Can you help me with the AND and what exactly have you done? Give me an example to understand.

              AlexanderKA X 2 Replies Last reply Reply Quote 1
              • AlexanderKA Offline
                AlexanderK @AlexanderK
                last edited by

                @alexanderk
                have tried something and issue on device manager is fixed but....
                3e43535a-085e-4d15-a701-7de2a52315b1-image.png

                and

                bce6fdf7-badd-489c-bbd2-a350069d778d-image.png

                X 1 Reply Last reply Reply Quote 0
                • X Offline
                  XCP-ng-JustGreat @AlexanderK
                  last edited by

                  @alexanderk I think you need to run the PowerShell command to get second-level (L2) nested virtualization within Hyper-V. You are now effectively running yet another VM inside of an existing VM (like Russian dolls-nested) so you need to expose the processor hardware virtualization from L1 to L2. See here: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by olivierlambert

                    I have to admit I don't grasp everything (also I'm not a Windows expert) I still wonder why changing what we did is making it work (in details, your feeling is likely right @XCP-ng-JustGreat )

                    So you are running Docker inside a nested Windows, which is itself nested in XCP-ng?

                    AlexanderKA 1 Reply Last reply Reply Quote 0
                    • AlexanderKA Offline
                      AlexanderK @olivierlambert
                      last edited by

                      @olivierlambert trying to do this but it is not working.

                      @XCP-ng-JustGreat docker works for you? even hyper-v shows error.
                      Can anyone explain with an example the calculation of new value?

                      1 Reply Last reply Reply Quote 0
                      • X Offline
                        XCP-ng-JustGreat @AlexanderK
                        last edited by

                        @alexanderk If you are running a current version of Windows 10, you can use the Calculator app. It has a programmer mode that allows you to enter the 32-bit source value from the platform cpuid in hex and bitwise AND it with 0x7FFFFFFF. Programmer mode is widely available on calculators for Linux too so use the tool with which you are most comfortable.

                        AlexanderKA 1 Reply Last reply Reply Quote 0
                        • AlexanderKA Offline
                          AlexanderK @XCP-ng-JustGreat
                          last edited by

                          @xcp-ng-justgreat said in Nested Virtualization of Windows Hyper-V on XCP-ng:

                          0x7FFFFFFF

                          there are some groups of hex digits at the cpu id.
                          I use the second group of digits?

                          The second group is 80b82221 and with bitwise and 7FFFFFF i have this

                          37e55a63-3c59-4eba-ad52-a5769b4b01f7-image.png

                          the result is 6 digits. Is this correct?

                          X 1 Reply Last reply Reply Quote 0
                          • X Offline
                            XCP-ng-JustGreat @AlexanderK
                            last edited by

                            @alexanderk Two leading hex zeros (8-bits) are implied.

                            AlexanderKA 2 Replies Last reply Reply Quote 0
                            • AlexanderKA Offline
                              AlexanderK @XCP-ng-JustGreat
                              last edited by

                              @xcp-ng-justgreat
                              ok thanks.
                              desktop docker worked at your vm?

                              1 Reply Last reply Reply Quote 0
                              • AlexanderKA Offline
                                AlexanderK @XCP-ng-JustGreat
                                last edited by

                                @xcp-ng-justgreat

                                hyperv is working without issues?

                                X 1 Reply Last reply Reply Quote 0
                                • X Offline
                                  XCP-ng-JustGreat @AlexanderK
                                  last edited by XCP-ng-JustGreat

                                  @alexanderk (With apologies to all for the following dense reading as there is much detail. TLDR; Hyper-V installs, but remains disabled on XCP-ng. Hyper-V installs, is enabled, and starts fine on ESXi where nested Hyper-V works. Methodology and items tested to follow.) No breakthrough yet. The Windows Hyper-V feature installs fine into the guest, but it does not appear to become active following the reboot. This is the same behavior that occurs using pure vanilla Xen hypervisor. I set up two parallel test environments: one using ESXi 6.7 Build 17700523 which works perfectly, and another using XCP-ng which does not yet work. Host hardware is very similar with identical Intel processors. Turning off the "hypervisor running" CPUID bit 31 in XCP-ng using the technique provided by @olivierlambert remains an essential prerequisite step. VMware's hypervisor.cpuid.v0 = "FALSE" .vmx setting is the equivalent configuration change required for nested virtualization to work on ESXi. After booting identical Windows 10 VMs on each hypervisor, I ran the Windows Sysinternals CoreInfo64.exe and CoreInfo64.exe -v commands inside the guests in order to compare the processor features available in each case. The active CPU flags appeared to all be the same except that XCP-ng had three additional flags set: acpi, hle and rtm that were not present in the ESXi guest. Likewise, ESXi had the x2apic flag set which was not present in the XCP-ng guest. It appears that the corresponding x2apic CPUID bit 21 from our modified platform:featureset=CPUID 32-bit word is on, presumably making that feature available in the VM despite it showing disabled in the guest. The other thing that is different between the two is that on ESXi, the CoreInfo64.exe -v command shows that all four CPU assisted virtualization feature flags are reversed after Hyper-V is installed and activated following a reboot. In other words, hypervisor present becomes true (Hyper-V is now active) and the other three flags: vmx, ept and urg (unrestricted guest) are all false. I continue to be frustrated that there doesn't seem to be an authoritative document for all of the platform:mapkey=value pairs for XenServer/XCP-ng. Similarly, there appear to be additional global hypervisor settings enabled with the /opt/xensource/libexec/xen-cmdline <param(s)> for which I can't find any documentation. For example, there is the /opt/xensource/libexec/xen-cmdline --set-xen force_software_vmcs_shadow global setting that is required by the one Citrix-supported nested virtualization configuration. Search for Bromium in the Citrix online docs for further information. If anybody knows where to find the complete documentation for the platform:mapped key/value pairs and the global hypervisor settings like the one cited, I'd be very interested. One other thing examined was the system Hyper-V drivers loaded in the two nested guest configs. Each appeared to have the same set of drivers loaded without any failures. However, the HV Host Service would not start on XCP-ng, yet starts fine on the working ESXi hosted VM. (Probably requires Hyper-V to be active so is an observed effect rather than a cause.) Any suggestions for additional productive avenues of inquiry are much appreciated. I will keep working on it. 😕

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    Thanks for the detailed feedback @XCP-ng-JustGreat

                                    Let me check around if I can find more information.

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      Okay so I've been told that we should try with an extra Xen patch: https://paste.debian.net/hidden/870debb8/

                                      @stormi do we have a guide to assist on rebuilding Xen with this extra modification?

                                      X 1 Reply Last reply Reply Quote 0
                                      • X Offline
                                        XCP-ng-JustGreat @olivierlambert
                                        last edited by

                                        @olivierlambert I'm happy to try out an alternate Xen kernel if that's what we're talking about. Please let me know if there is anything else that you need from me. It feels like we are so close to solving this 🤔 . . .

                                        1 Reply Last reply Reply Quote 0
                                        • stormiS Offline
                                          stormi Vates 🪐 XCP-ng Team
                                          last edited by stormi

                                          You can use our build container https://github.com/xcp-ng/xcp-ng-build-env on the RPM sources for xen in XCP-ng 8.2 https://github.com/xcp-ng-rpms/xen/tree/8.2 to build new packages.

                                          To add a patch, add it to SOURCES/ and list it in the spec file. The patch may fail to apply as it could have been made against a different version of xen, in which case it will need to be adjusted to the current codebase.

                                          X 1 Reply Last reply Reply Quote 0
                                          • X Offline
                                            XCP-ng-JustGreat
                                            last edited by

                                            I've not compiled an OS kernel before, but I'm willing to give it a try. It will probably take me a while to get the containerized development environment set up, etc. in order to build the updated kernel. I don't pretend to understand what this code does or how it may or may not fix nested Hyper-V on XCP-ng.

                                            Here is the source code that we have been asked to integrate:

                                            diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c
                                            index e9f94daf64..4c80912368 100644
                                            --- a/xen/arch/x86/hvm/vmx/vvmx.c
                                            +++ b/xen/arch/x86/hvm/vmx/vvmx.c
                                            @@ -2237,6 +2237,7 @@ int nvmx_msr_read_intercept(unsigned int msr, u64 *msr_content)
                                                     /* 1-settings */
                                                     data = PIN_BASED_EXT_INTR_MASK |
                                                            PIN_BASED_NMI_EXITING |
                                            +               PIN_BASED_VIRTUAL_NMIS |
                                                            PIN_BASED_PREEMPT_TIMER;
                                                     data = gen_vmx_msr(data, VMX_PINBASED_CTLS_DEFAULT1, host_data);
                                                     break;
                                            
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post