XCP-ng

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups

    Nested Virtualization of Windows Hyper-V on XCP-ng

    Compute
    7
    86
    13840
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambert
      olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

      It's still unclear what parameter is missing on XAPI vs libxl, waiting for more details from @XCP-ng-JustGreat 🙂

      1 Reply Last reply Reply Quote 0
      • olivierlambert
        olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

        @AlexanderK can you share the VM record here please? (the one that's nested but doesn't work with Docker). A simple output of xe vm-param-list uuid=<VM UUID> will do it.

        Just checked the xl config posted by @XCP-ng-JustGreat, and I can already answer for some parameters, but I'd like to check if they are already used or not (eg platform:viridian can be set to true if it's not already the case)

        AlexanderK 1 Reply Last reply Reply Quote 0
        • AlexanderK
          AlexanderK @olivierlambert last edited by

          @olivierlambert

          ready
          https://paste.vates.fr/?baadbc38f80b47a7#39oL16iVjNAnab5oHFpfQKt8rDw9AXjk1v5dMev8RAzM

          1 Reply Last reply Reply Quote 0
          • olivierlambert
            olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

            Can you try with a Windows 10 template from the start?

            AlexanderK 1 Reply Last reply Reply Quote 0
            • AlexanderK
              AlexanderK last edited by

              @olivierlambert ok will do it.

              X 1 Reply Last reply Reply Quote 0
              • X
                XCP-ng-JustGreat @AlexanderK last edited by

                @alexanderk @olivierlambert Did a bit more testing by removing one of the four nested-virtualization parameters at a time in the windows.cfg file on pure vanilla Xen hypervisor. Omitting the cpuid setting is the one where the Hyper-V machine bus provider driver fails to load in the guest and, basically, nested Hyper-V breaks. From what I can tell, entering that key/value pair under the platform parameter in the XCP-ng VM definition has no effect after starting from a "stock" Windows 10 VM template.
                Since the platform parameter would seem to be the logical place for that value, it looks like a developer needs to look at the code to see whether or not the mechanism for passing cpuid already exists and is not documented, or instead it needs to be implemented. Thanks again to @olivierlambert and the team at Vates for your continued innovation and enhancement of this remarkably useful software! 😊

                AlexanderK 1 Reply Last reply Reply Quote 0
                • olivierlambert
                  olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

                  Okay so it's the only thing we don't have is cpuid and that make the diff. I'll ask around to see how we could have this in XCP-ng/XAPI (xenops in fact).

                  Thanks for your precious feedback @XCP-ng-JustGreat !

                  1 Reply Last reply Reply Quote 0
                  • AlexanderK
                    AlexanderK @olivierlambert last edited by

                    @olivierlambert
                    here it is from the start

                    https://paste.vates.fr/?1d9650dbc857af53#3STr5czhCrom7y4HvyNGBH6nZ1bghHaiXKuChhcDrbCK

                    1 Reply Last reply Reply Quote 0
                    • AlexanderK
                      AlexanderK @XCP-ng-JustGreat last edited by

                      @xcp-ng-justgreat
                      @olivierlambert how can we implement manually the cpuid that @XCP-ng-JustGreat has found at a running vm?

                      1 Reply Last reply Reply Quote 0
                      • olivierlambert
                        olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by olivierlambert

                        So after discussing with my favorite Xen expert, here what could be do to get a similar result on the cpuid:

                        1. Get the pool CPU ID: xe pool-param-get uuid=<pool_uuid> param-name=cpu_info param-key=features_hvm. Should be something like 1fcbfbff-f7fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000
                        2. To compute the value you want to pass on the VM, you need to clear the hypervisor bit. So the previous string will become now 1fcbfbff-77fa3203-[...]-00000000
                        3. See this value for your VM, which will override the default computed featureset for this specific VM: xe vm-param-set uuid=<VM UUID> platform:featureset=1fcbfbff-77fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000
                        4. Start the VM, and report.

                        Reference for bit positions in the bitmap: https://github.com/xen-project/xen/blob/master/xen/include/public/arch-x86/cpufeatureset.h

                        The hypervisor bit is the top one, so you need to apply (val & 0x7fffffff) on it (it might be different in your example).

                        Note: we do NOT understand why doing this will solve the issue, but at least let's try first.

                        AlexanderK X 2 Replies Last reply Reply Quote 0
                        • AlexanderK
                          AlexanderK @olivierlambert last edited by

                          @olivierlambert

                          i am trying to calculate for my vm...

                          the cpu id on pool is
                          1fcbfbff-80b82221-2993fbff-00000403-00000000-00000000-00000000-00000000-00001000-9c000000-00000000-00000000-00000000-00000000-00000000

                          i must change the second "team" of 8 digits? I suppose that it is something in hex.

                          You wrote that the hypervisor bit is the top one. What do you mean?
                          I should add on 80b82221 the value 0x7fffffff?

                          X 1 Reply Last reply Reply Quote 0
                          • X
                            XCP-ng-JustGreat @AlexanderK last edited by

                            @alexanderk I think @olivierlambert wants us to logically AND the bits there. Will give it a try later on my pool.

                            1 Reply Last reply Reply Quote 0
                            • X
                              XCP-ng-JustGreat @olivierlambert last edited by

                              @olivierlambert @AlexanderK Wow! Changing the platform:featureset=value as prescribed does indeed enable support for nested Hyper-V on XCP-ng!! This is huge!!! I will now need to begin testing some of the actual Hyper-V-dependent functions in the nested Windows guest. The msinfo32.exe command displays no apparent knowledge that the guest OS is running on a hypervisor. It simply indicates that all four prerequisites for running Hyper-V are enabled. It would appear that this platform modification effectively tells the XCP-ng VM to lie to the nested guest OS: "No, you are not running on a hypervisor." BTW, I still have all of the latest pre-release patches installed that we tested a couple of weeks ago for @stormi . Therefore, the tested Windows VM is also booting with secure boot support by @beshleman along with this additional major breakthrough. Because of nested virtualization's complexity, there will no doubt be additional hurdles we need to jump over to make the desired guest functions work, but we have made tremendous progress in a rather short span of time. Congratulations everybody! If it's possible, we may want to change the title of the post to Nested Virtualization of Windows Hyper-V on XCP-ng as that will provide an easier way for it to be found by others. I can't really adequately express my gratitude to all of you for the amazing level of support! (Pinging some others who have fought with this issue in the past and will no doubt appreciate hearing of its progress: @Noiden @Haribo112 )

                              AlexanderK 1 Reply Last reply Reply Quote 1
                              • AlexanderK
                                AlexanderK @XCP-ng-JustGreat last edited by

                                @xcp-ng-justgreat Title changed.

                                Can you help me with the AND and what exactly have you done? Give me an example to understand.

                                AlexanderK X 2 Replies Last reply Reply Quote 1
                                • AlexanderK
                                  AlexanderK @AlexanderK last edited by

                                  @alexanderk
                                  have tried something and issue on device manager is fixed but....
                                  3e43535a-085e-4d15-a701-7de2a52315b1-image.png

                                  and

                                  bce6fdf7-badd-489c-bbd2-a350069d778d-image.png

                                  X 1 Reply Last reply Reply Quote 0
                                  • X
                                    XCP-ng-JustGreat @AlexanderK last edited by

                                    @alexanderk I think you need to run the PowerShell command to get second-level (L2) nested virtualization within Hyper-V. You are now effectively running yet another VM inside of an existing VM (like Russian dolls-nested) so you need to expose the processor hardware virtualization from L1 to L2. See here: https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/user-guide/nested-virtualization

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambert
                                      olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by olivierlambert

                                      I have to admit I don't grasp everything (also I'm not a Windows expert) I still wonder why changing what we did is making it work (in details, your feeling is likely right @XCP-ng-JustGreat )

                                      So you are running Docker inside a nested Windows, which is itself nested in XCP-ng?

                                      AlexanderK 1 Reply Last reply Reply Quote 0
                                      • AlexanderK
                                        AlexanderK @olivierlambert last edited by

                                        @olivierlambert trying to do this but it is not working.

                                        @XCP-ng-JustGreat docker works for you? even hyper-v shows error.
                                        Can anyone explain with an example the calculation of new value?

                                        1 Reply Last reply Reply Quote 0
                                        • X
                                          XCP-ng-JustGreat @AlexanderK last edited by

                                          @alexanderk If you are running a current version of Windows 10, you can use the Calculator app. It has a programmer mode that allows you to enter the 32-bit source value from the platform cpuid in hex and bitwise AND it with 0x7FFFFFFF. Programmer mode is widely available on calculators for Linux too so use the tool with which you are most comfortable.

                                          AlexanderK 1 Reply Last reply Reply Quote 0
                                          • AlexanderK
                                            AlexanderK @XCP-ng-JustGreat last edited by

                                            @xcp-ng-justgreat said in Nested Virtualization of Windows Hyper-V on XCP-ng:

                                            0x7FFFFFFF

                                            there are some groups of hex digits at the cpu id.
                                            I use the second group of digits?

                                            The second group is 80b82221 and with bitwise and 7FFFFFF i have this

                                            37e55a63-3c59-4eba-ad52-a5769b4b01f7-image.png

                                            the result is 6 digits. Is this correct?

                                            X 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post