XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Nested Virtualization of Windows Hyper-V on XCP-ng

    Scheduled Pinned Locked Moved Compute
    111 Posts 12 Posters 72.2k Views 12 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      First, thanks for taking time to test it 👍

      So did you check with the viridian param if it works on XCP-ng? Obviously, we'd love to document this behavior if it's just an extra param to add on the VM record!

      About the config file, you might be able to paste it directly here? Otherwise, go for https://paste.vates.fr/

      X 1 Reply Last reply Reply Quote 0
      • AlexanderKA Offline
        AlexanderK @XCP-ng-JustGreat
        last edited by

        @xcp-ng-justgreat can you make a guide? what have you done and it is working? Will save time and will be really appreciated

        1 Reply Last reply Reply Quote 2
        • X Offline
          XCP-ng-JustGreat @olivierlambert
          last edited by XCP-ng-JustGreat

          @olivierlambert @AlexanderK Unfortunately, I could not make it work on XCP-ng. I tried entering the same name/key pairs under the VM platform category, but it does not look like the xe toolstack maps them correctly to the hypervisor. I didn't find documentation for applying them from xe. The success was only realized in Xen hypervisor.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Please share the exact key/values you used to make it work on "vanilla" Xen. It could be trivial to "port" it to XCP-ng, but for that we need that you share what you found 🙂

            X 2 Replies Last reply Reply Quote 0
            • X Offline
              XCP-ng-JustGreat @olivierlambert
              last edited by

              @olivierlambert @AlexanderK I pasted my xl config file here: https://paste.vates.fr/?449c18ba665cd704#GzDDKa4fui6jqbvG7ssT5TpH7uj8uBVrg7zojpGqYmu7

              1 Reply Last reply Reply Quote 0
              • X Offline
                XCP-ng-JustGreat @olivierlambert
                last edited by

                @olivierlambert @AlexanderK One final thing, here is the Xen reference manual link for the xl config file parameters (I learned a lot reading it.) https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html The outstanding issue appears to be whether or not there is a corresponding setting configurable from the xe toolstack to turn on all of the required parameter/value pairs as implemented by my test configuration on the pure Xen hypervisor. If so, we should then be able to make nested virtualization of Hyper-V work on XCP-ng.

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  As long as we know exactly what's missing, we can fix it 🙂

                  AlexanderKA 1 Reply Last reply Reply Quote 0
                  • AlexanderKA Offline
                    AlexanderK @olivierlambert
                    last edited by

                    @olivierlambert said in Windows 10 Vm and Desktop Docker Issue:

                    As long as we know exactly what's missing, we can fix it 🙂

                    can't wait for the fix

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      It's still unclear what parameter is missing on XAPI vs libxl, waiting for more details from @XCP-ng-JustGreat 🙂

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        @AlexanderK can you share the VM record here please? (the one that's nested but doesn't work with Docker). A simple output of xe vm-param-list uuid=<VM UUID> will do it.

                        Just checked the xl config posted by @XCP-ng-JustGreat, and I can already answer for some parameters, but I'd like to check if they are already used or not (eg platform:viridian can be set to true if it's not already the case)

                        AlexanderKA 1 Reply Last reply Reply Quote 0
                        • AlexanderKA Offline
                          AlexanderK @olivierlambert
                          last edited by

                          @olivierlambert

                          ready
                          https://paste.vates.fr/?baadbc38f80b47a7#39oL16iVjNAnab5oHFpfQKt8rDw9AXjk1v5dMev8RAzM

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Can you try with a Windows 10 template from the start?

                            AlexanderKA 1 Reply Last reply Reply Quote 0
                            • AlexanderKA Offline
                              AlexanderK
                              last edited by

                              @olivierlambert ok will do it.

                              X 1 Reply Last reply Reply Quote 0
                              • X Offline
                                XCP-ng-JustGreat @AlexanderK
                                last edited by

                                @alexanderk @olivierlambert Did a bit more testing by removing one of the four nested-virtualization parameters at a time in the windows.cfg file on pure vanilla Xen hypervisor. Omitting the cpuid setting is the one where the Hyper-V machine bus provider driver fails to load in the guest and, basically, nested Hyper-V breaks. From what I can tell, entering that key/value pair under the platform parameter in the XCP-ng VM definition has no effect after starting from a "stock" Windows 10 VM template.
                                Since the platform parameter would seem to be the logical place for that value, it looks like a developer needs to look at the code to see whether or not the mechanism for passing cpuid already exists and is not documented, or instead it needs to be implemented. Thanks again to @olivierlambert and the team at Vates for your continued innovation and enhancement of this remarkably useful software! 😊

                                AlexanderKA 1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  Okay so it's the only thing we don't have is cpuid and that make the diff. I'll ask around to see how we could have this in XCP-ng/XAPI (xenops in fact).

                                  Thanks for your precious feedback @XCP-ng-JustGreat !

                                  1 Reply Last reply Reply Quote 0
                                  • AlexanderKA Offline
                                    AlexanderK @olivierlambert
                                    last edited by

                                    @olivierlambert
                                    here it is from the start

                                    https://paste.vates.fr/?1d9650dbc857af53#3STr5czhCrom7y4HvyNGBH6nZ1bghHaiXKuChhcDrbCK

                                    1 Reply Last reply Reply Quote 0
                                    • AlexanderKA Offline
                                      AlexanderK @XCP-ng-JustGreat
                                      last edited by

                                      @xcp-ng-justgreat
                                      @olivierlambert how can we implement manually the cpuid that @XCP-ng-JustGreat has found at a running vm?

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by olivierlambert

                                        So after discussing with my favorite Xen expert, here what could be do to get a similar result on the cpuid:

                                        1. Get the pool CPU ID: xe pool-param-get uuid=<pool_uuid> param-name=cpu_info param-key=features_hvm. Should be something like 1fcbfbff-f7fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000
                                        2. To compute the value you want to pass on the VM, you need to clear the hypervisor bit. So the previous string will become now 1fcbfbff-77fa3203-[...]-00000000
                                        3. See this value for your VM, which will override the default computed featureset for this specific VM: xe vm-param-set uuid=<VM UUID> platform:featureset=1fcbfbff-77fa3203-2d93fbff-00000523-0000000f-009c07ab-00000000-00000000-00101000-9c000400-00000000-00000000
                                        4. Start the VM, and report.

                                        Reference for bit positions in the bitmap: https://github.com/xen-project/xen/blob/master/xen/include/public/arch-x86/cpufeatureset.h

                                        The hypervisor bit is the top one, so you need to apply (val & 0x7fffffff) on it (it might be different in your example).

                                        Note: we do NOT understand why doing this will solve the issue, but at least let's try first.

                                        AlexanderKA X 2 Replies Last reply Reply Quote 0
                                        • AlexanderKA Offline
                                          AlexanderK @olivierlambert
                                          last edited by

                                          @olivierlambert

                                          i am trying to calculate for my vm...

                                          the cpu id on pool is
                                          1fcbfbff-80b82221-2993fbff-00000403-00000000-00000000-00000000-00000000-00001000-9c000000-00000000-00000000-00000000-00000000-00000000

                                          i must change the second "team" of 8 digits? I suppose that it is something in hex.

                                          You wrote that the hypervisor bit is the top one. What do you mean?
                                          I should add on 80b82221 the value 0x7fffffff?

                                          X 1 Reply Last reply Reply Quote 0
                                          • X Offline
                                            XCP-ng-JustGreat @AlexanderK
                                            last edited by

                                            @alexanderk I think @olivierlambert wants us to logically AND the bits there. Will give it a try later on my pool.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post