XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Nested Virtualization of Windows Hyper-V on XCP-ng

    Scheduled Pinned Locked Moved Compute
    111 Posts 12 Posters 72.2k Views 12 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • AlexanderKA Offline
      AlexanderK
      last edited by AlexanderK

      Hello everyone,

      I am trying to setup a vm with Threat Pursuit
      https://github.com/fireeye/ThreatPursuit-VM#installation-install-script
      It a custom open source vm with some utilities for forensics.

      It is based on windows and it installs Desktop Docker, which has some issues....

      3db7e8a1-1f58-4f81-b03b-9616a99691e3-image.png

      I have enabled nested virtualization at xen orchestra and followed the info of the error.

      At another vm with enabled nested virtualization i was capable of running a virtualbox vm inside the vm, so i suppose nested virtualization works.

      Anyone have faced this issue? Windows 10 vm with desktop docker?

      1 Reply Last reply Reply Quote 0
      • AlexanderKA Offline
        AlexanderK
        last edited by

        anyone using desktop docker with win10 vm?
        No one?

        X 1 Reply Last reply Reply Quote 0
        • olivierlambertO Online
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          I am not, maybe someone in the community, I don't know 🙂

          1 Reply Last reply Reply Quote 0
          • X Offline
            XCP-ng-JustGreat @AlexanderK
            last edited by XCP-ng-JustGreat

            AlexanderK Hello fellow traveler in the quest to make nested Hyper-V virtualization work on XCP-ng. A few of us have tried without success. At present, ESXi seems to be the champion for running a nested Windows VM with Hyper-V enabled to run all of the cool new Windows features such as containers, Windows Subsystem for Linux V2 (WSL2) etc. in the guest. The success of ESXi to do this was apparently due to collaboration between VMware and Microsoft rather than a lucky break. Nested virtualization on XCP-ng does work for some guest OSes, but so far, the showstopper seems to be that a Windows Hyper-V bus driver won't load in the guest. See this thread: https://xcp-ng.org/forum/topic/4070/uefi-setting-on-vm-for-nested-virtualization?_=1622861000688 I presume this means that the Windows guest is not finding the correct state as presented by the XCP-ng guest VM or some other incompatibility. The upstream Xen code has various CPU mask settings etc. to turn on nested virtualization, but it is not so clear in XenServer or XCP-ng what might fully enable this capability and make Hyper-V happy. Last I checked, Citrix Hypervisor (aka XenServer) only supports nested virtualization for one particular use case with a partner product. If you or somebody else here does get it to work, I'd certainly be interested to hear how.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Online
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by olivierlambert

              We can always create a detailed but report on XCP repo, then we can (from there) try to ask upstream Xen and see what could be done 🙂

              Obviously, this cost time and effort, so more the community bring stuff, higher is the probably to get a result at some point!

              AlexanderKA 1 Reply Last reply Reply Quote 0
              • AlexanderKA Offline
                AlexanderK @olivierlambert
                last edited by

                olivierlambert how can we create this report? i can provide everything from my issue

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Online
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  By opening an issue on https://github.com/xcp-ng/xcp and trying to reproduce it on "vanilla" Xen (eg any Arch Linux or Debian with a recent version of Xen) with xl toolstack.

                  X 1 Reply Last reply Reply Quote 0
                  • X Offline
                    XCP-ng-JustGreat @olivierlambert
                    last edited by XCP-ng-JustGreat

                    olivierlambert AlexanderK As Olivier requested, I set up a basic Xen Hypervisor configuration using Debian 10.9 (Buster) following the tutorial here https://wiki.xenproject.org/wiki/Xen_Project_Beginners_Guide in my home lab. It took a bit of work since the documentation is outdated and not entirely clear, but I was able to get a basic Windows 10 VM running on that. I then added the nested virtualization parameters from here https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen to the VM config file. [Next section edited retrospectively.] Nested Hyper-V installs and all the guest Hyper-V drivers load on the parent Xen platform, but Hyper-V does not activate following the finishing reboot. I have commented the working xl toolstack config file: windows.cfg detailing my findings if you would like me to upload that somewhere. Seems like we need to find out if it is possible to set the cpuid and hap parameters from xe vm-param-set platform:<param>=value in order to flip the appropriate switches in the underlying hypervisor. (There doesn't seem to be a documented way to fully activate the needed support right now from the xe toolstack.) It does look like all of the viridian parameters are available. Viridian exposes various interfaces and features that Hyper-V requires. The cpuid param appears to cause the guest OS to become unaware that it is running on a hypervisor. Please let me know if you'd like me to upload the working xl config file someplace. Nested virtualization was but an interesting parlor trick before, but the architecture of modern Windows is rapidly making it an essential feature for running fully functional Windows VMs.

                    AlexanderKA 1 Reply Last reply Reply Quote 0
                    • olivierlambertO Online
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      First, thanks for taking time to test it 👍

                      So did you check with the viridian param if it works on XCP-ng? Obviously, we'd love to document this behavior if it's just an extra param to add on the VM record!

                      About the config file, you might be able to paste it directly here? Otherwise, go for https://paste.vates.fr/

                      X 1 Reply Last reply Reply Quote 0
                      • AlexanderKA Offline
                        AlexanderK @XCP-ng-JustGreat
                        last edited by

                        XCP-ng-JustGreat can you make a guide? what have you done and it is working? Will save time and will be really appreciated

                        1 Reply Last reply Reply Quote 2
                        • X Offline
                          XCP-ng-JustGreat @olivierlambert
                          last edited by XCP-ng-JustGreat

                          olivierlambert AlexanderK Unfortunately, I could not make it work on XCP-ng. I tried entering the same name/key pairs under the VM platform category, but it does not look like the xe toolstack maps them correctly to the hypervisor. I didn't find documentation for applying them from xe. The success was only realized in Xen hypervisor.

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Online
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Please share the exact key/values you used to make it work on "vanilla" Xen. It could be trivial to "port" it to XCP-ng, but for that we need that you share what you found 🙂

                            X 2 Replies Last reply Reply Quote 0
                            • X Offline
                              XCP-ng-JustGreat @olivierlambert
                              last edited by

                              olivierlambert AlexanderK I pasted my xl config file here: https://paste.vates.fr/?449c18ba665cd704#GzDDKa4fui6jqbvG7ssT5TpH7uj8uBVrg7zojpGqYmu7

                              1 Reply Last reply Reply Quote 0
                              • X Offline
                                XCP-ng-JustGreat @olivierlambert
                                last edited by

                                olivierlambert AlexanderK One final thing, here is the Xen reference manual link for the xl config file parameters (I learned a lot reading it.) https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html The outstanding issue appears to be whether or not there is a corresponding setting configurable from the xe toolstack to turn on all of the required parameter/value pairs as implemented by my test configuration on the pure Xen hypervisor. If so, we should then be able to make nested virtualization of Hyper-V work on XCP-ng.

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Online
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  As long as we know exactly what's missing, we can fix it 🙂

                                  AlexanderKA 1 Reply Last reply Reply Quote 0
                                  • AlexanderKA Offline
                                    AlexanderK @olivierlambert
                                    last edited by

                                    olivierlambert said in Windows 10 Vm and Desktop Docker Issue:

                                    As long as we know exactly what's missing, we can fix it 🙂

                                    can't wait for the fix

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Online
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      It's still unclear what parameter is missing on XAPI vs libxl, waiting for more details from XCP-ng-JustGreat 🙂

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Online
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        AlexanderK can you share the VM record here please? (the one that's nested but doesn't work with Docker). A simple output of xe vm-param-list uuid=<VM UUID> will do it.

                                        Just checked the xl config posted by XCP-ng-JustGreat, and I can already answer for some parameters, but I'd like to check if they are already used or not (eg platform:viridian can be set to true if it's not already the case)

                                        AlexanderKA 1 Reply Last reply Reply Quote 0
                                        • AlexanderKA Offline
                                          AlexanderK @olivierlambert
                                          last edited by

                                          olivierlambert

                                          ready
                                          https://paste.vates.fr/?baadbc38f80b47a7#39oL16iVjNAnab5oHFpfQKt8rDw9AXjk1v5dMev8RAzM

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Online
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by

                                            Can you try with a Windows 10 template from the start?

                                            AlexanderKA 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post