XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Nested Virtualization of Windows Hyper-V on XCP-ng

    Scheduled Pinned Locked Moved Compute
    111 Posts 12 Posters 72.6k Views 12 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Online
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      So your post was great and its seriousness provided traction to move forward. Congrats!

      Andy is one of the best (likely the best) Xen expert I know. He's also a bit "pessimistic" sometimes, one would say realistic 😛 (the word "disaster" is from his point of view, I mean it's far from being great, but it's enough for some basic use cases).

      Anyway, with more and more urgency coming to get things sorted in Windows world, I'm pretty sure more resources will be attached to fix some Xen parts on that. As he said, an incremental approach will be the solution, testing some patches for Xen devs will be a tremendous help.

      That's a very good example on why cooperation is a powerful way to move forward. Thanks a lot for your efforts @XCP-ng-JustGreat 👍

      1 Reply Last reply Reply Quote 1
      • AlexanderKA Offline
        AlexanderK
        last edited by

        anything new on this issue? @XCP-ng-JustGreat @olivierlambert

        X 1 Reply Last reply Reply Quote 0
        • olivierlambertO Online
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Everything is on the public mailing list, I suggest you ask there 🙂

          1 Reply Last reply Reply Quote 0
          • X Offline
            XCP-ng-JustGreat @AlexanderK
            last edited by

            @alexanderk @olivierlambert Sorry to have not responded sooner to your question. It has been a very long, slow slog so far and I haven't been able to devote as much time as I'd like to working on this. Here's what I've done so far: Based on Andrew Cooper's recommendation, I installed a fully patched Windows Server 2008 R2 VM to Xen. (Hyper-V was initially released with Server 2008 so this is almost as far back as you can go.) Using the current unmodified Xen source code, the VM will permit Hyper-V to be enabled in the Windows Server 2008 R2 guest, but--as with newer versions of Windows--once you perform the finishing reboot, Hyper-V is not actually active. Adding the two recommended source-code patches, recompiling and performing the same test causes the VM to hang following the enablement of Hyper-V. I know that I need to set up a serial console for the VM in order to view any logging that might provide a clue as to what's failing during the boot, but I haven't worked that out just yet.

            I've also spent some considerable time reading through the Xen Dev email posts on the history of the development of nested virtualization in Xen. One very significant learning from that reading is that nested virtualization on Xen was initially developed by an AMD developer. Development of the NV feature-set for Intel came later after the AMD-focused design die had been cast. As far as I can tell given that I'm running Server 2008 R2, this never worked on Intel. (Maybe it did on an older Intel processor, but I am currently working with SkyLake i7-6700s so have no way to test older hardware.) Unfortunately, I also don't have appropriate AMD hardware on which to perform the same test to see whether or not it might work on AMD.

            On the Microsoft Hyper-V side, it seems as though the opposite evolution happened. Nested virtualization was developed on Intel first, then (very recently) AMD. This makes me suspect that it doesn't work on AMD either. In other words, I don't know that nested virtualization of Windows on Xen ever worked such that Hyper-V was actually active in the guest. I would be delighted to have somebody prove me wrong.

            1 Reply Last reply Reply Quote 1
            • W warriorcookie referenced this topic on
            • X XCP-ng-JustGreat referenced this topic on
            • W warriorcookie referenced this topic on
            • C Offline
              clockware
              last edited by clockware

              Hello, just decided to also share my experience with this. This specific topic seems to be the closest on point.

              I'm developer and yes, Win/Docker is my setup. In order to upgrade my hardware transparently I use VMs as more power is required, so that's my case.

              I'm moving from VMWare recently (free ESXi 6.5 restrictions on 8 vCPUs is starting to be significant), so I'm exploring my options. XCP-ng has my sympaties because of the best import/export comparing to others, so it was my first hypervisor to try moving to.

              I successfully converted the image and, as expected, without Nested V checkbox in Xen Orchestra, Hyper-V Driver in Windows wan't active.

              But when I enable Nested-V - I'm not even able to boot. It's the kind of boot failure that different BIOS would help to boot. But there is no other BIOS except UEFI (I mean, single UEFI and no other UEFIs, others don't work anyway).
              So technically, I can't say whether Hyper-V would work if machine booted.

              After that I moved to Proxmox, for which I had to specify specific BIOS (OVMF) to boot my image (so yes, it might matter). Regarding Nested-V all I had to do in Proxmox is to specify CPU:host to make it work. Which proves it is possible in KVM.

              I'm trying to make a point that this topic is still important, and apart from Hyper-V itself ability to boot VM images from other hypervisors is also important, so more BIOSes or startup settings might be required.

              My sympathies are still with XCP-ng because of the import/export features, they really help (and yes, kudos to XOA April 2022 release with the additional formats). If you're importing/exporting from Proxmox, be prepared to SSH/WinSCP all the time, have reserved about x1.5 of Thick VM disk size on Proxmox server or some local one where you make a conversion (but download 200G+ to your local machine first lol). I'll be back on XCP-ng as long this will be claimed as working.

              AlexanderKA 1 Reply Last reply Reply Quote 0
              • A Online
                Andrew Top contributor
                last edited by Andrew

                @olivierlambert I think this issue is the single biggest problem with XCP-ng (Xen). 2nd is just general windows guest support.

                While I have installed older Windows on XCP/Xen, the whole lack of Hyper-V support is an important issue because it's becoming a requirement and it works on other hypervisors. I know newer versions of Xen do a better job without guest tools installed, but they are still important.

                This is a major blocker for customers moving from VMware to XCP-ng/Xen. They just can't do it because Hyper-V fails and easy/good guest tools are also a problem.

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Online
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  Thanks for your feedback. We'd love to have better support for that, but those 2 problems are hard. This require a lot of work to debug and understand what's wrong.

                  We are still a small company despite growing fast, so we do our best with what we have.

                  1 Reply Last reply Reply Quote 0
                  • AlexanderKA Offline
                    AlexanderK @clockware
                    last edited by

                    @clockware you can convert your machine from bios booting to uefi booting. Simple by converting MBR to GPT but you will need a win 10 iso

                    C 1 Reply Last reply Reply Quote 0
                    • C Offline
                      clockware @AlexanderK
                      last edited by clockware

                      @AlexanderK VMWare setup had UEFI BIOS, when I'm portable on my laptop I use Hyper-V built into Win10Pro, it also uses UEFI / Gen2, in XCP-ng I tried both BIOS and UEFI settings. BIOS doesn't boot, checked just for the record. UEFI booting works, but UEFI+Nested-V doesn't (causes Win10 to troubleshoot itself). So it doesn't seem like partition recognition problem, it feels like Windows + Virtual Hardware to me (but may be UEFI image also matters, in my experience in different envs yes it might).

                      AlexanderKA 1 Reply Last reply Reply Quote 0
                      • AlexanderKA Offline
                        AlexanderK @clockware
                        last edited by

                        @clockware ok i misunderstood...

                        A 1 Reply Last reply Reply Quote 0
                        • A Offline
                          Alexander 0 @AlexanderK
                          last edited by

                          Is there any updates on nested virtualization and hyper-v?

                          I've tried to run docker on windows 10 vm and didn't succeed on this.

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Online
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Sadly, there's not a lot of market traction for it. Obviously, contributions are very welcome, otherwise, it won't make progress by itself (not until we continue to grow enough to tackle more stuff in our backlog at once. Note: we almost made +30% in headcount just in 2022, so it's moving fast, but still plenty to do and we can't do everything!)

                            A 1 Reply Last reply Reply Quote 0
                            • A Offline
                              Alexander 0 @olivierlambert
                              last edited by

                              @olivierlambert thank you for your comment.

                              1 Reply Last reply Reply Quote 0
                              • B Offline
                                Byte0
                                last edited by

                                I am finding myself in this really unfortunate situation. We want to use Microsoft Connected Cache which requires Hyper-V. Microsoft Connected Cache is a way to cache inTune installation packages to reduce internet usage.

                                1 Reply Last reply Reply Quote 0
                                • A Offline
                                  Alexander 0
                                  last edited by

                                  Is there any news about Nested Hyper-V support?

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Online
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    I'm not aware about any progress on that direction. It's more like an upstream Xen Project question by the way, you might ask on the Xen Project Matrix community: https://xenproject.org/help/matrix/

                                    B A 2 Replies Last reply Reply Quote 0
                                    • B Offline
                                      Byte0 @olivierlambert
                                      last edited by

                                      @olivierlambert Okay. Thank you. 😊

                                      1 Reply Last reply Reply Quote 0
                                      • A Offline
                                        Alexander 0 @olivierlambert
                                        last edited by

                                        @olivierlambert thanks. I asked. No estimate.

                                        1 Reply Last reply Reply Quote 0
                                        • F Offline
                                          FTSSupport
                                          last edited by

                                          Though a little old this is the only thread that seems applicable to my needs right now... Do we have any ideas on when this might become available in Xen?

                                          I have a client that needs to use part of Hyper-V for an application they are using. Are there any workarounds? I see Proxmox seems to support this, but I haven't tested.

                                          Thanks!

                                          1 Reply Last reply Reply Quote 0
                                          • stormiS Offline
                                            stormi Vates 🪐 XCP-ng Team
                                            last edited by

                                            There was a talk in Xen Summit about what needs to be done to complete Nested Virtualization support, but I don't know if anyone in the Xen Project followed-up.

                                            F 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post