Nested Virtualization of Windows Hyper-V on XCP-ng
-
Everything is on the public mailing list, I suggest you ask there
-
@alexanderk @olivierlambert Sorry to have not responded sooner to your question. It has been a very long, slow slog so far and I haven't been able to devote as much time as I'd like to working on this. Here's what I've done so far: Based on Andrew Cooper's recommendation, I installed a fully patched Windows Server 2008 R2 VM to Xen. (Hyper-V was initially released with Server 2008 so this is almost as far back as you can go.) Using the current unmodified Xen source code, the VM will permit Hyper-V to be enabled in the Windows Server 2008 R2 guest, but--as with newer versions of Windows--once you perform the finishing reboot, Hyper-V is not actually active. Adding the two recommended source-code patches, recompiling and performing the same test causes the VM to hang following the enablement of Hyper-V. I know that I need to set up a serial console for the VM in order to view any logging that might provide a clue as to what's failing during the boot, but I haven't worked that out just yet.
I've also spent some considerable time reading through the Xen Dev email posts on the history of the development of nested virtualization in Xen. One very significant learning from that reading is that nested virtualization on Xen was initially developed by an AMD developer. Development of the NV feature-set for Intel came later after the AMD-focused design die had been cast. As far as I can tell given that I'm running Server 2008 R2, this never worked on Intel. (Maybe it did on an older Intel processor, but I am currently working with SkyLake i7-6700s so have no way to test older hardware.) Unfortunately, I also don't have appropriate AMD hardware on which to perform the same test to see whether or not it might work on AMD.
On the Microsoft Hyper-V side, it seems as though the opposite evolution happened. Nested virtualization was developed on Intel first, then (very recently) AMD. This makes me suspect that it doesn't work on AMD either. In other words, I don't know that nested virtualization of Windows on Xen ever worked such that Hyper-V was actually active in the guest. I would be delighted to have somebody prove me wrong.
-
W warriorcookie referenced this topic on
-
X XCP-ng-JustGreat referenced this topic on
-
W warriorcookie referenced this topic on
-
Hello, just decided to also share my experience with this. This specific topic seems to be the closest on point.
I'm developer and yes, Win/Docker is my setup. In order to upgrade my hardware transparently I use VMs as more power is required, so that's my case.
I'm moving from VMWare recently (free ESXi 6.5 restrictions on 8 vCPUs is starting to be significant), so I'm exploring my options. XCP-ng has my sympaties because of the best import/export comparing to others, so it was my first hypervisor to try moving to.
I successfully converted the image and, as expected, without Nested V checkbox in Xen Orchestra, Hyper-V Driver in Windows wan't active.
But when I enable Nested-V - I'm not even able to boot. It's the kind of boot failure that different BIOS would help to boot. But there is no other BIOS except UEFI (I mean, single UEFI and no other UEFIs, others don't work anyway).
So technically, I can't say whether Hyper-V would work if machine booted.After that I moved to Proxmox, for which I had to specify specific BIOS (OVMF) to boot my image (so yes, it might matter). Regarding Nested-V all I had to do in Proxmox is to specify CPU:host to make it work. Which proves it is possible in KVM.
I'm trying to make a point that this topic is still important, and apart from Hyper-V itself ability to boot VM images from other hypervisors is also important, so more BIOSes or startup settings might be required.
My sympathies are still with XCP-ng because of the import/export features, they really help (and yes, kudos to XOA April 2022 release with the additional formats). If you're importing/exporting from Proxmox, be prepared to SSH/WinSCP all the time, have reserved about x1.5 of Thick VM disk size on Proxmox server or some local one where you make a conversion (but download 200G+ to your local machine first lol). I'll be back on XCP-ng as long this will be claimed as working.
-
@olivierlambert I think this issue is the single biggest problem with XCP-ng (Xen). 2nd is just general windows guest support.
While I have installed older Windows on XCP/Xen, the whole lack of Hyper-V support is an important issue because it's becoming a requirement and it works on other hypervisors. I know newer versions of Xen do a better job without guest tools installed, but they are still important.
This is a major blocker for customers moving from VMware to XCP-ng/Xen. They just can't do it because Hyper-V fails and easy/good guest tools are also a problem.
-
Thanks for your feedback. We'd love to have better support for that, but those 2 problems are hard. This require a lot of work to debug and understand what's wrong.
We are still a small company despite growing fast, so we do our best with what we have.
-
@clockware you can convert your machine from bios booting to uefi booting. Simple by converting MBR to GPT but you will need a win 10 iso
-
@AlexanderK VMWare setup had UEFI BIOS, when I'm portable on my laptop I use Hyper-V built into Win10Pro, it also uses UEFI / Gen2, in XCP-ng I tried both BIOS and UEFI settings. BIOS doesn't boot, checked just for the record. UEFI booting works, but UEFI+Nested-V doesn't (causes Win10 to troubleshoot itself). So it doesn't seem like partition recognition problem, it feels like Windows + Virtual Hardware to me (but may be UEFI image also matters, in my experience in different envs yes it might).
-
@clockware ok i misunderstood...
-
Is there any updates on nested virtualization and hyper-v?
I've tried to run docker on windows 10 vm and didn't succeed on this.
-
Sadly, there's not a lot of market traction for it. Obviously, contributions are very welcome, otherwise, it won't make progress by itself (not until we continue to grow enough to tackle more stuff in our backlog at once. Note: we almost made +30% in headcount just in 2022, so it's moving fast, but still plenty to do and we can't do everything!)
-
@olivierlambert thank you for your comment.
-
I am finding myself in this really unfortunate situation. We want to use Microsoft Connected Cache which requires Hyper-V. Microsoft Connected Cache is a way to cache inTune installation packages to reduce internet usage.
-
Is there any news about Nested Hyper-V support?
-
I'm not aware about any progress on that direction. It's more like an upstream Xen Project question by the way, you might ask on the Xen Project Matrix community: https://xenproject.org/help/matrix/
-
@olivierlambert Okay. Thank you.
-
@olivierlambert thanks. I asked. No estimate.
-
Though a little old this is the only thread that seems applicable to my needs right now... Do we have any ideas on when this might become available in Xen?
I have a client that needs to use part of Hyper-V for an application they are using. Are there any workarounds? I see Proxmox seems to support this, but I haven't tested.
Thanks!
-
There was a talk in Xen Summit about what needs to be done to complete Nested Virtualization support, but I don't know if anyone in the Xen Project followed-up.
-
@stormi Bummer... Because I really could use the ability to run Hyper-V in a Windows Server VM... It's needed for a certain application that a client of mine uses.
-
@FTSSupport I'd first consider talking to the client and/or the clients vendor that is requiring this. Nested virtualization, even on setups that work, is NEVER recommended for something that requires good up time and reliability, so it might be best to avoid this.
I have had issues with nested virt in ProxMox as well, admittedly less than in XCP-ng, but it's still not good enough on either that I'd use it consistently.
Is there a reason the application needs Hyper-V? The reason I ask is, I've had something similar due to a vendor who required ESXi for their VMs. Turns out there was no good reason for that and we ended up having to spend a ton of money for no real benefit. Has created more headaches down the road too w/ lack of central backups, etc... and massive wasted resources (time, money, all of the above). The vendor has since said they'll help us move to XCP-ng, but it's a big process and there was no reason for ESXi in the first place.
So maybe, just maybe, there's a chance the vendor/application developer can be convinced otherwise? One of the points of virtualization is keeping options open.
