Nested Virtualization of Windows Hyper-V on XCP-ng
-
@FTSSupport I'd first consider talking to the client and/or the clients vendor that is requiring this. Nested virtualization, even on setups that work, is NEVER recommended for something that requires good up time and reliability, so it might be best to avoid this.
I have had issues with nested virt in ProxMox as well, admittedly less than in XCP-ng, but it's still not good enough on either that I'd use it consistently.
Is there a reason the application needs Hyper-V? The reason I ask is, I've had something similar due to a vendor who required ESXi for their VMs. Turns out there was no good reason for that and we ended up having to spend a ton of money for no real benefit. Has created more headaches down the road too w/ lack of central backups, etc... and massive wasted resources (time, money, all of the above). The vendor has since said they'll help us move to XCP-ng, but it's a big process and there was no reason for ESXi in the first place.
So maybe, just maybe, there's a chance the vendor/application developer can be convinced otherwise? One of the points of virtualization is keeping options open.
-
@planedrop Just to keep things cohesive in these threads, I'm linking my response https://xcp-ng.org/forum/post/81103
But yeah I may take your advice and call the vendor.
-
i was searching for nested virtualization on windows vm because i wanted to install docker desktop and nested virtualization should have been enabled
-
@AlexanderK Right, which in some ways in similar in what I'm needing, because there are applications that need Hyper-V.
-
@FTSSupport Gotcha, yeah I think the first step would be finding out if there are workarounds from the vendor, then go from there if there aren't. May just require spinning up a physical Hyper-V box, but that would definitely suck. The entire idea of hypervisors was to not have to run so many different bare metal installs of things lol.
Good luck!
I may play around with nested Hyper-V here soon to see what I can get working, but honestly I'd still avoid it EVEN IF we can get it working stable, could be a headache to troubleshoot down the road.
-
@planedrop In general, the fact is that the current XCP-ng 8.3 RC1 and XenServer 8 simply do not support any kind of nested virtualization. For example, when I want to try a mock setup of say 5 hypervisors, I just don't buy or dedicate 5 hw servers and rather make virtualized hypervisors. And for the record, I'm referring to the lab environment for debugging configurations and scenarios, not production.
P. S.: In more legacy Xen, which is part of XCP-ng 8.2, nested virtualization works.
-
Sure but the solution to that is usually to buy a single server and then try each hypervisor on it in a lab type setup to see which you prefer the best.
I still agree that it would be nice to have 1 box that runs them all, for sure, but there aren't any that can actually do that.
-
@abudef said in Nested Virtualization of Windows Hyper-V on XCP-ng:
@planedrop In general, the fact is that the current XCP-ng 8.3 RC1 and XenServer 8 simply do not support any kind of nested virtualization.
Indeed.
Actually, Xen never officially supported Nested Virtualization. It was experimental, and broke when other needed changes were made to Xen. Now there's work to be done to make it fully supported, and this won't happen before the final release of XCP-ng 8.3. This will be documented in the release notes.
This is also an issue for us internally as we create a lot of virtual pools for our tests.
-
@stormi Excited to see more progress on this for sure.
Still should never be done in a production setup though so I don't think there should be any rush ha!
Would be very cool to have in lab environments though.
-
@planedrop Yeah Appreciated! I'm going to do a conference call with the vendor next week hopefully... They won't talk with me since I'm an MSP for my client... Stupid...
I'm experimenting with Nested virtualization with Proxmox... The application is installing but throwing errors, but I don't know if that's because of it being nested or if it's just throwing errors...
Regardless I may try XCP-NG 8.2 latest and see how it reacts with nested virtualization and just stick with that if it works... It'll be easier to backwards migrate everything from 8.3 to 8.2 than moving to Proxmox (I think?)
-
@FTSSupport Should be easier to move back to 8.2 than Proxmox, yes. I'm still not sure it's going to work that well though, either in 8.2 or in Proxmox. I've just never had a good experience with nested virt.
I got it working well on a Hyper-V setup once, but the nested VM still had some odd issues, bad latency, and a few other things, and that was all Windows based stuff so it was kinda a best case scenario.
Good luck with the vendor, hopefully they can be convinced that it will work just fine on other hypervisors.
One vendor I worked with that required ESXi for their Windows VM finally changed their minds and worked w/ me to do some validation on XCP-ng. After we ran through a ton of testing (this was a very high bandwidth/data usage platform with strict requirements) the engineers were flabbergasted because XCP-ng performed so much better lol. I was like "you're really surprised something is faster than bloated ESXi??"
-
@planedrop Here's hoping! Though I'm not hopeful because their installer application hooks into Hyper-V and creates an Ubuntu server on there and then extract tarballs into it... It's very strange.
-
@FTSSupport That's an interesting way to do it instead of just having an OVA file lol.
I'm honestly a little surprised any vendors require Hyper-V too, like, if you're going to require something, why not use the industry standard that is ESXi?
And if that was the case, it would be an OVA which would be something you could natively import to most hypervisors anyway.
What an interesting situation lol.
-
@planedrop Thought you might be interested to hear my results... The Vendor A) refuses to talk to me since I'm an MSP and not a direct employee of my client... Stupid, but I sent questions to my clients to give them...
They have straight-up refused to discuss trying anything outside of Hyper-V and completely acted stupid when I mentioned something about moving the VHD that their software creates in Hyper-V and put it in our hypervisor directly... Said that's unsupported and we will not assist you.
Still trying to figure out how to get around this... I had Hyper-V running on a Proxmox VM but the software seems to stall out and never finish installing. I'm trying XCP-NG 8.2 but based on all the conversations here, I'm assuming Hyper-V will refuse to run.
-
@FTSSupport Yeah that's really annoying, oof, sorry to hear that.
Maybe the only option is going to be getting a new physical host and then using Hyper-V on that? Hate to say it, but sounds like it could be the final result.
I did that with ESXi for this one vendor since I didn't have a choice at the time, it was lame, but the company understood the need for the expense, though it helped that we were going to need another (albeit not as powerful) host anyway.
-
Serious movement appears to be happening with respect to NV. See videos below cross-posted from this forum thread:
Nested Virtualization (X86) Part I - George Dunlap, Xen Server:
https://www.youtube.com/watch?v=8jKGYY1Bi_oNested Virtualization (X86) Part II - George Dunlap, Xen Server:
https://www.youtube.com/watch?v=3MxWvVTmY1s -
@stormi said in Nested Virtualization of Windows Hyper-V on XCP-ng:
Actually, Xen never officially supported Nested Virtualization. It was experimental, and broke when other needed changes were made to Xen. Now there's work to be done to make it fully supported, and this won't happen before the final release of XCP-ng 8.3. This will be documented in the release notes.
This is also an issue for us internally as we create a lot of virtual pools for our tests.
I read through a lot of the earlier posts and finally started scrolling to find this, which is the answer I was looking for. Why do I care? There is a Microsoft evaluation learning lab for things like Intune that runs in Hyper-V, basically a bunch of VHD (x) that get spawned as needed. Applications I need to teach myself. Running XCP-NG 8.3 current updates for this lab.
If it doesn't happen, then I'll just need to throw an eval version of Windows Server on something else like an HP T740 to run these labs, not the biggest issue for me.
Link for the labs if anyone is curious (free with an email registration like all the evals):
https://www.microsoft.com/en-us/evalcenter/evaluate-mem-evaluation-lab-kit
I'd think direct Docker support would be a higher priority than nested virtualization with a focus on Hyper-V. But that's just me.
