Nested Virtualization of Windows Hyper-V on XCP-ng
-
@stormi said in Nested Virtualization of Windows Hyper-V on XCP-ng:
Actually, Xen never officially supported Nested Virtualization. It was experimental, and broke when other needed changes were made to Xen. Now there's work to be done to make it fully supported, and this won't happen before the final release of XCP-ng 8.3. This will be documented in the release notes.
This is also an issue for us internally as we create a lot of virtual pools for our tests.
I read through a lot of the earlier posts and finally started scrolling to find this, which is the answer I was looking for. Why do I care? There is a Microsoft evaluation learning lab for things like Intune that runs in Hyper-V, basically a bunch of VHD (x) that get spawned as needed. Applications I need to teach myself. Running XCP-NG 8.3 current updates for this lab.
If it doesn't happen, then I'll just need to throw an eval version of Windows Server on something else like an HP T740 to run these labs, not the biggest issue for me.
Link for the labs if anyone is curious (free with an email registration like all the evals):
https://www.microsoft.com/en-us/evalcenter/evaluate-mem-evaluation-lab-kit
I'd think direct Docker support would be a higher priority than nested virtualization with a focus on Hyper-V. But that's just me.
-
Serious movement appears to be happening with respect to NV. See videos below cross-posted from this forum thread:
...
@XCP-ng-JustGreat Well after over a year and a half and AFAICT not much progress from upstream on NV support for Windows/Hyper-V in Xen and XCP-ng. It is discouraging that support from upstream for this feature has not come yet. So I think we, as users of XCP-ng and Xen who are interested in this feature could roll up our sleeves and start working on the problem again and hopefully jump start the process of getting this feature working in upstream Xen and XCP-ng. I will follow up with another post to propose what users of Xen and XCP-ng can do to help. -
@Chuckz Yeah it would be a nice feature to see. I think the issue though is how much work it takes when it's not something anyone should be using in production. It's really just a heavy homelab feature.
I want it to work, don't get me wrong, but no big org should be doing nested virt, it's just not a good idea and even Hyper-V recommends against it.
-
@Chuckz Yeah it would be a nice feature to see. I think the issue though is how much work it takes when it's not something anyone should be using in production. It's really just a heavy homelab feature.
I want it to work, don't get me wrong, but no big org should be doing nested virt, it's just not a good idea and even Hyper-V recommends against it.
For now, I think you are right except for Windows-centric shops. Going forward, there is no doubt that running Windows will be only on Hyper-V unless third party hypervisors can maintain support for the increasing number of features in Windows that rely on NV support. For example, important security features in Windows 11 such as core isolation do work on my Windows 11 guests, I suspect also because of lack of NV support in Xen. I also think over time this NV feature will become important also for other platforms that depend more on Linux than Windows does.
Has anyone here seen Windows 11 core isolation working on XCP-ng? One can check on a Windows 11 XCP-ng guest by looking at Windows Security -> Device Security -> Core Isolation -> Core Isolation Details in the Windows guest. I bet in every case it reports that it does not work. When I try to enable it, it successfully enables it and notifies me I need to reboot for the new setting to take effect, but when I reboot the core isolation feature is disabled again.
Apparently Windows virtualization, while important, is not important enough for deep-pocketed customers to push for this feature in XCP-ng and Xen upstream. The question I raise is, can a group of XCP-ng users, perhaps working in their home labs, get the ball rolling in upstream Xen without deep-pocket customers asking them to add the NV feature to Xen? I am hoping yes, because I think the Xen upstream developers really want to add this feature (Vates too, because it is a big negative for XCP-ng compared to other options that do support these Windows features that depend on NV).
But the Xen developers do not have the time to work on NV without deep-pocket customers asking for the feature. We can greatly improve the probability that the Xen developers will work on NV if we can do some of the work for them. I think there are some things we can do to help the Xen developers support NV. This is what I am proposing.
-
Chuckz said:
For example, important security features in Windows 11 such as core isolation do work on my Windows 11 guests...Edit: That is a typo. I meant to say that core isolation does not work on my Windows 11 guests, and I suspect it is because of the lack of the NV feature in Xen and XCP-ng.
So my point is that over time, you can forget about running Windows on any other hypervisor except Hyper-V if it is true that we can never use NV in production.
-
See my answer toward Nested Virtualization - https://xcp-ng.org/forum/post/105019
-
@acebmxer My take on those links: Nested virt is not currently supported by XCP-ng or Vates, but XCP-ng is committed to support nested virt at some time in the future.
-
XenServer developers recently contributed a patch series that removes a bit of technical debt from Xen, doing which was one of the steps towards proper nested virtualization support. There still remains a large amount of work onwards.
-
@stormi Indeed it is only very modest work so far. I am looking forward to seeing Xen upstream developers working in earnest on supporting this feature. I don't see much on the master branch of xen.git yet beyond these modest patch sets removing some technical debt, as you say. I suppose I would need to look at the xen-devel mailing list also for a better idea of how hard the Xen developers are working to support this feature.
To be perfectly honest, unless I am mistaken, I could not recommend XCP-ng today for users that need to run Windows in virtualized environments. This is mainly due to the lack of nested virtualization support which, AFAICT, is needed to support Windows 11 core isolation, an important security feature of Windows 11.
However, I could be wrong about that. Correct me if I am wrong. Does XCP-ng support core isolation in Windows 11 guests via some mechanism other than by nested virtualization? I presume that I cannot get core isolation to work in Windows guests because of lack of NV support in Xen.
-
@Chuckz Why do you need Core Isolation enabled in a VM? Core Isolation is designed to protect processes within Windows 11 by using VBS, if you're already isolating the VM I don't see a huge reason to have it enabled.
It's worth noting again that Microsoft themselves says to NOT use Nested Virt for production use, very specifically in their own documentation.
I get what you're wanting here but reality is 99% of places don't need nested virtualization and if they do they should probably rethink it since it's not considered stable or production ready on ANY hypervisor. This isn't specific to XCP-ng.
Hyper-V has probably the best nested virt support and even they say it should not be used in production environments.
I'm not saying I don't want this feature to work better, I do. But I can't imagine it should be a priority for Vates or anyone working on Xen because it's not really needed for production setups.
If I am missing some reason you have to have this enabled please let me know, but virtualizing Windows just to nest another Windows so you can enable Core Isolation is really cumbersome and not worth any benefits it provides as far as I can tell.
-
@Chuckz Why do you need Core Isolation enabled in a VM? Core Isolation is designed to protect processes within Windows 11 by using VBS, if you're already isolating the VM I don't see a huge reason to have it enabled.
It's worth noting again that Microsoft themselves says to NOT use Nested Virt for production use, very specifically in their own documentation.
I get what you're wanting here but reality is 99% of places don't need nested virtualization and if they do they should probably rethink it since it's not considered stable or production ready on ANY hypervisor. This isn't specific to XCP-ng.
Hyper-V has probably the best nested virt support and even they say it should not be used in production environments.
I'm not saying I don't want this feature to work better, I do. But I can't imagine it should be a priority for Vates or anyone working on Xen because it's not really needed for production setups.
If I am missing some reason you have to have this enabled please let me know, but virtualizing Windows just to nest another Windows so you can enable Core Isolation is really cumbersome and not worth any benefits it provides as far as I can tell.
Hi @planedrop Sure, I agree the isolation that XCP-ng's Xen hypervisor provides protection of other guests on the host from a compromised Windows guest, but it won't protect the confidential data and other services provided by that compromised Windows guest. Nor will it protect the other hosts on the network from attacks that can spread laterally via the network rather than via security holes between different guests on the host. The hypervisor is only one piece of the security process, it is not enough. So I want to protect that Windows guest from attacks that could have been stopped with core isolation, because I don't want even a single device or guest in my infrastructure compromised.
-
I'm not saying I don't want this feature to work better, I do. But I can't imagine it should be a priority for Vates or anyone working on Xen because it's not really needed for production setups.
I respectfully disagree. This should be a priority for Vates and Xen because currently, neither Vates nor Xen can provide the core isolation feature for their Windows guests. I think it is a mistake to think the isolation that Xen / Vates / XCP-ng provides between guests on a XCP-ng host can provide the same level of security for a Windows guest that the core isolation feature that nested Hyper-V can provide for that Windows guest.
-
@Chuckz wouldn't the better solution here be to prioritize making Core Isolation work within XCP-ng guests rather than focusing on nested virtualization?
Nested virt has other issues and again should not really be used with high priority VMs.
I guess that's the whole thing I'm getting at, nested virt isn't the fix for this specific issue.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login