Nested Virtualization of Windows Hyper-V on XCP-ng
-
@Chuckz Yeah it would be a nice feature to see. I think the issue though is how much work it takes when it's not something anyone should be using in production. It's really just a heavy homelab feature.
I want it to work, don't get me wrong, but no big org should be doing nested virt, it's just not a good idea and even Hyper-V recommends against it.
-
@Chuckz Yeah it would be a nice feature to see. I think the issue though is how much work it takes when it's not something anyone should be using in production. It's really just a heavy homelab feature.
I want it to work, don't get me wrong, but no big org should be doing nested virt, it's just not a good idea and even Hyper-V recommends against it.
For now, I think you are right except for Windows-centric shops. Going forward, there is no doubt that running Windows will be only on Hyper-V unless third party hypervisors can maintain support for the increasing number of features in Windows that rely on NV support. For example, important security features in Windows 11 such as core isolation do work on my Windows 11 guests, I suspect also because of lack of NV support in Xen. I also think over time this NV feature will become important also for other platforms that depend more on Linux than Windows does.
Has anyone here seen Windows 11 core isolation working on XCP-ng? One can check on a Windows 11 XCP-ng guest by looking at Windows Security -> Device Security -> Core Isolation -> Core Isolation Details in the Windows guest. I bet in every case it reports that it does not work. When I try to enable it, it successfully enables it and notifies me I need to reboot for the new setting to take effect, but when I reboot the core isolation feature is disabled again.
Apparently Windows virtualization, while important, is not important enough for deep-pocketed customers to push for this feature in XCP-ng and Xen upstream. The question I raise is, can a group of XCP-ng users, perhaps working in their home labs, get the ball rolling in upstream Xen without deep-pocket customers asking them to add the NV feature to Xen? I am hoping yes, because I think the Xen upstream developers really want to add this feature (Vates too, because it is a big negative for XCP-ng compared to other options that do support these Windows features that depend on NV).
But the Xen developers do not have the time to work on NV without deep-pocket customers asking for the feature. We can greatly improve the probability that the Xen developers will work on NV if we can do some of the work for them. I think there are some things we can do to help the Xen developers support NV. This is what I am proposing.
-
Chuckz said:
For example, important security features in Windows 11 such as core isolation do work on my Windows 11 guests...Edit: That is a typo. I meant to say that core isolation does not work on my Windows 11 guests, and I suspect it is because of the lack of the NV feature in Xen and XCP-ng.
So my point is that over time, you can forget about running Windows on any other hypervisor except Hyper-V if it is true that we can never use NV in production.
-
See my answer toward Nested Virtualization - https://xcp-ng.org/forum/post/105019
-
@acebmxer My take on those links: Nested virt is not currently supported by XCP-ng or Vates, but XCP-ng is committed to support nested virt at some time in the future.
-
XenServer developers recently contributed a patch series that removes a bit of technical debt from Xen, doing which was one of the steps towards proper nested virtualization support. There still remains a large amount of work onwards.
-
@stormi Indeed it is only very modest work so far. I am looking forward to seeing Xen upstream developers working in earnest on supporting this feature. I don't see much on the master branch of xen.git yet beyond these modest patch sets removing some technical debt, as you say. I suppose I would need to look at the xen-devel mailing list also for a better idea of how hard the Xen developers are working to support this feature.
To be perfectly honest, unless I am mistaken, I could not recommend XCP-ng today for users that need to run Windows in virtualized environments. This is mainly due to the lack of nested virtualization support which, AFAICT, is needed to support Windows 11 core isolation, an important security feature of Windows 11.
However, I could be wrong about that. Correct me if I am wrong. Does XCP-ng support core isolation in Windows 11 guests via some mechanism other than by nested virtualization? I presume that I cannot get core isolation to work in Windows guests because of lack of NV support in Xen.
-
@Chuckz Why do you need Core Isolation enabled in a VM? Core Isolation is designed to protect processes within Windows 11 by using VBS, if you're already isolating the VM I don't see a huge reason to have it enabled.
It's worth noting again that Microsoft themselves says to NOT use Nested Virt for production use, very specifically in their own documentation.
I get what you're wanting here but reality is 99% of places don't need nested virtualization and if they do they should probably rethink it since it's not considered stable or production ready on ANY hypervisor. This isn't specific to XCP-ng.
Hyper-V has probably the best nested virt support and even they say it should not be used in production environments.
I'm not saying I don't want this feature to work better, I do. But I can't imagine it should be a priority for Vates or anyone working on Xen because it's not really needed for production setups.
If I am missing some reason you have to have this enabled please let me know, but virtualizing Windows just to nest another Windows so you can enable Core Isolation is really cumbersome and not worth any benefits it provides as far as I can tell.
-
@Chuckz Why do you need Core Isolation enabled in a VM? Core Isolation is designed to protect processes within Windows 11 by using VBS, if you're already isolating the VM I don't see a huge reason to have it enabled.
It's worth noting again that Microsoft themselves says to NOT use Nested Virt for production use, very specifically in their own documentation.
I get what you're wanting here but reality is 99% of places don't need nested virtualization and if they do they should probably rethink it since it's not considered stable or production ready on ANY hypervisor. This isn't specific to XCP-ng.
Hyper-V has probably the best nested virt support and even they say it should not be used in production environments.
I'm not saying I don't want this feature to work better, I do. But I can't imagine it should be a priority for Vates or anyone working on Xen because it's not really needed for production setups.
If I am missing some reason you have to have this enabled please let me know, but virtualizing Windows just to nest another Windows so you can enable Core Isolation is really cumbersome and not worth any benefits it provides as far as I can tell.
Hi @planedrop Sure, I agree the isolation that XCP-ng's Xen hypervisor provides protection of other guests on the host from a compromised Windows guest, but it won't protect the confidential data and other services provided by that compromised Windows guest. Nor will it protect the other hosts on the network from attacks that can spread laterally via the network rather than via security holes between different guests on the host. The hypervisor is only one piece of the security process, it is not enough. So I want to protect that Windows guest from attacks that could have been stopped with core isolation, because I don't want even a single device or guest in my infrastructure compromised.
-
I'm not saying I don't want this feature to work better, I do. But I can't imagine it should be a priority for Vates or anyone working on Xen because it's not really needed for production setups.
I respectfully disagree. This should be a priority for Vates and Xen because currently, neither Vates nor Xen can provide the core isolation feature for their Windows guests. I think it is a mistake to think the isolation that Xen / Vates / XCP-ng provides between guests on a XCP-ng host can provide the same level of security for a Windows guest that the core isolation feature that nested Hyper-V can provide for that Windows guest.
-
@Chuckz wouldn't the better solution here be to prioritize making Core Isolation work within XCP-ng guests rather than focusing on nested virtualization?
Nested virt has other issues and again should not really be used with high priority VMs.
I guess that's the whole thing I'm getting at, nested virt isn't the fix for this specific issue.
-
That's interesting. Microsoft's Enterprise Connected Cache is a containerized setup on Linux containers, however the installer is only available for Windows. Running Linux-based containers requires nested virtualization.
This is what got me wondering about nested virtualization support for XCP-ng.
-
@Byte0 I think this is a fair use case, containers are a bit different and what you described is basically how containers work on most setups. I mean K8s on XCP-ng works that way, you deploy VMs which then have containers running inside them.
So not quite the same as nested virt.
-
@Chuckz wouldn't the better solution here be to prioritize making Core Isolation work within XCP-ng guests rather than focusing on nested virtualization?
Nested virt has other issues and again should not really be used with high priority VMs.
I guess that's the whole thing I'm getting at, nested virt isn't the fix for this specific issue.
@planedrop That would be cool if core isolation could work directly in XCP-ng guests without help of nested Hyper-V. I already asked @stormi if this is possible in earlier post today:
Chuckz said:
Does XCP-ng support core isolation in Windows 11 guests via some mechanism other than by nested virtualization? I presume that I cannot get core isolation to work in Windows guests because of lack of NV support in Xen.
So far no reply to that question but I only asked earlier today. I presume it is a complex technical question that cannot be answered without discussion with upstream Xen developers.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login