Nested Virtualization of Windows Hyper-V on XCP-ng
-
@stormi Sorry. I may not have been completely clear. It sounds like you are suggesting that I run a nested XCP-ng VM on my XCP-ng host. That way, all the dependencies are in place without the need for containers. Just git clone and compile in the VM running XCP-ng. Is that what you meant?
-
@xcp-ng-justgreat My fault, I misread. No, I don't suggest that. I thought that's what you were going to do so I explained how you could possibly use a nested XCP-ng for the build.
However the container solution is preferred. It may be a bit harder to setup, but then you will be able to easily rebuild any XCP-ng package, and benefit from the automation of dependency installation each time you do a build, whatever the RPM you need to build.
So now you've got two possible ways
-
@stormi Makes sense. So which OS do you guys use to host your Docker build environment?
-
I think most of us use either Fedora (not the easiest due to its bleeding edge nature, but guides exist) or Arch. But really you should pick whatever distro you manage to find an easy "install and setup docker" guide for.
-
By the way, if someone wants to create a VM image that already contains everything setup and ready to use, we'd gladly host it to make this kind of tests easier.
-
@stormi Noted. And thank you!
-
@AlexanderK Based on @olivierlambert - provided patch for /xen/arch/x86/hvm/vmx/vvmx.c module, and @stormi - provided build information plus the excellent XCP-ng docs, I was able to spin-up a CentOS 8 Stream VM on XCP-ng, install latest Docker and create the XCP-ng-build-env container using build.sh 8.2. I got a bit confused by the run.py command nipping back and forth between the host and the container so I opted to run everything inside the container as it seemed more straightforward. After pulling the XCP-ng Xen source with git, updating the SPECS file with Patch245: nested-hyper-v-on-xen.patch and adding the nested-hyper-v-on-xen.patch file to SOURCES, I was ready to rpmbuild -ba SPECS/*.spec
This is where I could use some guidance. Thirteen RPMS were successfully generated as follows:
image url)
yum list installed "xen-*" gives:
image url)
so the five matching installed packages are copied to the INCLUDE directory:
and installed from the INCLUDE directory as follows:
yum reinstall ./*
I wasn't sure if it was necessary, but I also rebuilt initrd using dracut -f then rebooted the host.Did I install the updated packages correctly?
If so, then here is the final result: The nested Windows VM where Hyper-V is installed no longer boots. It just freezes at the Tiano firmware boot screen. At that point, only a forced shutdown will bring the machine down. If I start the VM on one of the other unpatched hosts, it starts fine, though as before, Hyper-V will not activate in the guest OS. BCDEdit shows the hypervisorlaunchtype = Auto. Please let me know if something seems wrong with my build or application process and also if there are additional next steps. I haven't given up yet!
-
@xcp-ng-justgreat said in Nested Virtualization of Windows Hyper-V on XCP-ng:
Did I install the updated packages correctly?
Yes, but you could have done better, by increasing the value of the
Releasetag in the spec file so that your built RPMs are seen as updates of the installed ones. Then you would just copy all the built RPMs (or just the five that are actually needed, doesn't matter) and yumyum update *.rpmfrom the directory and yum would have detected that they update those five RPMs from the system.Example: change
Releasefrom9.11.1%{?dist}to9.11.1.0.hypervtest.1%{?dist}which has the following bonus perks:- Higher than the currently installed Xen packages
- Lower than any future security update thanks to the leading
0.. - Documented as your test build thanks to
hypervtest.in the middle. Thus when you query the RPM database to see at first glance that a test build is currently installed. - You can differentiate several iterations of your test package with the last digit: increment it each time you do a new build.
-
@stormi Thank you. I saw the versioning info in the documentation, but was anxious to give it a try and went with "quick and dirty." I'll do it that way in the future. I've been wanting to play with Docker so it was also an excellent opportunity to level-up my skills on a number of fronts. Because the Windows VM freezes on boot, we are now at an impasse. Unlike Linux which would be spraying error logs and thus giving us some clue as to what is happening, Windows is silent, sphinx-like! I know that this is a Xen problem, but am kind of mystified that after more than a decade, this still doesn't work. VMware had nested Hyper-V running on ESXi in 2008. I have read some of the developer mailing lists for the Xen hypervisor and can see that they have various heavy-hitters from Citrix and Microsoft intimately involved in the development of Xen. Does @olivierlambert or anybody else know who we might ask about this? I'm willing to apply and test code patches and configuration changes as necessary to make it happen if we can find somebody at the Xen project to make this a priority. Given all of the important new functionality in Windows that requires Hyper-V support, this is a significant deficiency that would remove Xen and its derivatives from the running against other hypervisors. Many of us like XCP-ng and XO so much that we would really like to prevent that from happening. Please let me know if there is anything at all that you need from me.
-
At some point, I think Xen mailing list would be more relevant. Let's try to sum up: it didn't work on Xen upstream, right?
-
@olivierlambert That's right. Same pre-patch behavior in pure vanilla Xen hypervisor. The guest shows all four prerequisites for running Hyper-V are available. You can turn on the Hyper-V feature in the VM, but after the finishing reboot, Hyper-V is not actually active. The vmcompute.exe (Hyper-V VMM service) is running as expected, but all hypervisor capabilities are not active.
-
Okay so that's a good case on pushing it to the upstream! That should work, I agree entirely
Upstream devs might now what to do in the end. -
@olivierlambert Cool. Thank you!
-
@stormi what do you suggest in terms of approaching Xen ML? Should we guide @XCP-ng-JustGreat or do it ourselves? I'm under the impression we should do that together at some point, but that the original requesting user is leading the questions (able to make the tests)
-
I'm unsure about the Xen ML because previous tests in this thread have shown that it does work on Vanilla then, if I followed things correctly. So if I'm not mistaken we have a Citrix Hypervisor / XCP-ng issue at hand. Maybe we'll still get help from Xen developers in terms of guidance though.
In any case, a nice post by @XCP-ng-JustGreat to Xen's user mailing list (rather than the development mailing list, which would be for issues that we can reproduce in latest vanilla Xen), that summarizes the situation and tests done would probably be useful to help them help us.
-
I missed the last few messages, so as it looks like Vanilla Xen unpatched doesn't work, that's a case for the xen-devel mailing list
-
@stormi @olivierlambert All, something stormi mentioned yesterday made me double-check the version of vanilla Xen that was packaged with the Debian 10 test distro. Turns out, it's older than the version used in XCP-ng 8.2. We really do need to see whether or not nested Hyper-V works in Xen 4.15 (the latest) before bringing it to the attention of the Xen dev ML. Toward that end, I astonished myself last night by compiling Xen 4.15 from the source code! Most of the time was spent identifying and installing the many prerequisites--now documented--so subsequent builds will be quite fast. One packaging issue remains: the final make install command installed the xen kernel etc., but did not add the grub entry to boot it. What is the proper way to add the grub menu Debian with Xen boot choice? I considered doing it in a hacky fashion using the leftover grub menu entry from the packaged version in Debian 10. Can you tell me the right way? Please let me know and I'll give it a try this weekend. Thank you.
-
@olivierlambert @stormi It was quite an odyssey getting everything to run with pure vanilla Xen 4.15 compiled from source on Debian 10.10, but I finally accomplished it. (Learned a lot in the process too!) The final sticking point was that the Windows VM xl config file previously built and working on the older version of Xen packaged with Debian 10, wouldn't boot. Something wasn't working with the guest UEFI support so I switched to BIOS boot and that worked. The net result is that nested Hyper-V installs fine as before, but still won't activate on reboot. I also note that the x2apic CPU capability is now present in the guest as it is with VMware ESXi. That flag is missing when running nested Windows under XCP-ng 8.2 on my Intel i7-6700 processor-based system. Now that we know for sure it is still not working in the very latest Xen kernel, what next steps should we take for getting this issue to the attention of the Xen developers?
-
@XCP-ng-JustGreat the next step would be to send a detailed bug report to the
xen-develmailing list (see https://xenproject.org/help/mailing-list/). You don't need to subscribe to it to post, and all answers should put your address in CC.If you want, you can first write your e-mail here for proofreading.
-
@stormi OK. I'll put it together here first.
