"Hardware-assisted virtualization is not enabled on this host" even though platform:exp-nested-hvm=true is set

stormi

Hello.

First, I need to remind all users that Nested Virtualization is not supported even in XCP-ng 8.2.1. It is useful for testing purposes, and we do use it a lot internally (despite its flaws - it can crash badly).

It is not supported within the Xen Project either, because it hasn't yet gotten out of the experimental stage.

This means that workloads running in a VM with Nested Virtualization enabled could theoretically exploit vulnerabilities in Xen and do nasty things, possibly including compromising the host and all its VMs.

This being said, we are aware that Nested Virtualization is a useful feature in various use cases, and are committed to bring it to XCP-ng in an officially supported way in the future.

Now, regarding the current situation: changes made to Xen broke the fragile experimental nested virtualization feature. In a recent talk at Xen Summit, George Dunlap described what needs to be done to make it work, this time in a fully supported way, but now there's a lot of work for developers. In a nutshell, this means that XCP-ng 8.3 likely won't offer Nested Virtualization, even in an experimental way, or at least not at the time of its initial release.

I'll address the topic of downgrading to XCP-ng 8.2.1 in the next message.

stormi

There is a downgrade feature on our installation ISOs, that can be used to restore the backup made automatically by the installer when upgrading from 8.2.1 to 8.3 beta/rc.

Make sure you have backups.
This is supposed to be used shortly after the upgrade, because the version of the XAPI database which will be restored will be that of the backup. If you made changes which affected the metadata stored by XAPI, you may end up with a mismatch between what's really on your storage and what XAPI believes is the current state of VMs, storage, etc.
The format for storing UEFI variables for UEFI VMs changed in 8.3 and is not backwards compatible. So UEFI VMs won't start anymore. I'm not sure whether the NVRAM store is converted at upgrade time or only the first time the VMs boot (pinging @BenjiReis about this). Anyway, any UEFI VM whose NVRAM store was upgraded to the format used in 8.3 won't boot anymore when started back on 8.2.1. There are solutions, involving wiping the NVRAM store (which is enough for most VMs, but Debian, notably, may need fixing the boot loader afterwards, using a Live media).

Another option is Warm Migrating VMs from the 8.3 pool to a 8.2.1 pool, using Xen Orchestra, but point 3. above still applies to UEFI VMs.

Andrew

@stormi For Debian UEFI boot failures, which I have, if UEFI boots to a shell, you can load GRUB quickly from the shell and boot Debian then fix it from the OS (without booting an ISO).

From the UEFI shell use the command: FS0:\EFI\debian\grubx64.efi

It does not fix anything but lets you start Debian manually.

You can also copy that command into /boot/efi/startup.nsh or use GRUB to install the standard UEFI boot files that XCP can use for the next boot.

abudef

To pass the time during a long wait

Nested Virtualization (X86) Part I - George Dunlap, Xen Server:
https://www.youtube.com/watch?v=8jKGYY1Bi_o

Nested Virtualization (X86) Part II - George Dunlap, Xen Server:
https://www.youtube.com/watch?v=3MxWvVTmY1s

XCP-ng-JustGreat

@abudef Thank you for providing these links to George Dunlap's Xen Summit nested virtualization talk. It was very informative and also demonstrates a strong commitment to bringing NV to Xen Hypervisor and its derivatives. Particularly in light of Broadcom's acquisition of VMware and the resulting customer exodus, adopting XCP-ng and Vates looks to be an increasingly smart play. I will cross-post the provided links to the big NV thread on here.

abudef

As I'm waiting there, I'm wondering why is implementing nested virtualization so difficult and lengthy in the case of Xen? VMware, H-V, VirtualBox, KVM - they all support it, so I wonder what the reasons might be that Xen still doesn't...

olivierlambert

How many dev dedicated to this task on VMware or HyperV? That's the explanation, it's a question of resources. We are doing our best at Vates to do more and more Xen dev, but ramping up takes time;

abudef

@olivierlambert Well, I guess that too, but I meant something else, whether it might be somehow related to the architecture of individual hypervisors in general, whether simply the way Xen is built is a complication for implementing nested virtualization.

olivierlambert

No especially. Nested is a tricky problem that requires a fair amount of resources to be done correctly, regardless the hypervisor.

abudef

@olivierlambert So there are basically no dramatic architectural differences in the individual hypervisors in this regard?

olivierlambert

There is some diff, but not that much between HyperV, ESXi and Xen. However, it's big enough to require a lot of effort to get nested working correctly, one small mistake and your VM is dead.