Nested Virtualization in xcp-ng

AlexanderK

Right now we are migrating many servers from esxi to xcp-ng.
Many users have vms with nested virtualization enabled. Some for docker desktop some other for android studio etc.
They keep asking for this and i am searching for a solution.
Do you have any plans to implement this? Any solution to propose?
Thanks in advance

abudef

Nested virtualization doesn’t work very well in Xen. For example, when I set up a small test playground, I had XCP-ng 8.3 as the primary host and another XCP-ng running on it as a nested host. When I then booted Debian 12 on the nested host, it caused the entire nested host to crash and reboot. On the other hand, Windows VMs on the nested host run quite well.

Later, when I was preparing a test lab, I installed ESXi 8.0U3e on a Dell server and then deployed four virtualized XCP-ng 8.3 hosts on top of it. A number of Linux and Windows VMs run on them without any issues.

abudef

You can enable it in the VM configuration simply by clicking a button in Xen Orchestra:

or use this command:
~~xe vm-param-set platform:nested_virt=true uuid=<VM UUID>~~ mistake here, see below

MajorP93

@abudef said:

Nested virtualization doesn’t work very well in Xen. For example, when I set up a small test playground, I had XCP-ng 8.3 as the primary host and another XCP-ng running on it as a nested host. When I then booted Debian 12 on the nested host, it caused the entire nested host to crash and reboot. On the other hand, Windows VMs on the nested host run quite well.

Later, when I was preparing a test lab, I installed ESXi 8.0U3e on a Dell server and then deployed four virtualized XCP-ng 8.3 hosts on top of it. A number of Linux and Windows VMs run on them without any issues.

Did you follow the official documentation for nested virtualization?

https://docs.xcp-ng.org/guides/xcpng-in-a-vm/#nested-xcp-ng-using-xcp-ng

Most importantly setting via command line on pool master:

xe vm-param-set uuid=<UUID> platform:exp-nested-hvm=true

and

xe vm-param-set uuid=<UUID> platform:nic_type="e1000"

abudef

@MajorP93 Sure I did

MajorP93

@abudef said:

@MajorP93 Sure I did

Hmm you previously mentioned different command so I was not sure if you really followed documentation correctly.

abudef

@MajorP93 Overall it works, Windows VMs run more or less reliably, but I wasn’t able to resolve the nested host crashing when starting Debian. That’s why I put the free version of ESXi underneath the nested XCP-ng hosts, and that works reliably.

AlexanderK

@abudef this causes some vms not to boot

abudef

@AlexanderK said:

@abudef this causes some vms not to boot

You mean that when you enable Nested virtualization in the VM settings under the Advanced tab, the VM then fails to boot?