Early testable PVH support

TeddyAstie

Hello !

Xen supports 3 virtualization modes, PV (deprecated), HVM (used in XCP-ng) and PVH.
While HVM is supported in XCP-ng (and used), PVH hasn't been integrated yet, but today in XCP-ng 8.3 we have some early support for it.

The PVH mode has been officially introduced in Xen 4.10 as leaner, simpler variant of HVM (it was initially named HVM-lite) with little to no emulation, only PV devices, and less overall complexity.
It aims to be a great and simpler alternative to traditional HVM for modern guests.

A quick comparison of all modes
PV mode :

needs specific guest support
only PV devices (no legacy hardware)
relies on PV MMU (less efficient than VT-x EPT/AMD-V NPT overall, but works without virtualization technologies)
unsafe against Spectre-style attacks
supports: direct kernel boot, pygrub
deprecated

HVM mode :

emulate a real-behaving machine (using QEMU)
- including legacy platform hardware (IOAPIC, HPET, PIT, PIC, ...)
- including (maybe legacy) I/O hardware (network card, storage ...)
- some can be disabled by the guest (PVHVM), but they exist at the start of the guest
relies on VT-x/AMD-V
traditional PC boot flow (BIOS/UEFI)
optional PV devices (opt-in by guest; PVHVM)
performs better than PV mode on most machines
compatible with pretty much all guests (including Windows and legacy OS)

PVH mode :

relies on VT-x/AMD-V (regarding that, on the Xen side, it's using the same code as HVM)
minimal emulation (e.g no QEMU), way simpler overall, lower overhead
only PV devices
support : direct kernel boot (like PV), PVH-GRUB, or UEFI boot (PVH-OVMF)
needs guest support (but much less intrusive than PV)
works with most Linux distros and most BSD; doesn't work with Windows (yet)

Installation

Keep in mind that this is very experimental and not officially supported.

PVH vncterm patches (optional)

While XCP-ng 8.3 actually has support for PVH, due to a XAPI bug, you will not be able to access the guest console. I provide a patched XAPI with a patched console.

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# You may need to update to testing repositories.
yum update --enablerepo=xcp-ng-testing

# Installing the patched XAPI packages (you should see `.pvh` XAPI packages)
yum update --enablerepo=xcp-ng-tae2

This is optional, but you probably want that to see what's going on in your guest without having to rely on SSH or xl console.

Making/converting into a PVH guest

You can convert any guest into a PVH guest by modifying its domain-type parameter.

xe vm-param-set uuid={UUID} domain-type=pvh

And revert this change by changing it back to HVM

xe vm-param-set uuid={UUID} domain-type=hvm

PVH OVMF (boot using UEFI)

You also need a PVH-specific OVMF build that can be used to boot the guest in UEFI mode.

Currently, there is no package available for getting it, but I provide a custom-built OVMF with PVH support
https://nextcloud.vates.tech/index.php/s/L8a4meCLp8aZnGZ

You need to place this file in the host as /var/lib/xcp/guest/pvh-ovmf.elf (create all missing parents).
Then sets it as PV-kernel

xe vm-param-set uuid={UUID} PV-kernel=/var/lib/xcp/guest/pvh-ovmf.elf

Once done, you can boot your guest as usual.

Tested guests

On many Linux distros, you need to add console=hvc0 in the cmdline, otherwise, you may not have access to a PV console.

Alpine Linux
Debian

Known limitations

Some stats shows "no stats" (XAPI bug ?)
No support for booting from ISO, you can workaround this by importing your iso as a disk and using it as read-only disk
No live migration support (or at least, don't expect it to work properly)
No PCI passthrough support
No actual display (only PV console)