@bogikornel As discussed in the thread, pygrub doesn't work.

(It might be fixable, because copying kernel and initrd to dom-0 and directly setting PV-kernel, PV-ramdisk and PV-args works, which is basically what pygrub is supposed to do. But I stopped investigating as pvhgrub is a much better option which acually works.)

You have 3 options

1. copy kernel and initrd do dom-0 and configure

xe vm-param-set uuid=... domain-type=pvh
xe vm-param-set uuid=... PV-kernel=/dom-0/path/to/kernel
xe vm-param-set uuid=... PV-ramdisk=/dom-0/path/to/initrd
xe vm-param-set uuid=... PV-args="root=... ro console=hvc0 ..."
xe vm-param-clear uuid=... param-name=PV-bootloader

Not a very practical option, just PoC.

2. use pvh-ovmf, but this requires UEFI-enabled VM (ie. GPT disk layout with EFI partition and some EFI bootloader or kernel directly in EFI with proper config (or as UKI)

3. use pvhgrub. You need a recent grub2 to build the image for i386-xen_pvh target. Or just get the blob - this one is from Alpine package

curl https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/grub-xenhost-2.12-r8.apk | tar -xzf - --strip-components=3 usr/lib/grub-xen/grub-i386-xen_pvh.bin

Save it to dom-0 (e.g. to /var/lib/xcp/guest/grub-i386-xen_pvh.bin) and configure the VM

xe vm-param-set uuid=... domain-type=pvh
xe vm-param-set uuid=... PV-kernel=/var/lib/xcp/guest/grub-i386-xen_pvh.bin
xe vm-param-clear uuid=... param-name=PV-ramdisk
xe vm-param-clear uuid=... param-name=PV-args
xe vm-param-clear uuid=... param-name=PV-bootloader

If the VM has valid grub2 config, it should work. Of course, you need a linux kernel with CONFIG_XEN_PVH enabled.

@TeddyAstie Perfect! I didn't know pv(h)grub2. I tried and confirm if works fine with XCP-ng 8.3. So no more trouble with pygrub, PV-bootloader or anything pygrub-like to extract kernel and initrd from the VM's drive.

I'll summarize for others. You can choose between a generic UEFI boot (xe vm-param-set PV-kernel=/var/lib/xcp/guest/pvh-ovmf.elf) and direct grub execution (xe vm-param-set PV-kernel=/var/lib/xcp/guest/grub-i386-xen_pvh.bin), just get the binary from standard grub2 package.

BTW this answers @bogikornel question about booting a bios-mode VM. Not exactly a bios mode, but if a HVM is using bios firmware and grub as a bootloader, it can easily be converted to PVH and use the same grub config, with the same partition layout (works with mbr, no need for EFI partition ...).

@TeddyAstie I'm just playing with it. I installed a fresh Alpine Linux 3.21 (6.x kernel) into a (normal) HVM VM and configured it to boot using grub from UEFI (EFI partition contains just grub, kernel and initrd is inside /boot on the root partition). Works as expected.

Then I changed it to PVH using your instructions, ie. set domain-type=pvh and PV-kernel=/var/lib/xcp/guest/pvh-ovmf.elf. Again, works fine.

Then I tried to get rid of the (fake) UEFI magic. I thought it should work to just change the PV-bootloader to pygrub. Calling pygrub on the disk image works fine and is able to extract the images and args

# pygrub -l alpine.img
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
title: Alpine, with Linux virt
  root: None
  kernel: /boot/vmlinuz-virt
  args: root=UUID=4c6dcb06-20ff-4bcf-be4d-cb399244c4c6 ro  rootfstype=ext4 console=hvc0
  initrd: /boot/initramfs-virt

But starting the VM fails. It looks like it starts but then immediately something calls force shutdown, I'll dive deeper into the logs later.

But setting everything manually actually works. If extract the kernel and initrd to dom-0 and configure

PV-kernel=/var/lib/xcp/guest/kernel
PV-ramdisk=/var/lib/xcp/guest/ramdisk
PV-args="root=/dev/xvda1 ro rootfstype=ext4 console=hvc0"

it boots and I looks pretty much the same as with the pvh-ovmf magic. So perhaps the idea to use pygrub is wrong.

@TeddyAstie said in Early testable PVH support:

... PV-kernel=/var/lib/xcp/guest/pvh-ovmf.elf

Works fine. But IIUC, direct kernel boot should work as well. I tried setting pygrub, the VM loads the kernel and starts but then immediately stops. Any idea what's wrong?