Warning: Using (K)PTI with (at least Debian Stable) PV Linux guests may cause trouble

schories

Dear kind members,

a quick word of warning after an unpleasant experience:

Debian Stable with linux-image-4.9.0-6-amd64 - 4.9.88-1+deb9u1 works flawlessly under XCP-ng 7.4.1
however, Debian Stable linux-image-4.9.0-7-amd64 - 4.9.110-1 enables (K)PTI (Kernel page-table isolation) which causes this Kernel to crash: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903767
mitigation: either wait for 4.9.110-2 , to be released for Debian Stable soon or disable (K)PTI with: extra = 'elevator=noop pti=off'

Cheers

Alexander

olivierlambert

Are you in PV or HVM mode for your Debian guest?

schories

PV mode. It seems to be still faster than HVM.

olivierlambert

Hmm doing what kind of load? On modern hardware, it's very likely that PVHVM is superior to purely PV.

schories

Web, Mail, no HPC. That's what I thought. It's a quite modern system: Dell R730XD with Intel Xeon E5 2600 v3 and Intel C610 series chipset.

schories

Any suggestion on how to convert to PVHVM? I'll give it a try with clone/template VM. And then run some benchmarks. I am curious, too - whether it has changed. Last test I ran was with XS 7.0 as far as I remember. Most guests are PVHVM, but Debian I kept PV.

olivierlambert

You just need to add the value "BIOS order" to HVM-boot-policy key. Eg:

xe vm-param-set HVM-boot-policy="BIOS\ order" uuid=<YOUR_VM_UUID>

That's it! Check the boot order then (cdn with c as disk, d as CDROM and n for network/PXE). Can be any combination like dcn.

olivierlambert

Note: for a web server, expect +15% perfs on PVHVM vs PV. That's mainly due to how the RAM requests are handled: thanks to virt instructions, you remove the PV software layer to add/remove data in RAM (address translation is done by Xen in PV mode. In HVM, it's done by your CPU/chipset directly).

schories

Great, thank you, Olivier. I'll give it a try and will report back to you. Would be nice to finally also have Debian VMs running in PVHVM. CentOS 7.5 works flawlessly. It's about time then..

schories

Converting a Debian VM from PV (paravirtualized) to PVHVM (PV-on-HVM drivers)

Well, as often there's a little bit more to do than just running a oneliner. So I created a summary based on information found here, elsewhere and the one I added to successfully convert a Debian VM 9.x from PV to PVHVM on XCP-ng 7.4.x.

Within the Debian VM:

Reconfigure GRUB defaults in /etc/default/grub (Debian). Make sure GRUB_TERMINAL is uncommented and set to console (disabling graphical modes, which caused a black screen in XenCenter / VNC for me):

#GRUB_CMDLINE_LINUX="console=hvc0"
GRUB_TERMINAL=console

Update GRUB:

update-grub

Poweroff the vm:

poweroff

Within the XCP-ng:

Retrieve the UUID of the virtual machine:

xe vm-list name-label=your_vm_name_goes_here params=uuid

Set HVM boot mode:

xe vm-param-set uuid=your_vm_uuid_goes_here HVM-boot-policy="BIOS\ order"

Set local disk (c) and cdrom (d) as boot options:

xe vm-param-set uuid=your_vm_uuid_goes_here HVM-boot-params="cd"

Clear pygrub as boot loader:

xe vm-param-set uuid=your_vm_uuid_goes_here PV-bootloader=""

Clear the display arguments:

xe vm-param-set uuid=your_vm_uuid_goes_here PV-args=""

Find the UUID of the interface of the virtual disk:

xe vm-disk-list uuid=your_vm_uuid_goes_here

Set the disk device (VBD) as bootable:

xe vbd-param-set uuid=your_vbd_uuid_goes_here bootable=true

Start the VM and verify your VM actually is running in PVHVM mode:

https://wiki.xen.org/wiki/Xen_Linux_PV_on_HVM_drivers#Verifying_Xen_Project_Linux_PVHVM_drivers_are_using_optimizations

Hope this saves someone else a little bit of time.

schories

@olivierlambert Now, that I have PVHVM it seems I want to have the latest PVH(v2) instead.

https://wiki.xen.org/wiki/Xen_Project_Software_Overview#Summary

However, "PVH (v2) requires guests with Linux 4.11 or newer kernel." - and Debian 9.x runs on kernel 4.9.

https://lists.xenproject.org/archives/html/xen-devel/2018-01/msg00540.html

Too bad..

olivierlambert

PVH mode is not support yet in XenServer/XCP-ng anyway. It's like a HVM but lighter (better name is "HVM-lite"). Basically (and in very short, so it's not a perfectly accurate description), it removes some bits of emulation to get rid of qemu in the middle, thanks to latest hardware improvements (in virt support).

Anyway, PVHVM is still far better than PV for a lot of various loads.

bongo

@schories It should be noted here that the emulated BIOS in XCPNG7.6 seemingly doesn't care about the bootflags. If the first one (per device id) isn't bootable it will fail, even if xvdb is bootable! So if you e.g. have swap on xvda, move it to another device id at the end (and temporarily disable it in fstab or so)