Guest VM UEFI NVRAM not saved / not persistent

Andrew

@olivierlambert Banging my head against the screen late at night provided the following results:

There is still a UEFI/NVRAM save issue (XCP/OVMF).
There is a Tianocore UEFI boot issue not looking for the vendor boot files correctly.
There is a Debian assumption that your UEFI firmware is not broken and supports up-to-date processes.

To workaround the GRUB UEFI boot issue with Debian, simply run the command dpkg-reconfigure grub-efi-amd64 and enable Force installation to the removable media path, this will update the Debian config. For a quick fix, run: grub-install --force-extra-removable. It will install a good old normal standard default UEFI boot config that works with XCP/Tianocore/UEFI boot (and other VM hosts having the same issue). Ubuntu already does this. FreeBSD already does this. This may cause problems with multi-OS boot on a single disk (like a desktop), but since this is a VM I don't care. One VM, One OS. Just boot it.

Some official Debian Docs about the issue.
A user report about the Debian issue on another platform
And another one

stormi

I never met this issue but we'll try to reproduce. We met a similar issue which may have the same root cause, though: if you clear the EFI variable store from a debian VM (at least the version we had tested at the time, 10 or 11, maybe both), then it becomes unbootable, where most other OSes can boot and populate it back.

stormi

Is this on XCP-ng 8.2 or 8.3?

Andrew

@stormi I'm using 8.2.1 (updated July 2024).

If you read Debian's docs about UEFI boot it explains why they do something different than everyone else and how to fix it so it works. I use UEFI boot for Debian and use the removable media path install it boots every time on XCP.

I think there are several issues, each can cause the same failure to boot the VM.

If you upgrade XOA to UEFI with Debian then you need to install the extra boot files so it always boots on XCP.

stormi

@Andrew do you still have the exact steps to reproduce? Must you start with a Debian VM booting in BIOS mode and then switch to UEFI?

Andrew

@stormi I'll build a new VM and test it.

Andrew

@stormi Simple install: Using XCP 8.2.1 pool (current updates), build new VM, use Generic Linux UEFI template, using shared pool storage, install Debian 11 from CD as UEFI boot. No custom config needed.

Guest VM boots correctly using the special EFI/debian startup. VM can reboot correctly. VM can be powered off and on and boot correctly as long as it stays on the same host in the pool. It also works correctly on a single host system even after the host reboot.

Starting the VM on a different host or live migrating to a different host in the pool and then rebooting the VM, it fails to boot correctly because it does not have the correct EFI boot variables. Booting the VM on the original host (after the guest has been used on a different host) does not fix the issue.

You can view the EFI boot config vars with the command efibootmgr. You can see the before (good) and after (non-working) output:

BootCurrent: 0004
Timeout: 0 seconds
BootOrder: 0004,0001,0000,0003,0002
Boot0000* UiApp
Boot0001* UEFI Misc Device
Boot0002* UEFI PXEv4 (MAC:D6C87DE081C8)
Boot0003* EFI Internal Shell
Boot0004* debian

Boot0000
Boot0001
Boot0002
Boot0003
Boot0004

Running grub-install on the debian VM will 'fix' the EFI problem until the VM moves again.

stormi

With such simple reproduction steps, I wonder why we haven't more reports about it.

Andrew

@stormi said

With such simple reproduction steps, I wonder why we haven't more reports about it.

Because the default VM install uses BIOS boot... ?
(edit) This is only(?) a Debian Linux UEFI boot on a XCP pool issue.

stormi

@Andrew Yes, probably, but still, I thought more people would use UEFI.

stormi

We are implementing tests in our CI to detect this issue and protect against regressions, then we'll investigate the cause.