@kagbasi-ngc Does your VM have a vTPM? If so, could you try without?

@kagbasi-ngc @Mefosheez We're trying to find developer time to diagnose the cause, but it's not the best time of the year, so I can't promise anything on delays.

@kagbasi-ngc Looks like we missed something, as the log indicates it loaded OVMF-release.fd:

Dec 13 13:54:59 VMH01 xenguest-2-build[8345]: Loaded OVMF from /usr/share/edk2/OVMF-release.fd

We'll come back to you with a better test procedure.

For now I just have a quick and dirty one, consisting in overwriting the release file with the debug one:

cp /usr/share/edk2/OVMF-debug.fd /usr/share/edk2/OVMF-release.fd

You can revert this change with yum reinstall edk2.

I'm not spooked, I just think it's the relevant error message we need to understand :).

Since this is a test pool, could you make it use the debug version of OVMF?

ln -sf OVMF-debug.fd /usr/share/edk2/OVMF.fd 

Then attempt to start the VM again, and get /var/log/daemon.log and /var/log/xensource.log once again.

Relevant logs, from daemon.log:

Dec 12 05:08:18 VMH01 qemu-dm-1[8691]: SyncPcrAllocationsAndPcrMask!
Dec 12 05:08:18 VMH01 qemu-dm-1[8691]: Set PcdTpm2Hash Mask to 0x0000000F
Dec 12 05:08:19 VMH01 qemu-dm-1[8691]: AllocatePages failed: No 0x8400 Pages is available.
Dec 12 05:08:19 VMH01 qemu-dm-1[8691]: There is only left 0x3AA8 pages memory resource to be allocated.
Dec 12 05:08:19 VMH01 qemu-dm-1[8691]: ERROR: Out of aligned pages
Dec 12 05:08:19 VMH01 qemu-dm-1[8691]: ASSERT /builddir/build/BUILD/edk2-20220801/MdeModulePkg/Core/DxeIplPeim/X64/VirtualMemory.c(814): BigPageAddress != 0

ERROR: Out of aligned pages does not look good to me.

@kagbasi-ngc I was inviting you to have a look first, but I'll try to give it a look. Still, try to find something relevant in them, it's a good troubleshooting exercise.

I assumed you wanted live migration, but if you can do with a few minutes of downtime, you can also do a warm migration: https://xen-orchestra.com/blog/warm-migration-with-xen-orchestra/

@OlegF I can't promise anything. It's designed to work, but sometimes, with such a big jump, VMs can crash. Especially if there are unfixed bugs in the sender host that manifest themselves only when migrating to some more recent versions (this happened in the past with cross pool migrations from XenServer <7.1 towards more recent pools).

@kagbasi-ngc I'd check the output of xl dmesg and the contents of /var/log/daemon.log and /var/log/xensource.log after trying to start the VM.

@Danp This is about hypervisors nested inside XCP-ng 8.3, but here as I understood it it was XCP-ng nested inside another hypervisor, so in theory it should not be related.

@kagbasi-ngc So I believe this has more to do with nested virtualization. Does the XCP-ng VM have all the required virtualization capabilities?

Regarding the "Copy the pool's default UEFI certificates to the VM" button, I believe it is only displayed if the pool was setup for Secure Boot first. However, an imported VM comes with its own certificates, so if it was booting up correctly with Secure Boot enforced before the export, it should still do so after an import, regardless of the pool state.

Does it boot if you set Secure Boot to not enforced? I don't think it will, be let's try anyway.

We are implementing tests in our CI to detect this issue and protect against regressions, then we'll investigate the cause.

The checksum is valid. Pinging @gthvn1 who worked on this plugin.

@Dezerd What does sha256sum /etc/xapi.d/plugins/ipmitool.py report?

Hi!

It's not packaged yet in XCP-ng 8.3, but you can get it from https://github.com/xcp-ng/xcp-ng-xapi-plugins/blob/gtn_add_ipmitool_plugin/SOURCES/etc/xapi.d/plugins/ipmitool.py and copy the file as /etc/xapi.d/plugins/ipmitool.py.

@Andrew Yes, probably, but still, I thought more people would use UEFI.

With such simple reproduction steps, I wonder why we haven't more reports about it.

@Andrew do you still have the exact steps to reproduce? Must you start with a Debian VM booting in BIOS mode and then switch to UEFI?

@olivierlambert said in XCP-NG server crashes/reboots unexpectedly:

Check the /var/log/xen* stuff (on top of my head, @stormi probably knows that better than me)

I think this would be the output of xl dmesg.

Also, if the kern.log file is empty, this means that there were no messages for this day, but older messages may be visible in kern.log.1, kern.log.2.gz, etc.