It seem to happen on most RHEL derivative distributions when migrating from 8.7 to 8.8. As suggested, the bug is in the kernel.

Starting with 4.18.0-466.el8 the patch: x86/idt: Annotate alloc_intr_gate() with __init is integrated and will create the issue. It is missing x86/xen: Split HVM vector callback setup and interrupt gate allocation that should have been integrated as well.

The migration to 8.8 will move you to 4.18.0-477.* versions that are also raising this issue, that's what you reported.

We found that the 4.18.0-488 that can be found in CentOS 8 Stream integrates the missing patch, and do indeed work when installed manually.

Your report helped us identify and reproduce the issues. That allowed us to provide a callstack to Xen devs. Then Roger Pau Monné found that it was this patch missing quickly, and we were able to find which versions of the kernel RPMs were integrating it and when the fix was integrated.

This means the issue was identified on RH side, and it is now a matter of having an updated kernel in derivative distributions like Rocky and Alma.

The Rocky Linux bugtracker indeed mentions it's mostly fixed, but there are still some kernel errors present: https://bugs.rockylinux.org/view.php?id=3565#c4293

it's frustrating that it's taking them so long to fix this, especially when it seems like the bug was caused by a human error to begin with.

Hopefully things will be fixed at some point. Maybe Red Hat is focused on doing other things right now…

It seem to happen on most RHEL derivative distributions when migrating from 8.7 to 8.8. As suggested, the bug is in the kernel.

Starting with 4.18.0-466.el8 the patch: x86/idt: Annotate alloc_intr_gate() with __init is integrated and will create the issue. It is missing x86/xen: Split HVM vector callback setup and interrupt gate allocation that should have been integrated as well.

The migration to 8.8 will move you to 4.18.0-477.* versions that are also raising this issue, that's what you reported.

We found that the 4.18.0-488 that can be found in CentOS 8 Stream integrates the missing patch, and do indeed work when installed manually.

Your report helped us identify and reproduce the issues. That allowed us to provide a callstack to Xen devs. Then Roger Pau Monné found that it was this patch missing quickly, and we were able to find which versions of the kernel RPMs were integrating it and when the fix was integrated.

This means the issue was identified on RH side, and it is now a matter of having an updated kernel in derivative distributions like Rocky and Alma.

It looks like a simple VM suspend + resume also crashes the VM. Do you confirm, @Weppel?

I've created a bug report at Rocky Linux: https://bugs.rockylinux.org/view.php?id=3565

Feel free to add to this if I missed any relevant information.

This is what I saw in the log:

Jun 8 09:40:31 gpool05-nodeA xenopsd-xc: [error||4030359 ||backtrace] Connection to VM console R:a08260fac6a2 failed, exception Xenops_interface.Xenopsd_error([S(Does_not_exist);[S(VM);S(87b062dd-b53f-04d7-c49d-32deec6716d6/config)]])
Jun 8 09:40:31 gpool05-nodeA xenopsd-xc: [error||4030359 ||backtrace] Raised Xenops_interface.Xenopsd_error([S(Does_not_exist);[S(VM);S(87b062dd-b53f-04d7-c49d-32deec6716d6/config)]])
Jun 8 09:40:31 gpool05-nodeA xenopsd-xc: [error||4030359 ||xenops_interface] Xenops_interface.Xenopsd_error([S(Does_not_exist);[S(VM);S(87b062dd-b53f-04d7-c49d-32deec6716d6/config)]]) (File "xen/xenops_interface.ml", line 165, characters 51-58)

I've done some testing with a colleague and it looks to be kernel related. The stock Rocky Linux 8.8 kernel (4.18.0-477.13.1.el8_8.x86_64) causes the reboot to happen. Upgrading the kernel to kernel-lt (5.4.245-1.1.el8.elrepo.x86_64) allows the VM to be live-migrated again without reboot/crash.

We managed to reproduce the issue. We are investigating, thinking on a bug on 8.8 or a component inside it.