XCP vm fails to boot after newest kernel applied.

edsilber

We have a ticket from some users who were unable to boot their Xen vms after updating to the newest Red Hat kernel (4.18.0-553.50.1.el8_10.x86_64) That same kernel seems to work ok on VMware. We've seen references to it in the wild https://forums.almalinux.org/t/kernel-panic-4-18-0-553-50-1-el8-10-x86-64-on-xenserver-8-4/5754
We're telling our users to hold off on that patch until we know more.
The xen security advisory is from back in December but looks like the fix implementation has broken something. https://xenbits.xen.org/xsa/advisory-466.html
Any workarounds known?

olivierlambert

Hi,

For such issues, it's a lot better to create a support ticket. We could point you to get more detailed output (or serial) to understand exactly what's causing the boot to fail

edsilber

@olivierlambert said in XCP vm fails to boot after newest kernel applied.:

For such issues, it's a lot better to create a support ticket. We could point you to get more detailed output (or serial) to understand exactly what's causing the boot to fail

Reply

Sure thing, we were just trying to triage this morning. "Yes this seems to be real & Don't update until we know more" bought us a little time until we could move to next steps. Ticket incoming...

edsilber

@edsilber Ticket#7737661 submitted

olivierlambert

Investigating, thanks for the report!

yomeyo

We have the same issue here with the latest kernel since installing it yesterday.
4.18.0-553.50.1.el8_10.x86_64 does not boot while 4.18.0-553.47.1.el8_10.x86_64 boots fine.
Is there a fix or workaround available yet?

kpark

Ticket 7737647, same issue.

olivierlambert

This is a dual bug:

In the Linux PV driver upstream
How it was backported by RedHat

Both upstream and RH are aware, avoid doing the update for now until we have the confirmation it's fixed.

@stormi we need someone assigned to this to monitor and report any news and progress.

stormi

I also passed the information to Almalinux maintainers, they're now aware of the situation.

bberndt

Just popping in to say, I also found this today.
Yesterday and today was my set aside time to do some updates on my VMs, and came across this today.

Last night they were updating to 4.18.0-553.47.1.el8_10.x86_64
today, 4.18.0-553.50.1.el8_10.x86_64

in my looking around I thought this was related to something that was called: support for x86-64 v2, or lack there of. My CPU's on the problems are E5-2620 v0, and a E5-2430 V2. Moving one of the problem VMs (all Rock Linux 8, I believe) to a Xeon SIlver 4210 made no help. Turns out not the issue, but something to be aware of RHEL 10 is losing this v2 support, which would affect me.

Greg_E

@bberndt

Not to go off topic, but lack of v2 and lack of uefi were reasons I moved my lab to newer hardware.

As of yesterday, the xenserver management agent was still at v8.4, I'm guessing the drivers haven't changed either.

I have one Alma 9 on my vSphere lab, I'll have to update it on Monday and see if it breaks. Debian 12 on XCP-ng 8.3 is not effected as of yesterday.

bberndt

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

Not to go off topic, but lack of v2 and lack of uefi were reasons I moved my lab to newer hardware.

@Greg_E you're not wrong, but I don't get to hold the checkbook.

Greg_E

@bberndt

Wasn't a choice I wanted either, but decided I had to do it. Went with HP T740 for everything which is way less than I'd really like, but seems to be working so far for my lab.

I'll probably load up an Alma 9 (assuming it is affected too) later today. If anyone knows that this is limited to Alma 8, then I'll switch gears and use 8. Need a simple LAMP stack running to test something, and need some other VMs for backup testing too.

bberndt

@olivierlambert @kpark

Can you please let me know where this bug is filed (I see some numbers mentioned above), so I might keep an eye on it for my own curiosity. A PM is fine as well.

Thanks!

Greg_E

@bberndt
I just installed a fresh Alma 8, did yum update to see what would happen, and it's still working. Gave out the same kernel as above.

This was installed UEFI on XCP-ng 8.3 which was a fresh install a few days ago from a nightly (near release?) ISO. It was installed to an NFS share, 2 cores and 4GB with an Intel i1000 interface. Xenserver tools 8.4.0-1 installed.

There are no extra packages installed yet, could this be a package conflict.

Anything else I can check to see why mine works and others are failing?