[RHEL kernel bug] XCP vm fails to boot after newest kernel applied.

olivierlambert

This is a dual bug:

In the Linux PV driver upstream
How it was backported by RedHat

Both upstream and RH are aware, avoid doing the update for now until we have the confirmation it's fixed.

@stormi we need someone assigned to this to monitor and report any news and progress.

stormi

I also passed the information to Almalinux maintainers, they're now aware of the situation.

bberndt

Just popping in to say, I also found this today.
Yesterday and today was my set aside time to do some updates on my VMs, and came across this today.

Last night they were updating to 4.18.0-553.47.1.el8_10.x86_64
today, 4.18.0-553.50.1.el8_10.x86_64

in my looking around I thought this was related to something that was called: support for x86-64 v2, or lack there of. My CPU's on the problems are E5-2620 v0, and a E5-2430 V2. Moving one of the problem VMs (all Rock Linux 8, I believe) to a Xeon SIlver 4210 made no help. Turns out not the issue, but something to be aware of RHEL 10 is losing this v2 support, which would affect me.

Greg_E

@bberndt

Not to go off topic, but lack of v2 and lack of uefi were reasons I moved my lab to newer hardware.

As of yesterday, the xenserver management agent was still at v8.4, I'm guessing the drivers haven't changed either.

I have one Alma 9 on my vSphere lab, I'll have to update it on Monday and see if it breaks. Debian 12 on XCP-ng 8.3 is not effected as of yesterday.

bberndt

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

Not to go off topic, but lack of v2 and lack of uefi were reasons I moved my lab to newer hardware.

@Greg_E you're not wrong, but I don't get to hold the checkbook.

Greg_E

@bberndt

Wasn't a choice I wanted either, but decided I had to do it. Went with HP T740 for everything which is way less than I'd really like, but seems to be working so far for my lab.

I'll probably load up an Alma 9 (assuming it is affected too) later today. If anyone knows that this is limited to Alma 8, then I'll switch gears and use 8. Need a simple LAMP stack running to test something, and need some other VMs for backup testing too.

bberndt

@olivierlambert @kpark

Can you please let me know where this bug is filed (I see some numbers mentioned above), so I might keep an eye on it for my own curiosity. A PM is fine as well.

Thanks!

Greg_E

@bberndt
I just installed a fresh Alma 8, did yum update to see what would happen, and it's still working. Gave out the same kernel as above.

This was installed UEFI on XCP-ng 8.3 which was a fresh install a few days ago from a nightly (near release?) ISO. It was installed to an NFS share, 2 cores and 4GB with an Intel i1000 interface. Xenserver tools 8.4.0-1 installed.

There are no extra packages installed yet, could this be a package conflict.

Anything else I can check to see why mine works and others are failing?

bberndt

@Greg_E
I checked a few of mine, and they appear to all be BIOS mode. not UEFI.
I've had a couple hardware machines as well, that updated OK. I know at least one was UEFI.

Greg_E

@bberndt that's why I mentioned uefi, wondering if legacy is part of the problem. I won't have time to fiddle with this for a while, broke a couple things today that I need to fix, and need to set up glpi for some testing.

stormi

@bberndt https://bugzilla.redhat.com/show_bug.cgi?id=2331326

There's also a KB now: https://access.redhat.com/solutions/7116307

stormi

CCing @anthonyper who is tasked with following this regression closely and letting us know about any progress.

bberndt

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt that's why I mentioned uefi, wondering if legacy is part of the problem. I won't have time to fiddle with this for a while, broke a couple things today that I need to fix, and need to set up glpi for some testing.

Made a new Rocky Linux 8 install, on a lab host. UEFI boot mode, and mostly all defaults. on XCP-ng 8.2 on a E5 2620 v0 host.
Used the guest tool from the Rocky and or EPEL repository. (added the EPEL repo and then installed xe-guest-utilities)
Does NOT boot after updating to the latest kernel.

Greg_E

@bberndt

Is there something different about Alma? Friday I may have some time to fiddle and can try a Rocky 8 to see what happens.

Is this an 8.2 and 8.3 issue or just 8.2? I have 8.2 in production and could try there, 8.3 in my lab with a very fresh build.

bberndt

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

Is there something different about Alma? Friday I may have some time to fiddle and can try a Rocky 8 to see what happens.

Is this an 8.2 and 8.3 issue or just 8.2? I have 8.2 in production and could try there, 8.3 in my lab with a very fresh build.

Migrated to a XCP-ng 8.3 host. Xeon E5-2689 v4
No change.

Greg_E

@bberndt

If I find an hour, I'll give Rocky 8 a try.

Could you use LEAPP to migrate that to Alma, maybe they are doing something differently which is why mine are working.

I can tell you, there is nothing special that I'm doing, my systems are as vanilla as they get.

bberndt

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

If I find an hour, I'll give Rocky 8 a try.

Could you use LEAPP to migrate that to Alma, maybe they are doing something differently which is why mine are working.

I can tell you, there is nothing special that I'm doing, my systems are as vanilla as they get.

@Greg_E
google AI says what I think is familiar:
Rocky Linux strives for 1:1 bug-for-bug compatibility with RHEL, while Alma Linux is more of an RHEL rebuild, making some adjustments and adding its own features

Greg_E

@bberndt

Ok, that might explain the difference.

Would a LEAPP from Rocky 8 up to Alma 9 be possible and solve the issue?

bufanda

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt
I just installed a fresh Alma 8, did yum update to see what would happen, and it's still working. Gave out the same kernel as above.

This was installed UEFI on XCP-ng 8.3 which was a fresh install a few days ago from a nightly (near release?) ISO. It was installed to an NFS share, 2 cores and 4GB with an Intel i1000 interface. Xenserver tools 8.4.0-1 installed.

There are no extra packages installed yet, could this be a package conflict.

Anything else I can check to see why mine works and others are failing?

Hmm....I did the same and that VM died just like any other with the broken kernel.
Although I used the Cloud Image so not completly new install from scratch.
Maybe I'll check building a new cloud image what will happen then.

Greg_E

@bufanda

How many of these that are failing have been upgraded from a previous version? Could it be something left over from EL7 or early EL8?

My Alma 8.10 base install is still running fine, did a yum update to apply a few more things and reboot and still working with the same kernel version above. But again, this was a clean fresh install, not something that's been running for a while.