CPU pegged at 100% in several Rocky Linux 8 VMs without workload in guest
-
I am afraid that we have the same problem: ~90 Rocky8 VMs migrated from VMware, pegging one CPU very often. We have suspended further migration to XCP-ng due to this issue.
Has been the root cause identified since 2024? Is there a solution or workaround (apart from upgrading to Rocky9)? -
@laszlobortel We never reached a definitive root cause and did end up fully migrating to XCP-NG from VMware.
We still have roughly 100 VMs running Rocky 8.10. The 4.18.0-553.94.1 kernels and above don't seem to have the same CPU issues but I'm not sure if that's because a kernel bug was mitigated or because we upgraded our backend storage to all flash arrays (Pure Storage C50's).
The CPU still gets pegged on a Rocky 8 VM every once in a blue moon but not often enough to warrant more time being spent tracking it down.
-
@laszlobortel we've seen far less of this issue since my last message, not sure what made it better and when. But we're still making sure to reboot monthly (during patching, as we normally do anyways) + after live migration, and that helps. We don't use load balancing, so once a VM is staying put on one hypervisor, there is no issue. Live migration and time triggers the issue for us.
What changed in our infra is upgrade to XCP-NG 8.3 and moving to XOSTOR as shared storage. We've seen no issue with AlmaLinux 9 and CloudLinux 9 at all. They also perform better I/O wise.
-
@aflons @jgrafton First of all, I would like to thank very much both of you for replying so quickly to this old thread!
Our failure rate is roughly 1 frozen VM / 90 Rocky8 VMs / day, which is not tolerable. We have further hundreds of Rocky8 VMs on VMware, waiting for migration to XCP-ng.
I tried to summarise our options:- Our kernels are pretty fresh, but we can try the very latest available for Rocky 8.
- Upgrading to Rocky 9 on the sort term is not an option. We have to migrate Rocky 8 from VMware to XCP-ng first, then we can think about switching to Rocky 9 later.
- VMware tools removed during migration as part of the migration procedure.
- We are aready on shared lvmohba storage, which is a production grade Hitachi Vantara all SSD, same as under VMware, so I see no room for change/improvement here.
- As last resort we can try disable load-balancing plugin and reboot monthly during our maintenance window, but this would be an ugly workaround.
Is there anything I forgot?
@jgrafton Was there any useful suggestion or conclusion in your Vates support ticket #7726289? I am afraid that we are facing a tricky interworking issue between the xen hypervisor and the 4.18.0 kernel and both components are independent from XCP-ng and Vates.
-
@laszlobortel yes I definately think load balancing is the issue for you. Since live migrations is the biggest trigger.
-
That would be an interesting lead to see if the issue is triggered by live migrations, this could be a hint on the issue.
-
@laszlobortel While I can understand "Upgrading not being an option" you're lift and shifting the workload (or at least have been attempting to do this to date).
Are you unable to build new and migrate data over to XCP-ng, while I could see this causing more work, lift and shifting is almost always a guaranteed way to cause headaches - like the ones you're experiencing.
That is why each service provider recommends building new if you can. At the same time that you're building new, you're updating which of course can cause issues - but continuing to run Rocky8 is only receiving security updates until 2029. Sure it has a few years left, but why not take the opportunity to upgrade?
-
@laszlobortel While I can understand "Upgrading not being an option" you're lift and shifting the workload (or at least have been attempting to do this to date).
Are you unable to build new and migrate data over to XCP-ng, while I could see this causing more work, lift and shifting is almost always a guaranteed way to cause headaches - like the ones you're experiencing.
That is why each service provider recommends building new if you can. At the same time that you're building new, you're updating which of course can cause issues - but continuing to run Rocky8 is only receiving security updates until 2029. Sure it has a few years left, but why not take the opportunity to upgrade?
They can possibly go up to Rocky 9 but Rocky 10 later may be harder, as it requires a higher baseline on the CPU. Also they may have legacy software that only works on Rocky 8.
-
@laszlobortel We concluded that older Linux kernels plus live migrations plus lvmohba storage seems to trigger the issue. Our workaround was to upgrade to a mainline 6.x kernel packaged by ElRepo https://elrepo.org/wiki/doku.php?id=start for Rocky 8 systems that were especially prone to the CPU hang.
The kernel upgrades effectively stopped the issue from occurring.
-
@laszlobortel We concluded that older Linux kernels plus live migrations plus lvmohba storage seems to trigger the issue. Our workaround was to upgrade to a mainline 6.x kernel packaged by ElRepo https://elrepo.org/wiki/doku.php?id=start for Rocky 8 systems that were especially prone to the CPU hang.
The kernel upgrades effectively stopped the issue from occurring.
That's ultra helpful interesting @jgrafton
Maybe it's even worth a KB/known issue in our official doc, let me ping @thomas-dkmt
I suppose https://docs.xcp-ng.org/troubleshooting/common-problems/ might be the right place to document it.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login