High Fan Speed Issue on Lenovo ThinkSystem Servers
-
Let me know if you figure this out, I've been fighting with this issue for the last couple days as well. We have SR635v3 servers with AMD EPYC 9354P processors. After GRUB the fans spin down, and the kernel begins to load, once I see efi: EFI_MEMMAP is not enabled the fans spin back up.
In the interim the fix has been to roll back UEFI firmware back to 1.43 which loses some microcode updates but at least our fans aren't screaming at 16k RPM in an office environment.
-
I may be wrong, but I don't think we have much in term of controlling/handling cpu fans and such in XCP-ng or in Xen itself for that matter.
@rmaclachlan were you testing on 8.2 too? I don't think there is much chance 8.3 beta 2 would change much in this regard, but it would be interesting to be sure this also happens on it.
-
@bleader yeah I'm not sure how the OS could affect that either and certainly haven't seen that in the past...very odd.
I am on 8.3 beta 2. I haven't tested with 8.2
-
Given Rix_IT is on 8.2 that allows us to know both versions seems affected, thanks!
-
@bleader My older UEFI fix didn't work long as the fans were spun up again this morning. The servers are nice and quiet when booting to AlmaLinux or like RIX_IT mentioned Windows Server so could this be an issue with Kernal 4.19 on XCP not playing well with new EPYC chipsets?
-
Thank you all for the input!
I'll be experimenting with other Linux distributions officially supported by Lenovo to see if the issue persists.
I'm not really optimistic about receiving assistance from Lenovo regarding XCP, but maybe if we are able to reproduce elsewhere, they will point us in the right direction.
Rest assured, I'll keep you updated on any progress made.
-
@rmaclachlan it could be the kernel, but there is Xen between the kernel and the hardware, which does for example handle the cpu frequency scaling. If you're on a test machine and can spare the time maybe you can give a shot to Xen 4.17 on XCP-ng 8.3. But to be honest I would not think this will change much, but who knows until it's tested
-
@bleader, I followed your suggestion and updated our installation from 8.2 to 8.3 and went through the steps outlined in the post you mentioned.
Unfortunately, there were no changes regarding our fan speed issue - as expected.Also, I installed RHEL 8.8 (Kernel 4.18) and Ubuntu Server 18.06 (Kernel 5.4), both officially supported by Lenovo. However, the fans were still running at a comfortable 5000 RPMs in both cases. It appears that the problem is specific to XCP.
I've digged through the logs but haven't found anything really obvious.In the kernel log, I noticed a couple of warnings:
- Warning: Processor Platform Limit not supported.
- WARNING: CPU: < VARYING CORES > PID: 1 at drivers/i2c/busses/i2c-designware-common.c:245 i2c_dw_clk_rate+0x16/0x30
Additionally, I observed that the mcelog service failed, also I found a warning checking journalctl.
Given that we're running a beta build, I'm uncertain if these issues are related, or if they even matter, but I wanted to share this information in case it may be helpful.
-
@rmaclachlan have you found a solution in the meantime.
-
@RIX_IT I rolled my XCC and UEFI back to last May and the fans finally quieted down.
BMC Version
2.10 (Build ID: KAX318R)
UEFI Version
1.41 (Build ID: KAE110K)I noticed when the fans spun up the XCC page was showing multiple DIMMs temp as NA and when I checked
ipmitool sensorin the XCPNG terminal the DIMM # Temp was also 0 when the fans were up. I'm assuming the old kernel XCPNG runs is causing some havoc with the UEFI Lenovo has on these servers and is preventing that sensor from being read so XCC freaks out and spins the fans up because it assumes the temp on that sensor is really high. No idea why the old XCC/UEFI seem to work though.
-
Weird but thanks a lot @rmaclachlan for the feedback!
-
We have exactly the same issue with a Lenovo ThinkSystem SR665 V3 (Model #7D9AA01SEA) running a single AMD EPYC 9354 32C. We are running currently the following latest firmwares on our system;
We installed xcp-ng-8.3.0-beta2 (due to the Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter which is by default not supported in XCP 8.2.1) and have exactly similar issues. When the server is turned on but not in XCP yet, we get the following temperature reading in Xclarity Controller 2 (having 2x 64 GB RAM MTC40F2046S1RC48BR);
However, when XCP is booted we get the following temperature reading;
This results indeed that the fans go to max power due to missing readings.The downgrade of Firmwares I haven't tried yet, but would be nice if this could be solved in XCP. We have another identical system in stock which we still need to install. I'm planning to try the installation on that server without upgrading the default factory firmwares and see if that works better.
-
I wonder what's missing exactly and where so we can bundle it (or not if it's not possible, but since it seems to work in a 4.18 kernel…)
-
Adding @gduperrey in the conversation
-
@olivierlambert ; as it's a clean install I can see if I can provide you access to the server with Xclarity Controller 2 so you have full access to a system for testing? Would that help?
-
Hey,
Just wanted to provide an update from our end as well.
We've conducted tests with various versions of the xClarity Controller firmware / UEFI.
Lenovo seems to be onto something, as they recently released a new version with the following changelog: (XClarity Controller Firmware 2.40 KAX326G).
They've also released a new UEFI in the meantime. However, the fan speed issue persists despite these updates.
We attempted to consolidate support from Lenovo, but they were unable to escalate the ticket due to XCP-ng being considered unsupported.
What seems to be working for us, as suggested by @rmaclachlan , is the UEFI version 1.41 (Build ID: KAE110K).
With this version, the fan speeds have decreased to around 9k RPMs, which, while still slightly high, is within acceptable sound levels.
That's really not the best option as they have adressed a few CVE's since this release, but at least we can start setting up the Server without getting angry calls all the time..
The XClarity Controller firmware doesn't have an impact on the fan speed at all as it seems. -
@RIX_IT said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
XClarity Controller Firmware 2.40 KAX326G
On our system we are running XClarity Controller Firmware 2.40 KAX326G, having similar issues.
-
@LennertvdBerg As mentioned, XCC firmware doesn't seem to change anything regarding the fan speed. So we're sticking with the latest version for now. You could try rolling back to UEFI 1.41 (Build ID: KAE110K) as well and see if it makes a difference.
-
Please keep us posted, more details we have, more chances we can find what's missing in XCP-ng to make it work correctly
-
Could one of you try the
kernel-altpackage? It is not meant for production as it is not fully tested and supported, but if a higher patch level of the 4.19 helps, it could give us more idea of what's happening.EDIT: it should be updated to a new patch level soon-ish, so if current one does not fix, we should soon have another shot with a more recent update.
