High Fan Speed Issue on Lenovo ThinkSystem Servers

bleader

@rmaclachlan it could be the kernel, but there is Xen between the kernel and the hardware, which does for example handle the cpu frequency scaling. If you're on a test machine and can spare the time maybe you can give a shot to Xen 4.17 on XCP-ng 8.3. But to be honest I would not think this will change much, but who knows until it's tested

RIX_IT

@bleader, I followed your suggestion and updated our installation from 8.2 to 8.3 and went through the steps outlined in the post you mentioned.
Unfortunately, there were no changes regarding our fan speed issue - as expected.

Also, I installed RHEL 8.8 (Kernel 4.18) and Ubuntu Server 18.06 (Kernel 5.4), both officially supported by Lenovo. However, the fans were still running at a comfortable 5000 RPMs in both cases. It appears that the problem is specific to XCP.
I've digged through the logs but haven't found anything really obvious.

In the kernel log, I noticed a couple of warnings:

Warning: Processor Platform Limit not supported.
WARNING: CPU: < VARYING CORES > PID: 1 at drivers/i2c/busses/i2c-designware-common.c:245 i2c_dw_clk_rate+0x16/0x30

Additionally, I observed that the mcelog service failed, also I found a warning checking journalctl.

Given that we're running a beta build, I'm uncertain if these issues are related, or if they even matter, but I wanted to share this information in case it may be helpful.

RIX_IT

@rmaclachlan have you found a solution in the meantime.

rmaclachlan

@RIX_IT I rolled my XCC and UEFI back to last May and the fans finally quieted down.

BMC Version
2.10 (Build ID: KAX318R)
UEFI Version
1.41 (Build ID: KAE110K)

I noticed when the fans spun up the XCC page was showing multiple DIMMs temp as NA and when I checked

ipmitool sensor

in the XCPNG terminal the DIMM # Temp was also 0 when the fans were up. I'm assuming the old kernel XCPNG runs is causing some havoc with the UEFI Lenovo has on these servers and is preventing that sensor from being read so XCC freaks out and spins the fans up because it assumes the temp on that sensor is really high. No idea why the old XCC/UEFI seem to work though.

olivierlambert

Weird but thanks a lot @rmaclachlan for the feedback!

LennertvdBerg

We have exactly the same issue with a Lenovo ThinkSystem SR665 V3 (Model #7D9AA01SEA) running a single AMD EPYC 9354 32C. We are running currently the following latest firmwares on our system;

We installed xcp-ng-8.3.0-beta2 (due to the Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter which is by default not supported in XCP 8.2.1) and have exactly similar issues. When the server is turned on but not in XCP yet, we get the following temperature reading in Xclarity Controller 2 (having 2x 64 GB RAM MTC40F2046S1RC48BR);
Temerature reading.png
However, when XCP is booted we get the following temperature reading;
Temperature missing.png
This results indeed that the fans go to max power due to missing readings.

The downgrade of Firmwares I haven't tried yet, but would be nice if this could be solved in XCP. We have another identical system in stock which we still need to install. I'm planning to try the installation on that server without upgrading the default factory firmwares and see if that works better.

olivierlambert

I wonder what's missing exactly and where so we can bundle it (or not if it's not possible, but since it seems to work in a 4.18 kernel…)

olivierlambert

Adding @gduperrey in the conversation

LennertvdBerg

@olivierlambert ; as it's a clean install I can see if I can provide you access to the server with Xclarity Controller 2 so you have full access to a system for testing? Would that help?

RIX_IT

Hey,

Just wanted to provide an update from our end as well.

We've conducted tests with various versions of the xClarity Controller firmware / UEFI.
Lenovo seems to be onto something, as they recently released a new version with the following changelog: (XClarity Controller Firmware 2.40 KAX326G).

They've also released a new UEFI in the meantime. However, the fan speed issue persists despite these updates.

We attempted to consolidate support from Lenovo, but they were unable to escalate the ticket due to XCP-ng being considered unsupported.

What seems to be working for us, as suggested by @rmaclachlan , is the UEFI version 1.41 (Build ID: KAE110K).
With this version, the fan speeds have decreased to around 9k RPMs, which, while still slightly high, is within acceptable sound levels.
That's really not the best option as they have adressed a few CVE's since this release, but at least we can start setting up the Server without getting angry calls all the time..
The XClarity Controller firmware doesn't have an impact on the fan speed at all as it seems.

LennertvdBerg

@RIX_IT said in High Fan Speed Issue on Lenovo ThinkSystem Servers:

XClarity Controller Firmware 2.40 KAX326G

On our system we are running XClarity Controller Firmware 2.40 KAX326G, having similar issues.

RIX_IT

@LennertvdBerg As mentioned, XCC firmware doesn't seem to change anything regarding the fan speed. So we're sticking with the latest version for now. You could try rolling back to UEFI 1.41 (Build ID: KAE110K) as well and see if it makes a difference.

olivierlambert

Please keep us posted, more details we have, more chances we can find what's missing in XCP-ng to make it work correctly

bleader

Could one of you try the kernel-alt package? It is not meant for production as it is not fully tested and supported, but if a higher patch level of the 4.19 helps, it could give us more idea of what's happening.

EDIT: it should be updated to a new patch level soon-ish, so if current one does not fix, we should soon have another shot with a more recent update.

LennertvdBerg

@bleader ; do you have somewhere the instructions how to do this procedure?

gduperrey

You can find information about kernel-alt here: https://docs.xcp-ng.org/installation/hardware/#-alternate-kernel

RIX_IT

@bleader

Just did that on 8.2 (kernel-alt.x86_64 0:4.19.265-1.xcpng8.2), not the testing one yet.

Got a few errors on startup, not related afaik, but still fyi:

That didn't make a difference as it seems:

I'm on UEFI 1.41, could try the most recent somewhat later once people leave their offices.

LennertvdBerg

@gduperrey said in High Fan Speed Issue on Lenovo ThinkSystem Servers:

https://docs.xcp-ng.org/installation/hardware/#-alternate-kernel

I tried installing the kernel-alt using

yum install kernel-alt

However I receive the following error;

python: can't open file '/opt/xensource/bin/updategrub.py': [Errno 2] No such file or directory
warning: %postun(kernel-alt-4.19.227-5.xcpng8.3.x86_64) scriptlet failed, exit status 2

kernel-alt failure.png

rmaclachlan

@RIX_IT I tried kernel-alt before rolling back the uefi and it didn't help. Also tried the new Xen beta stormi posted (4.19 was it?) in case something was added but that didn't fix it either.

Thanks for trying the new Lenovo fw - I saw it come out and was going to test when I had time but now I don't need to!

Gheppy

Did it come fully equipped?
Did you install Lenovo signed HDD in it? Or you have compatibles HDD.
If you have compatibility, try to put a HDD with (original) Lenovo firmware.
I had the same problem with an Lenovo ThinkSystem SR630, I had installed a Lenovo non-firmware HDD, all of them. I bought one with Lenovo firmware, I installed and the problem disappeared.