High Fan Speed Issue on Lenovo ThinkSystem Servers
-
@olivierlambert ; as it's a clean install I can see if I can provide you access to the server with Xclarity Controller 2 so you have full access to a system for testing? Would that help?
-
Hey,
Just wanted to provide an update from our end as well.
We've conducted tests with various versions of the xClarity Controller firmware / UEFI.
Lenovo seems to be onto something, as they recently released a new version with the following changelog: (XClarity Controller Firmware 2.40 KAX326G).
They've also released a new UEFI in the meantime. However, the fan speed issue persists despite these updates.
We attempted to consolidate support from Lenovo, but they were unable to escalate the ticket due to XCP-ng being considered unsupported.
What seems to be working for us, as suggested by @rmaclachlan , is the UEFI version 1.41 (Build ID: KAE110K).
With this version, the fan speeds have decreased to around 9k RPMs, which, while still slightly high, is within acceptable sound levels.
That's really not the best option as they have adressed a few CVE's since this release, but at least we can start setting up the Server without getting angry calls all the time..
The XClarity Controller firmware doesn't have an impact on the fan speed at all as it seems. -
@RIX_IT said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
XClarity Controller Firmware 2.40 KAX326G
On our system we are running XClarity Controller Firmware 2.40 KAX326G, having similar issues.
-
@LennertvdBerg As mentioned, XCC firmware doesn't seem to change anything regarding the fan speed. So we're sticking with the latest version for now. You could try rolling back to UEFI 1.41 (Build ID: KAE110K) as well and see if it makes a difference.
-
Please keep us posted, more details we have, more chances we can find what's missing in XCP-ng to make it work correctly
-
Could one of you try the
kernel-altpackage? It is not meant for production as it is not fully tested and supported, but if a higher patch level of the 4.19 helps, it could give us more idea of what's happening.EDIT: it should be updated to a new patch level soon-ish, so if current one does not fix, we should soon have another shot with a more recent update.
-
@bleader ; do you have somewhere the instructions how to do this procedure?
-
You can find information about
kernel-althere: https://docs.xcp-ng.org/installation/hardware/#-alternate-kernel -
Just did that on 8.2 (kernel-alt.x86_64 0:4.19.265-1.xcpng8.2), not the testing one yet.
Got a few errors on startup, not related afaik, but still fyi:
That didn't make a difference as it seems:
I'm on UEFI 1.41, could try the most recent somewhat later once people leave their offices.
-
@gduperrey said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
https://docs.xcp-ng.org/installation/hardware/#-alternate-kernel
I tried installing the kernel-alt using
yum install kernel-altHowever I receive the following error;
python: can't open file '/opt/xensource/bin/updategrub.py': [Errno 2] No such file or directory warning: %postun(kernel-alt-4.19.227-5.xcpng8.3.x86_64) scriptlet failed, exit status 2
-
@RIX_IT I tried kernel-alt before rolling back the uefi and it didn't help. Also tried the new Xen beta stormi posted (4.19 was it?) in case something was added but that didn't fix it either.
Thanks for trying the new Lenovo fw - I saw it come out and was going to test when I had time but now I don't need to!
-
Did it come fully equipped?
Did you install Lenovo signed HDD in it? Or you have compatibles HDD.
If you have compatibility, try to put a HDD with (original) Lenovo firmware.
I had the same problem with an Lenovo ThinkSystem SR630, I had installed a Lenovo non-firmware HDD, all of them. I bought one with Lenovo firmware, I installed and the problem disappeared. -
@Gheppy Our SR635v3 are running factory installed Lenovo hardware, although I did test swapping out the Broadcom 57504 OCP NIC with Intel X710 but the fan issue persisted. Thanks for the suggestion!
-
@Gheppy I just installed Ubuntu 22.044 LTS with kernel 5.15.0-102-generic just to test if there could be anything like a 'vendor lock'. Using Ubuntu I just see my memory temperatures and all my fan speeds are around 6000 rpm. So it really seems to be something with XCP and Lenovo.
-
Try to install lm-sensors and see what it detects.
It seems to me that the kernel it does not have drivers for IPMI BMC KCS.
My fans have 8000 - 8100 rpmyum -y install lm_sensors sensors -v sensors-detect sensors -
@Gheppy I've just reinstalled xcp-ng-8.3.0-beta2 after my Ubuntu experiment and installed lm_sensors. The output is indeed:
Driver `to-be-written': * ISA bus, address 0xcc0 Chip `IPMI BMC KCS' (confidence: 8) Note: there is no driver for IPMI BMC KCS yet. Check http://www.lm-sensors.org/wiki/Devices for updates. No modules to load, skipping modules configuration. Unloading i2c-dev... OK Unloading cpuid... OKThe complete output is:
What will be the solution for this?
-
We need to talk to the XCP-ng team and see if they can provide the module for the xen kernel.
-
@olivierlambert can you help us with providing the module for the xen kernel, which @Gheppy is talking about?
-
@LennertvdBerg The above message is indicating there is no driver written for IPMI BMC KCS for the LM_Sensors application, not that a kernel module is missing in XCPNG.
You can read more about this on the lm_sensors github issue https://github.com/lm-sensors/lm-sensors/issues/69
If you wish to view sensor data in XPCNG you can do so through IPMI still using ipmitool
ipmitool sensorThis will list the sensors in the server.
DIMM 1 | 0x0 | discrete | 0x0080| na | na | na | na | na | na DIMM 1 Temp | na | degrees C | na | na | na | na | 85.000 | 87.000 | 91.000 DIMM 2 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 2 Temp | 23.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 3 | 0x0 | discrete | 0x0080| na | na | na | na | na | na DIMM 3 Temp | na | degrees C | na | na | na | na | 85.000 | 87.000 | 91.000 DIMM 4 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 4 Temp | 25.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 5 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 5 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 6 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 6 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 7 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 7 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 8 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 8 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 9 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 9 Temp | 25.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 10 | 0x0 | discrete | 0x0080| na | na | na | na | na | na DIMM 10 Temp | na | degrees C | na | na | na | na | 85.000 | 87.000 | 91.000 DIMM 11 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 11 Temp | 24.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000The issue is when upgrading the UEFI we start seeing the sensor data read NA for the RAM modules which spins up the fans on the server, I don't know how to determine what on the OS is causing that but it sounds like something is trying to read that information and is locking up the sensor.
closed Driver for IPMI BMC KCS #69
-
@rmaclachlan Thanks. I'm also unsure how we can determine what in the OS is causing this issue. Are there other installations or modifications we could try to help isolate the problem, such as another Linux distribution with the same kernel, to see if it's a kernel-related issue? @gduperrey or @olivierlambert any suggestions how we can help the team with identifying this?
