High Fan Speed Issue on Lenovo ThinkSystem Servers
-
@RIX_IT I tried kernel-alt before rolling back the uefi and it didn't help. Also tried the new Xen beta stormi posted (4.19 was it?) in case something was added but that didn't fix it either.
Thanks for trying the new Lenovo fw - I saw it come out and was going to test when I had time but now I don't need to!
-
Did it come fully equipped?
Did you install Lenovo signed HDD in it? Or you have compatibles HDD.
If you have compatibility, try to put a HDD with (original) Lenovo firmware.
I had the same problem with an Lenovo ThinkSystem SR630, I had installed a Lenovo non-firmware HDD, all of them. I bought one with Lenovo firmware, I installed and the problem disappeared. -
@Gheppy Our SR635v3 are running factory installed Lenovo hardware, although I did test swapping out the Broadcom 57504 OCP NIC with Intel X710 but the fan issue persisted. Thanks for the suggestion!
-
@Gheppy I just installed Ubuntu 22.044 LTS with kernel 5.15.0-102-generic just to test if there could be anything like a 'vendor lock'. Using Ubuntu I just see my memory temperatures and all my fan speeds are around 6000 rpm. So it really seems to be something with XCP and Lenovo.
-
Try to install lm-sensors and see what it detects.
It seems to me that the kernel it does not have drivers for IPMI BMC KCS.
My fans have 8000 - 8100 rpmyum -y install lm_sensors sensors -v sensors-detect sensors -
@Gheppy I've just reinstalled xcp-ng-8.3.0-beta2 after my Ubuntu experiment and installed lm_sensors. The output is indeed:
Driver `to-be-written': * ISA bus, address 0xcc0 Chip `IPMI BMC KCS' (confidence: 8) Note: there is no driver for IPMI BMC KCS yet. Check http://www.lm-sensors.org/wiki/Devices for updates. No modules to load, skipping modules configuration. Unloading i2c-dev... OK Unloading cpuid... OKThe complete output is:
What will be the solution for this?
-
We need to talk to the XCP-ng team and see if they can provide the module for the xen kernel.
-
@olivierlambert can you help us with providing the module for the xen kernel, which @Gheppy is talking about?
-
@LennertvdBerg The above message is indicating there is no driver written for IPMI BMC KCS for the LM_Sensors application, not that a kernel module is missing in XCPNG.
You can read more about this on the lm_sensors github issue https://github.com/lm-sensors/lm-sensors/issues/69
If you wish to view sensor data in XPCNG you can do so through IPMI still using ipmitool
ipmitool sensorThis will list the sensors in the server.
DIMM 1 | 0x0 | discrete | 0x0080| na | na | na | na | na | na DIMM 1 Temp | na | degrees C | na | na | na | na | 85.000 | 87.000 | 91.000 DIMM 2 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 2 Temp | 23.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 3 | 0x0 | discrete | 0x0080| na | na | na | na | na | na DIMM 3 Temp | na | degrees C | na | na | na | na | 85.000 | 87.000 | 91.000 DIMM 4 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 4 Temp | 25.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 5 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 5 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 6 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 6 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 7 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 7 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 8 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 8 Temp | 26.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 9 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 9 Temp | 25.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000 DIMM 10 | 0x0 | discrete | 0x0080| na | na | na | na | na | na DIMM 10 Temp | na | degrees C | na | na | na | na | 85.000 | 87.000 | 91.000 DIMM 11 | 0x0 | discrete | 0x4080| na | na | na | na | na | na DIMM 11 Temp | 24.000 | degrees C | ok | na | na | na | 85.000 | 87.000 | 91.000The issue is when upgrading the UEFI we start seeing the sensor data read NA for the RAM modules which spins up the fans on the server, I don't know how to determine what on the OS is causing that but it sounds like something is trying to read that information and is locking up the sensor.
closed Driver for IPMI BMC KCS #69
-
@rmaclachlan Thanks. I'm also unsure how we can determine what in the OS is causing this issue. Are there other installations or modifications we could try to help isolate the problem, such as another Linux distribution with the same kernel, to see if it's a kernel-related issue? @gduperrey or @olivierlambert any suggestions how we can help the team with identifying this?
-
I have no idea, sorry. Hopefully someone could tell us what we need to add in XCP-ng to fix it.
-
@LennertvdBerg To verify that the IPMI drivers are not messing up with the firmware, you could try to blacklist the IPMI modules.
Create the file/etc/modprobe.d/blacklist-ipmi.confcontaining the following and rebootblacklist ipmi_si blacklist ipmi_devintf blacklist ipmi_msghandlerActually add a line for whatever modules the command
lsmod | grep ipmigives you -
@ThierryEscande . When I do
lsmod | grep ipmiI get the following resultsipmi_si 65536 0 ipmi_devintf 20480 0 ipmi_msghandler 61440 2 ipmi_devintf,ipmi_siSo, I created the file with
vi /etc/modprobe.d/blacklist-ipmi.confand added the following:blacklist ipmi_si blacklist ipmi_devintf blacklist ipmi_msghandlerI saved the file and rebooted the system using shutdown -r now. However, I still don't see the memory temperatures in Xclarity, and the server's fans are still running at over 13,000 RPM. The system is running XCP-NG 8.3 beta 2 with kernel 4.19.0+1.
-
We have a new
kernel-altrelease available for testing purpose. This is only for XCP-ng 8.2.1 for now. The package for XCP-ng 8.3 should land soon.
THIS IS A TEMPORARY REPO FOR TESTING PURPOSE ONLY To install this
kernel-altpackage:- Edit the
xcp-ng.repofile
# vi /etc/yum.repos.d/xcp-ng.repo- Add the following lines at the end of the file:
[xcp-ng-tescande] name=XCP-ng 8.2 tescande User Repository baseurl=https://koji.xcp-ng.org/repos/user/8/8.2/tescande1/x86_64/ enabled=0 gpgcheck=0 priority=1- Install the
kernel-altpackage
# yum --enablerepo=xcp-ng-tescande install kernel-alt- Reboot and select
XCP-ng kernel-alt 4.19.309at the grub screen.
- Edit the
-
@ThierryEscande
I've upgraded to the kernel-alt 4.19.309 while still using the old UEFI (kae110k 1.41). Currently the IPMI modules are not blacklisted in the modprobe config.The fan speeds remain stuck at 9k RPMs, consistent with both the stable and previous alt kernel versions.
I also attempted to upgrade again to the newest UEFI (kae118m 4.11) but didn't notice any discernible difference in fan behavior - here it's still around 13k RPMs.
The ipmitool output also didn't change from the one @rmaclachlan provided.If there are any other suggestions for testing or specific logs you'd like me to provide, please let me know.
-
@RIX_IT Thanks for your feedback.
Can you please share the output of
dmesgandxl dmesg? -
Even if this will probably be same as with XCP-ng 8.2, the
kernel-altpackage is available for 8.3.Same steps as in my previous post, except for the repo URL in
xcp-ng.repo:[xcp-ng-tescande] name=XCP-ng 8.3 tescande User Repository baseurl=https://koji.xcp-ng.org/repos/user/8/8.3/tescande2/x86_64/ enabled=0 gpgcheck=0 priority=1 -
And for those on XCP-ng 8.3, it might be worth giving a try to
Xen 4.17. You'll find all the necessary information on how to upgrade in the thread Xen 4.17 on XCP-ng 8.3. -
@ThierryEscande I'm experiencing difficulties with installing the
kernel-altpackage on my system. Currently, I am using XCP-ng version 8.3.0-beta2. It appears that there might be a problem with updategrub.py. Any guidance on how to resolve this would be greatly appreciated.I've update
xcp-ng.repoand this is my output ofyum --enablerepo=xcp-ng-tescande install kernel-alt:[20:12 xcp-ng-test1 ~]# yum --enablerepo=xcp-ng-tescande install kernel-alt Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org Resolving Dependencies --> Running transaction check ---> Package kernel-alt.x86_64 0:4.19.309-1.0.lenovotest.2.xcpng8.3 will be installed --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================================================================================================================================================================================== Package Arch Version Repository Size =========================================================================================================================================================================================================================================================================================== Installing: kernel-alt x86_64 4.19.309-1.0.lenovotest.2.xcpng8.3 xcp-ng-tescande 30 M Transaction Summary =========================================================================================================================================================================================================================================================================================== Install 1 Package Total download size: 30 M Installed size: 154 M Is this ok [y/d/N]: y Downloading packages: kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64.rpm | 30 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64 1/1 /var/tmp/rpm-tmp.l9rbzO: line 9: /opt/xensource/bin/updategrub.py: No such file or directory warning: %post(kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64) scriptlet failed, exit status 127 Non-fatal POSTIN scriptlet failure in rpm package kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64 Verifying : kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64 1/1 Installed: kernel-alt.x86_64 0:4.19.309-1.0.lenovotest.2.xcpng8.3 -
@ThierryEscande I've updated to
Xen 4.17and it seems the upgrade went fine:host : xcp-ng-test1 release : 4.19.0+1 version : #1 SMP Wed Jan 24 17:19:11 CET 2024 machine : x86_64 nr_cpus : 64 max_cpu_id : 63 nr_nodes : 1 cores_per_socket : 32 threads_per_core : 2 cpu_mhz : 3245.126 hw_caps : 178bf3ff:7efa320b:2e500800:244037ff:0000000f:f1bf97a9:00405fce:00000780 virt_caps : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2 total_memory : 130850 free_memory : 121721 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 17 xen_extra : .3-3 xen_version : 4.17.3-3 xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : $Format:%H$, pq ??? xen_commandline : dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 cc_compiler : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1) cc_compile_by : mockbuild cc_compile_domain : [unknown] cc_compile_date : Wed Feb 28 10:12:19 CET 2024 build_id : 9a011a28e29a21a7643376b36aec959253587d42 xend_config_format : 4However, the issues with the fan speeds and missing memory temperature readings still persist.
