High Fan Speed Issue on Lenovo ThinkSystem Servers
-
@ThierryEscande
Of course. See the attached files:What I also did notice was that networking isn't working at all with the 4.19.309 kernel. Settings were the same as before and I couldn't even ping the gateway. I tried an emergency reset and reconfiguring, but still no luck. Upon booting with the stable kernel, it worked again as expected. But that's an issue for another day, I guess.
-
I have recently purchased four SR635v3 servers with AMD EPYC 9354P cpu.
I managed to get some info from Lenovo about how to manually change the fan speeds.
How to get the existing fan speeds. If this fails, check IPMI settings in the BMC.
[09:10 xcp01 ~]# ipmitool sdr list | grep -i fan Fan Mismatch | 0x00 | ok Fan 1 Front Tach | 19072 RPM | ok Fan 2 Front Tach | 19072 RPM | ok Fan 3 Front Tach | 18944 RPM | ok Fan 4 Front Tach | 19072 RPM | ok Fan 5 Front Tach | 18944 RPM | ok Fan 6 Front Tach | 19072 RPM | ok Fan 7 Front Tach | 0 RPM | ok Fan 1 Rear Tach | 15768 RPM | ok Fan 2 Rear Tach | 15768 RPM | ok Fan 3 Rear Tach | 15876 RPM | ok Fan 4 Rear Tach | 15876 RPM | ok Fan 5 Rear Tach | 15768 RPM | ok Fan 6 Rear Tach | 15768 RPM | ok Fan 7 Rear Tach | 0 RPM | ok Sys Fan Pwr | 48 Watts | ok
To change the fans to a fixed percentage - note the decimal "35" this is the percentage of maximum fan speed to set to.
ipmitool raw 0x3a 0x07 0xff 35 0x01
After a short while for the fans to spin down.
Fan Mismatch | 0x00 | ok Fan 1 Front Tach | 12288 RPM | ok Fan 2 Front Tach | 12160 RPM | ok Fan 3 Front Tach | 12160 RPM | ok Fan 4 Front Tach | 12288 RPM | ok Fan 5 Front Tach | 12160 RPM | ok Fan 6 Front Tach | 12288 RPM | ok Fan 7 Front Tach | 0 RPM | ok Fan 1 Rear Tach | 10152 RPM | ok Fan 2 Rear Tach | 10152 RPM | ok Fan 3 Rear Tach | 10260 RPM | ok Fan 4 Rear Tach | 10152 RPM | ok Fan 5 Rear Tach | 10260 RPM | ok Fan 6 Rear Tach | 10152 RPM | ok Fan 7 Rear Tach | 0 RPM | ok Sys Fan Pwr | 15 Watts | ok
To revert to auto-fan control you either need to restart the BMC/XCC controller which isn't system impacting.
Please keep in mind this was stressed to me that its NOT SUPPORTED/RECOMMENDED - and could cause thermal damage to internal components!!
We only have the M.2 mirror kit with two small disks, no other raid cards or internal disks.
If you have RAID and internal disks you might want to fix your fans to a higher speed.Someone with scripting knowledge could also monitor a few temps and adjust the IPMI fan control accordingly.
This could even be managed externally via IPMI over LAN for multiple servers. -
Forgot to mention - I pointed Lenovo at this thread, but got the usual "OS isn't supported, find an alternative"
-
Well, it would be more productive if they could only tell which component and version is needed to correctly handle their own fansโฆ
-
@olivierlambert would there be a way after GRUB to walk step by step through the boot and see where it goes wrong?
-
I had some time to test today so I upgraded the FW on a server, disabled IPMI as suggested above in the kernel and the fans remained spun up.
I disabled ACPI in grub (acpi=off) and the fans didn't spin up but then dom0 failed to fully load so that isn't great lol
Is it just a matter of the ACPI kernel driver being outdated? I'm not sure how to check that
-
@rmaclachlan If it's an ACPI issue and since Lenovo doesn't seem to be very cooperative, one could try to downgrade the firmware to a working version (i.e. one that runs fans at normal speed) and dump the ACPI table. Then upgrade to the latest firmware, dump the ACPI tables again, and then compare them.
I don't know ACPI much but I can have a look if you can share them.
The ACPI tools should be already installed on any XCP-ng host.
- Dump the ACPI tables in binary format
Do so in an empty folder as this produces numerous files
# acpidump -b
- Decompile the
dsdt.dat
file
# iasl -e ssdt*.dat -d dsdt.dat
- Do the same operations for both firmwares and share the
dsdt.dsl
files.
The files are pretty big so don't hesitate to compress them before sharing.
- Dump the ACPI tables in binary format
-
-
@rmaclachlan Thanks for the files. Did not see anything obvious at first sight.
I forgot to ask you for ssdt files too. Would it be possible to do the same with these files ?
iasl -d ssdt*.dat
(I hope you kept the old firmware ones somewhere, otherwise don't bother to downgrade again. Just share the new firmware ssdt files)
-
@ThierryEscande I kept all the files from the acpidump from both new and old fw. I've ran that on both sets of acpi dumps which produced quite a few dsl files (one per ssdt) so I've just zipped both folders for you here:
-
@rmaclachlan Thanks a lot. Unfortunately I did not find any evidence of what could be wrong from the ACPI tables.
It obviously does not come from the IPMI devices as there is no modification in this area.
So without help from Lenovo it will be difficult for us to go further. If you manage to get Lenovo involved one way or another we will be happy to collaborate and help.