High Fan Speed Issue on Lenovo ThinkSystem Servers

Riven

Forgot to mention - I pointed Lenovo at this thread, but got the usual "OS isn't supported, find an alternative"

olivierlambert

Well, it would be more productive if they could only tell which component and version is needed to correctly handle their own fans…

LennertvdBerg

@olivierlambert would there be a way after GRUB to walk step by step through the boot and see where it goes wrong?

rmaclachlan

I had some time to test today so I upgraded the FW on a server, disabled IPMI as suggested above in the kernel and the fans remained spun up.

I disabled ACPI in grub (acpi=off) and the fans didn't spin up but then dom0 failed to fully load so that isn't great lol

Is it just a matter of the ACPI kernel driver being outdated? I'm not sure how to check that

ThierryEscande

@rmaclachlan If it's an ACPI issue and since Lenovo doesn't seem to be very cooperative, one could try to downgrade the firmware to a working version (i.e. one that runs fans at normal speed) and dump the ACPI table. Then upgrade to the latest firmware, dump the ACPI tables again, and then compare them.

I don't know ACPI much but I can have a look if you can share them.

The ACPI tools should be already installed on any XCP-ng host.

Dump the ACPI tables in binary format
Do so in an empty folder as this produces numerous files

# acpidump -b

Decompile the dsdt.dat file

# iasl -e ssdt*.dat -d dsdt.dat

Do the same operations for both firmwares and share the dsdt.dsl files.
The files are pretty big so don't hesitate to compress them before sharing.

rmaclachlan

Thanks having a look! I've dumped the ACPI before and after firmware updates and linked them below for you.

New FW

Old FW

ThierryEscande

@rmaclachlan Thanks for the files. Did not see anything obvious at first sight.

I forgot to ask you for ssdt files too. Would it be possible to do the same with these files ?

iasl -d ssdt*.dat

(I hope you kept the old firmware ones somewhere, otherwise don't bother to downgrade again. Just share the new firmware ssdt files)

rmaclachlan

@ThierryEscande I kept all the files from the acpidump from both new and old fw. I've ran that on both sets of acpi dumps which produced quite a few dsl files (one per ssdt) so I've just zipped both folders for you here:

ACPI-Lenovo.zip

ThierryEscande

@rmaclachlan Thanks a lot. Unfortunately I did not find any evidence of what could be wrong from the ACPI tables.

It obviously does not come from the IPMI devices as there is no modification in this area.

So without help from Lenovo it will be difficult for us to go further. If you manage to get Lenovo involved one way or another we will be happy to collaborate and help.

LennertvdBerg

@ThierryEscande Has anyone made any progress on this? @Riven you got contact details at Lenovo for contacting regarding this?

ThierryEscande

@LennertvdBerg no update on that, sorry. As said before, it will be hard to tell what's going on without feedback from Lenovo.

RIX_IT

@LennertvdBerg
We already tried getting into contact with Lenovo a while ago. But like I already stated, they weren't able to escalate the ticket because of the unsupported OS. That's the same response that @Riven got.

Maybe you could drop Lenovo a ticket as well and point them to this thread. Let's see if it helps if more people report this issue. Otherwise we seem to be pretty much out of Luck.
We have one of our two servers now in production running the old UEFI, Sound Level is not great, but bearable.
Still definitely far from an optimal solution.

LennertvdBerg

@RIX_IT I've dropped today a ticket as well, hoping them to realise it would be beneficial for all parties if they could help solving this.

LennertvdBerg

@LennertvdBerg Lenovo didn't want to provide any support. However, they just published a new UEFI/BIOS. Not sure if this is going to fix things;
Screenshot 2024-07-07 at 23.57.02.png

Riven

Nothing that seems relevant in the changelog.

==================================================
Version 4.20, Build ID KAE120J [Critical]
Release date: [Jun/ 2024]
Release Ref: FW Agile 24-2 [AMD 2S V3][AMD 1S V3]    
==================================================

Support Systems: 
  Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H
  Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F
  Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B
  Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D

This UEFI supports AMD processor:
 [SR665v3][SR645v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo","Genoa-X"), AGESA GenoaPI-SP5_1.0.0.C
 [SR655v3][SR635v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo"), AGESA GenoaPI-SP5_1.0.0.C
 
1.0 Prerequisites and dependencies
 Supported "Continuous" option for boot order through the IPMI command or redfish interface -- this feature also needs the related support at XCC side through version KAX334O-3.20.
 Prerequisite build is XCC Version KAX322Z-2.31.

2.0 Fixes
  [SR655v3] Fixed the issue that the PCIe slot 7 of ThinkSystem SR655 x8/x8 PCIe Riser3 could not detect PCIe device rightly.
  [SR665v3][SR645v3][SR655v3][SR635v3] Added more MMIO reservation padding to support ThinkSystem 2.5" U.3 PM1743 NVME SSD hot add.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that NVMe SSD (connects to backplane) speed was degraded to Gen4.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that assert tag of SMBIOS type 3 shows incorrectly in ESXi 7.0 U3n and later version.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that power-on password could not be entered rightly when SOL was activate via IPMI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Removed the option of "Disable" for "System Settings > Processors > 3D V-Cache" in System Setup Utility and Processors.3DV-Cache to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "System Settings > Processors > MONITOR/MWAIT" is not grey out when "System Settings > Processors > Global C-state Control" is "Enabled".
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would hang if executing "Diagnostics > Memory Test > Advanced Memory Test" on LXPM when ThinkSystem 2.5 U.3 6500 ION 30.72TB Read Intensive NVMe PCIe 4.0 x4 HS SSD is attached. (TT2149)
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would not load default when the number of consecutive failed POST attempts reaches the POST Attempts Limit.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "Processors > Number of Enabled CPU Cores Per Socket" can not configured the value smaller than 16.
  [SR665v3][SR645v3][SR655v3][SR635v3] Moved NVMe devices information from "System Settings > Storage" to "System Settings > NVMe" in System Setup Utility.

3.0 Enhancements
  [SR665v3][SR645v3][SR655v3][SR635v3] Integrated AMD GenoaPI version 1.0.0.C
  - Updated uCode to
  Genoa B1: 0x0A101148
  Genoa B2: 0x0A101248
  Bergamo A1: 0x0AA00116
  Bergamo A2: 0x0AA00215
  - Address CVE-2024-21978 (Medium), CVE-2024-21980 (High), CVE-2023-31355(High), CVE-2023-31315(High) 
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Security > Trusted Platform Module >TPM 2.0 > Hide TPM from OS" in System Setup Utility and TrustedComputingGroup.HideTPMfromOS to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Periodic Directory Rinse" in System Setup Utility and Processors.PeriodicDirectoryRinse to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Probe Filter Organization" in System Setup Utility and Processors.ProbeFilterOrganization to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > P-State" in System Setup Utility and Processors.P-State to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed the default value of "System Settings > Devices and I/O Ports > PCIe Ten Bit Tag" in System Setup Utility and DevicesandIOPorts.PCIeTenBitTagSupport in OneCLI from  "Disabled" to "Enabled" 
  [SR665v3][SR645v3][SR655v3][SR635v3] Supported "Continuous" option for boot order through the IPMI command or redfish interface.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed minimum memory frequency from 3200 Mhz to 3600 Mhz.

RIX_IT

@LennertvdBerg I just tested the new UEFI combined with the new XCC Firmware. Also installed all pending XCP-ng updates while being at it.
XCC Firmware didn't make any difference, so this one should be updated as Lenovo addressed quite a few CVE's.

However, the new UEFI is still not the solution to our problem, just as @Riven already figured by the changelogs. Fans are still sitting at around 12-13k RPMs. Also the DIMM Temps are still not shown in XCC.

DeOccultist

@RIX_IT

I just noticed this issue today when I installed XCP-ng 8.2.1 on a ThinkSystem SR645 v3 with UEFI KAE120J-4.20.

The fans are spinning at high 16,800 to 20,224 RPM the high pitch sound is unbearably annoying.

I changed operating mode in the UEFI settings from maximum performance to maximum efficiency. It made no difference fans are still spinning as fast as before.

rfx77

Maybe someone could try XenServer-8 which might be supported by Lenovo and reopen the ticket again. https://hcl.xenserver.com/servers/?serversupport__version=21&vendor=4&form_factor=2

DeOccultist

@rfx77
That's what I thought of too and tried booting into the Xenserver 8 iso XenServer8_2024-06-03.iso.
Even booting with the ISO installer will cause the fans to spin up abnormally fast, up to 20,000RPM.

rfx77

@DeOccultist i expected this but my thought is that now you can open a ticket at lenovo because maybe xenserver (not xcp-ng) is on the list of supported os and they will help you.