High Fan Speed Issue on Lenovo ThinkSystem Servers

LennertvdBerg

@olivierlambert would there be a way after GRUB to walk step by step through the boot and see where it goes wrong?

rmaclachlan

I had some time to test today so I upgraded the FW on a server, disabled IPMI as suggested above in the kernel and the fans remained spun up.

I disabled ACPI in grub (acpi=off) and the fans didn't spin up but then dom0 failed to fully load so that isn't great lol

Is it just a matter of the ACPI kernel driver being outdated? I'm not sure how to check that

ThierryEscande

@rmaclachlan If it's an ACPI issue and since Lenovo doesn't seem to be very cooperative, one could try to downgrade the firmware to a working version (i.e. one that runs fans at normal speed) and dump the ACPI table. Then upgrade to the latest firmware, dump the ACPI tables again, and then compare them.

I don't know ACPI much but I can have a look if you can share them.

The ACPI tools should be already installed on any XCP-ng host.

Dump the ACPI tables in binary format
Do so in an empty folder as this produces numerous files

# acpidump -b

Decompile the dsdt.dat file

# iasl -e ssdt*.dat -d dsdt.dat

Do the same operations for both firmwares and share the dsdt.dsl files.
The files are pretty big so don't hesitate to compress them before sharing.

rmaclachlan

Thanks having a look! I've dumped the ACPI before and after firmware updates and linked them below for you.

New FW

Old FW

ThierryEscande

@rmaclachlan Thanks for the files. Did not see anything obvious at first sight.

I forgot to ask you for ssdt files too. Would it be possible to do the same with these files ?

iasl -d ssdt*.dat

(I hope you kept the old firmware ones somewhere, otherwise don't bother to downgrade again. Just share the new firmware ssdt files)

rmaclachlan

@ThierryEscande I kept all the files from the acpidump from both new and old fw. I've ran that on both sets of acpi dumps which produced quite a few dsl files (one per ssdt) so I've just zipped both folders for you here:

ACPI-Lenovo.zip

ThierryEscande

@rmaclachlan Thanks a lot. Unfortunately I did not find any evidence of what could be wrong from the ACPI tables.

It obviously does not come from the IPMI devices as there is no modification in this area.

So without help from Lenovo it will be difficult for us to go further. If you manage to get Lenovo involved one way or another we will be happy to collaborate and help.

LennertvdBerg

@ThierryEscande Has anyone made any progress on this? @Riven you got contact details at Lenovo for contacting regarding this?

ThierryEscande

@LennertvdBerg no update on that, sorry. As said before, it will be hard to tell what's going on without feedback from Lenovo.

RIX_IT

@LennertvdBerg
We already tried getting into contact with Lenovo a while ago. But like I already stated, they weren't able to escalate the ticket because of the unsupported OS. That's the same response that @Riven got.

Maybe you could drop Lenovo a ticket as well and point them to this thread. Let's see if it helps if more people report this issue. Otherwise we seem to be pretty much out of Luck.
We have one of our two servers now in production running the old UEFI, Sound Level is not great, but bearable.
Still definitely far from an optimal solution.

LennertvdBerg

@RIX_IT I've dropped today a ticket as well, hoping them to realise it would be beneficial for all parties if they could help solving this.

LennertvdBerg

@LennertvdBerg Lenovo didn't want to provide any support. However, they just published a new UEFI/BIOS. Not sure if this is going to fix things;
Screenshot 2024-07-07 at 23.57.02.png

Riven

Nothing that seems relevant in the changelog.

==================================================
Version 4.20, Build ID KAE120J [Critical]
Release date: [Jun/ 2024]
Release Ref: FW Agile 24-2 [AMD 2S V3][AMD 1S V3]    
==================================================

Support Systems: 
  Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H
  Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F
  Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B
  Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D

This UEFI supports AMD processor:
 [SR665v3][SR645v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo","Genoa-X"), AGESA GenoaPI-SP5_1.0.0.C
 [SR655v3][SR635v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo"), AGESA GenoaPI-SP5_1.0.0.C
 
1.0 Prerequisites and dependencies
 Supported "Continuous" option for boot order through the IPMI command or redfish interface -- this feature also needs the related support at XCC side through version KAX334O-3.20.
 Prerequisite build is XCC Version KAX322Z-2.31.

2.0 Fixes
  [SR655v3] Fixed the issue that the PCIe slot 7 of ThinkSystem SR655 x8/x8 PCIe Riser3 could not detect PCIe device rightly.
  [SR665v3][SR645v3][SR655v3][SR635v3] Added more MMIO reservation padding to support ThinkSystem 2.5" U.3 PM1743 NVME SSD hot add.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that NVMe SSD (connects to backplane) speed was degraded to Gen4.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that assert tag of SMBIOS type 3 shows incorrectly in ESXi 7.0 U3n and later version.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that power-on password could not be entered rightly when SOL was activate via IPMI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Removed the option of "Disable" for "System Settings > Processors > 3D V-Cache" in System Setup Utility and Processors.3DV-Cache to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "System Settings > Processors > MONITOR/MWAIT" is not grey out when "System Settings > Processors > Global C-state Control" is "Enabled".
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would hang if executing "Diagnostics > Memory Test > Advanced Memory Test" on LXPM when ThinkSystem 2.5 U.3 6500 ION 30.72TB Read Intensive NVMe PCIe 4.0 x4 HS SSD is attached. (TT2149)
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would not load default when the number of consecutive failed POST attempts reaches the POST Attempts Limit.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "Processors > Number of Enabled CPU Cores Per Socket" can not configured the value smaller than 16.
  [SR665v3][SR645v3][SR655v3][SR635v3] Moved NVMe devices information from "System Settings > Storage" to "System Settings > NVMe" in System Setup Utility.

3.0 Enhancements
  [SR665v3][SR645v3][SR655v3][SR635v3] Integrated AMD GenoaPI version 1.0.0.C
  - Updated uCode to
  Genoa B1: 0x0A101148
  Genoa B2: 0x0A101248
  Bergamo A1: 0x0AA00116
  Bergamo A2: 0x0AA00215
  - Address CVE-2024-21978 (Medium), CVE-2024-21980 (High), CVE-2023-31355(High), CVE-2023-31315(High) 
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Security > Trusted Platform Module >TPM 2.0 > Hide TPM from OS" in System Setup Utility and TrustedComputingGroup.HideTPMfromOS to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Periodic Directory Rinse" in System Setup Utility and Processors.PeriodicDirectoryRinse to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Probe Filter Organization" in System Setup Utility and Processors.ProbeFilterOrganization to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > P-State" in System Setup Utility and Processors.P-State to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed the default value of "System Settings > Devices and I/O Ports > PCIe Ten Bit Tag" in System Setup Utility and DevicesandIOPorts.PCIeTenBitTagSupport in OneCLI from  "Disabled" to "Enabled" 
  [SR665v3][SR645v3][SR655v3][SR635v3] Supported "Continuous" option for boot order through the IPMI command or redfish interface.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed minimum memory frequency from 3200 Mhz to 3600 Mhz.

RIX_IT

@LennertvdBerg I just tested the new UEFI combined with the new XCC Firmware. Also installed all pending XCP-ng updates while being at it.
XCC Firmware didn't make any difference, so this one should be updated as Lenovo addressed quite a few CVE's.

However, the new UEFI is still not the solution to our problem, just as @Riven already figured by the changelogs. Fans are still sitting at around 12-13k RPMs. Also the DIMM Temps are still not shown in XCC.

DeOccultist

@RIX_IT

I just noticed this issue today when I installed XCP-ng 8.2.1 on a ThinkSystem SR645 v3 with UEFI KAE120J-4.20.

The fans are spinning at high 16,800 to 20,224 RPM the high pitch sound is unbearably annoying.

I changed operating mode in the UEFI settings from maximum performance to maximum efficiency. It made no difference fans are still spinning as fast as before.

rfx77

Maybe someone could try XenServer-8 which might be supported by Lenovo and reopen the ticket again. https://hcl.xenserver.com/servers/?serversupport__version=21&vendor=4&form_factor=2

DeOccultist

@rfx77
That's what I thought of too and tried booting into the Xenserver 8 iso XenServer8_2024-06-03.iso.
Even booting with the ISO installer will cause the fans to spin up abnormally fast, up to 20,000RPM.

rfx77

@DeOccultist i expected this but my thought is that now you can open a ticket at lenovo because maybe xenserver (not xcp-ng) is on the list of supported os and they will help you.

DeOccultist

@rfx77
I opened a support ticket with Lenovo but still no luck since XenServer 8 is a Vendor Certified OS and not a Lenovo Supported OS.

Vendor Certified: The server has met the OS Partner Certified/Logo program requirements, however, hardware device driver updates may not be available beyond the inbox OS device drivers. As a result support is limited and customers should strongly consider this when determining which OS version to install. The OS vendor provides OS support with inbox drivers only. Lenovo only provides hardware level support, and will work with hardware partners and OS vendors in a best-effort capacity to close any compatibility issues customers may experience with their installation. Lenovo will work with the OS provider to submit hardware compatibility (HCL) test suite results so that the supported solution can be posted to the respective OS providers HCL site.

The SR645 v3 has different UEFI firmware versions to the SR635 v3, it looks like v1.25 for SR645 v3 is equivilent to v1.43 for SR635 v3

==================================================
Version 1.25, Build ID KAE106Y [Critical]
Release date: [May/ 2023]
Release Ref: Genoa Wave2 EAR602 [AMD 2S V3]
==================================================

Support Systems: 
  Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D
  Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B
  
This UEFI supports AMD processor:
  AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.3

1.0 Prerequisites and dependencies
  None
 
2.0 Fixes
  Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page.
  Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior.
 
3.0 Enhancements
 None

4.0 Limitations 
  None


==================================================
Version 1.43, Build ID KAE110O [Critical]
Release date: [May/ 2023]
Release Ref: Genoa Wave2 EAR602 [AMD 1S V3]
==================================================

Support Systems: 
  Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H
  Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F

This UEFI supports AMD processor:
    AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.4
  
1.0 Prerequisites and dependencies
  None

2.0 Fixes
   Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page.
   Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior.
  
3.0 Enhancements
   None
	
4.0 Limitations 
  None

What I want to know though is where to download the older version of the UEFI firmware? I can only see the latest version on the Lenovo website.

RIX_IT

@DeOccultist
That's pretty much what I was expecting from Lenovo at this point. Sad, but thanks a lot for trying!

You can find the older versions here:

Scroll all the way to the bottom to find the Button that says "Show Previous Versions":