High Fan Speed Issue on Lenovo ThinkSystem Servers

rmaclachlan

@ThierryEscande I kept all the files from the acpidump from both new and old fw. I've ran that on both sets of acpi dumps which produced quite a few dsl files (one per ssdt) so I've just zipped both folders for you here:

ACPI-Lenovo.zip

ThierryEscande

@rmaclachlan Thanks a lot. Unfortunately I did not find any evidence of what could be wrong from the ACPI tables.

It obviously does not come from the IPMI devices as there is no modification in this area.

So without help from Lenovo it will be difficult for us to go further. If you manage to get Lenovo involved one way or another we will be happy to collaborate and help.

LennertvdBerg

@ThierryEscande Has anyone made any progress on this? @Riven you got contact details at Lenovo for contacting regarding this?

ThierryEscande

@LennertvdBerg no update on that, sorry. As said before, it will be hard to tell what's going on without feedback from Lenovo.

RIX_IT

@LennertvdBerg
We already tried getting into contact with Lenovo a while ago. But like I already stated, they weren't able to escalate the ticket because of the unsupported OS. That's the same response that @Riven got.

Maybe you could drop Lenovo a ticket as well and point them to this thread. Let's see if it helps if more people report this issue. Otherwise we seem to be pretty much out of Luck.
We have one of our two servers now in production running the old UEFI, Sound Level is not great, but bearable.
Still definitely far from an optimal solution.

LennertvdBerg

@RIX_IT I've dropped today a ticket as well, hoping them to realise it would be beneficial for all parties if they could help solving this.

LennertvdBerg

@LennertvdBerg Lenovo didn't want to provide any support. However, they just published a new UEFI/BIOS. Not sure if this is going to fix things;
Screenshot 2024-07-07 at 23.57.02.png

Riven

Nothing that seems relevant in the changelog.

==================================================
Version 4.20, Build ID KAE120J [Critical]
Release date: [Jun/ 2024]
Release Ref: FW Agile 24-2 [AMD 2S V3][AMD 1S V3]    
==================================================

Support Systems: 
  Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H
  Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F
  Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B
  Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D

This UEFI supports AMD processor:
 [SR665v3][SR645v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo","Genoa-X"), AGESA GenoaPI-SP5_1.0.0.C
 [SR655v3][SR635v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo"), AGESA GenoaPI-SP5_1.0.0.C
 
1.0 Prerequisites and dependencies
 Supported "Continuous" option for boot order through the IPMI command or redfish interface -- this feature also needs the related support at XCC side through version KAX334O-3.20.
 Prerequisite build is XCC Version KAX322Z-2.31.

2.0 Fixes
  [SR655v3] Fixed the issue that the PCIe slot 7 of ThinkSystem SR655 x8/x8 PCIe Riser3 could not detect PCIe device rightly.
  [SR665v3][SR645v3][SR655v3][SR635v3] Added more MMIO reservation padding to support ThinkSystem 2.5" U.3 PM1743 NVME SSD hot add.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that NVMe SSD (connects to backplane) speed was degraded to Gen4.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that assert tag of SMBIOS type 3 shows incorrectly in ESXi 7.0 U3n and later version.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that power-on password could not be entered rightly when SOL was activate via IPMI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Removed the option of "Disable" for "System Settings > Processors > 3D V-Cache" in System Setup Utility and Processors.3DV-Cache to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "System Settings > Processors > MONITOR/MWAIT" is not grey out when "System Settings > Processors > Global C-state Control" is "Enabled".
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would hang if executing "Diagnostics > Memory Test > Advanced Memory Test" on LXPM when ThinkSystem 2.5 U.3 6500 ION 30.72TB Read Intensive NVMe PCIe 4.0 x4 HS SSD is attached. (TT2149)
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would not load default when the number of consecutive failed POST attempts reaches the POST Attempts Limit.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "Processors > Number of Enabled CPU Cores Per Socket" can not configured the value smaller than 16.
  [SR665v3][SR645v3][SR655v3][SR635v3] Moved NVMe devices information from "System Settings > Storage" to "System Settings > NVMe" in System Setup Utility.

3.0 Enhancements
  [SR665v3][SR645v3][SR655v3][SR635v3] Integrated AMD GenoaPI version 1.0.0.C
  - Updated uCode to
  Genoa B1: 0x0A101148
  Genoa B2: 0x0A101248
  Bergamo A1: 0x0AA00116
  Bergamo A2: 0x0AA00215
  - Address CVE-2024-21978 (Medium), CVE-2024-21980 (High), CVE-2023-31355(High), CVE-2023-31315(High) 
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Security > Trusted Platform Module >TPM 2.0 > Hide TPM from OS" in System Setup Utility and TrustedComputingGroup.HideTPMfromOS to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Periodic Directory Rinse" in System Setup Utility and Processors.PeriodicDirectoryRinse to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Probe Filter Organization" in System Setup Utility and Processors.ProbeFilterOrganization to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > P-State" in System Setup Utility and Processors.P-State to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed the default value of "System Settings > Devices and I/O Ports > PCIe Ten Bit Tag" in System Setup Utility and DevicesandIOPorts.PCIeTenBitTagSupport in OneCLI from  "Disabled" to "Enabled" 
  [SR665v3][SR645v3][SR655v3][SR635v3] Supported "Continuous" option for boot order through the IPMI command or redfish interface.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed minimum memory frequency from 3200 Mhz to 3600 Mhz.

RIX_IT

@LennertvdBerg I just tested the new UEFI combined with the new XCC Firmware. Also installed all pending XCP-ng updates while being at it.
XCC Firmware didn't make any difference, so this one should be updated as Lenovo addressed quite a few CVE's.

However, the new UEFI is still not the solution to our problem, just as @Riven already figured by the changelogs. Fans are still sitting at around 12-13k RPMs. Also the DIMM Temps are still not shown in XCC.

DeOccultist

@RIX_IT

I just noticed this issue today when I installed XCP-ng 8.2.1 on a ThinkSystem SR645 v3 with UEFI KAE120J-4.20.

The fans are spinning at high 16,800 to 20,224 RPM the high pitch sound is unbearably annoying.

I changed operating mode in the UEFI settings from maximum performance to maximum efficiency. It made no difference fans are still spinning as fast as before.

rfx77

Maybe someone could try XenServer-8 which might be supported by Lenovo and reopen the ticket again. https://hcl.xenserver.com/servers/?serversupport__version=21&vendor=4&form_factor=2

DeOccultist

@rfx77
That's what I thought of too and tried booting into the Xenserver 8 iso XenServer8_2024-06-03.iso.
Even booting with the ISO installer will cause the fans to spin up abnormally fast, up to 20,000RPM.

rfx77

@DeOccultist i expected this but my thought is that now you can open a ticket at lenovo because maybe xenserver (not xcp-ng) is on the list of supported os and they will help you.

DeOccultist

@rfx77
I opened a support ticket with Lenovo but still no luck since XenServer 8 is a Vendor Certified OS and not a Lenovo Supported OS.

Vendor Certified: The server has met the OS Partner Certified/Logo program requirements, however, hardware device driver updates may not be available beyond the inbox OS device drivers. As a result support is limited and customers should strongly consider this when determining which OS version to install. The OS vendor provides OS support with inbox drivers only. Lenovo only provides hardware level support, and will work with hardware partners and OS vendors in a best-effort capacity to close any compatibility issues customers may experience with their installation. Lenovo will work with the OS provider to submit hardware compatibility (HCL) test suite results so that the supported solution can be posted to the respective OS providers HCL site.

The SR645 v3 has different UEFI firmware versions to the SR635 v3, it looks like v1.25 for SR645 v3 is equivilent to v1.43 for SR635 v3

==================================================
Version 1.25, Build ID KAE106Y [Critical]
Release date: [May/ 2023]
Release Ref: Genoa Wave2 EAR602 [AMD 2S V3]
==================================================

Support Systems: 
  Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D
  Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B
  
This UEFI supports AMD processor:
  AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.3

1.0 Prerequisites and dependencies
  None
 
2.0 Fixes
  Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page.
  Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior.
 
3.0 Enhancements
 None

4.0 Limitations 
  None


==================================================
Version 1.43, Build ID KAE110O [Critical]
Release date: [May/ 2023]
Release Ref: Genoa Wave2 EAR602 [AMD 1S V3]
==================================================

Support Systems: 
  Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H
  Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F

This UEFI supports AMD processor:
    AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.4
  
1.0 Prerequisites and dependencies
  None

2.0 Fixes
   Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page.
   Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior.
  
3.0 Enhancements
   None
	
4.0 Limitations 
  None

What I want to know though is where to download the older version of the UEFI firmware? I can only see the latest version on the Lenovo website.

RIX_IT

@DeOccultist
That's pretty much what I was expecting from Lenovo at this point. Sad, but thanks a lot for trying!

You can find the older versions here:

Scroll all the way to the bottom to find the Button that says "Show Previous Versions":

olivierlambert

Don't forget to save them somewhere, we'd be happy to store them on our side and have a dedicated section to download it if needed.

DeOccultist

@RIX_IT Thank you for the tips.

Just for the record, I downgraded the SR645 v3 to UEFI v1.25 KAE106Y and the fans are now spinning at 9000RPM and the fan speed adjusts accordingly depending on the CPU load so I think this is the correct behaviour.

But once upgrade to version UEFI 2.10 KAE112N for the SR645 v3, the fan spins up to 20000RPM with unbearable sound. This also occurs with the latest firmware 4.20 KAE120J.

I opened a ticket with XenServer 8 Trial Edition bugs portal and they finally replied with "This issue is on our radar. We're currently developing a solution. To avoid duplicates, we'll archive this ticket for now"
Not sure when or how it will be fixed though.

RIX_IT

@DeOccultist

Thanks again for your effort!
That gives at least a little hope for it to be fixed in the future.

Perhaps @olivierlambert could give us some insides on whether the changes can or will be implemented if they're fixed from XenServer's side?

I did also find this:
XenServer 8 cannot acquire the DIMM temperature and fan speed is too fast on the AMD platform - Lenovo ThinkSystem

Looks like Lenovo finally acknowledged that there is a problem. They even provided a Workaround for this.

I'll see if I can test it. It might take some time because we already have our servers running in production, we just couldn't wait any longer for a solution.

bleader

Olivier is currently on holidays, so I'll try to answer: it will mostly depend where the fix is implemented by XenServer, they do have some closed source parts, if that fix goes in such a part, we won't be able to get it back. If it is implemented in some of the open source part, we should be able to get it and integrate the change on our side.

Let's hope the fix ends up on the right side

Riven

@ThierryEscande @bleader @olivierlambert Hope this helps if XenServer don't come to the party.
Given the new information about the I2C-DesignWare driver - I've disabled it on one of my SR635V3 systems, and I can read the DIMM temps now

Lenovo ThinkSystem SR635 V3, BIOS KAE118M-4.11

USE AT YOUR OWN RISK - I have not tested this in production, or found out what else it might effect
However given the "workaround" provided by Lenovo for XenServer - it should be OK.

The driver is built into the kernel instead of a module, so had to be blacklisted slightly differently.

Found which driver to blacklist from kernel init.
fgrep i2c /boot/System.map-4.19.0+1 | grep init

ffffffff815172f0 T drm_i2c_encoder_init
ffffffff815574c0 T __regmap_init_i2c
ffffffff81557510 T __devm_regmap_init_i2c
ffffffff815d0700 t i2c_dw_init_master
ffffffff815d14f0 t i2c_dw_init_slave
ffffffff81ea9b40 r __ksymtab_drm_i2c_encoder_init
ffffffff81eb0570 r __ksymtab___devm_regmap_init_i2c
ffffffff81eb0838 r __ksymtab___regmap_init_i2c
ffffffff81edc1a9 r __kstrtab_drm_i2c_encoder_init
ffffffff81edffcd r __kstrtab___devm_regmap_init_i2c
ffffffff81edffe4 r __kstrtab___regmap_init_i2c
ffffffff8248156d t i2c_init
ffffffff82481b65 t dw_i2c_init_driver
ffffffff8255fe48 t __initcall_i2c_init2
ffffffff8255ffa8 t __initcall_dw_i2c_init_driver4

Guessed dw_i2c_init_driver as the most likely candidate

In /etc/grub-efi.cfg I added initcall_blacklist=dw_i2c_init_driver and ran grub-mkconfig then rebooted.

menuentry 'XCP-ng' {
        search --label --set root root-ltcoyb
        multiboot2 /boot/xen.gz watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 dom0_mem=16384M,max:16384M
        module2 /boot/vmlinuz-4.19-xen root=LABEL=root-ltcoyb ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver
        module2 /boot/initrd-4.19-xen.img
}

Restarted the BMC to be sure, and checked. Previously these reported 0 degrees C for populated DIMM slots.
ipmitool sdr | grep -i temp

Ambient Temp     | 18 degrees C      | ok
Exhaust Temp     | 35 degrees C      | ok
CPU 1 Temp       | 31 degrees C      | ok
DIMM 1 Temp      | no reading        | ns
DIMM 2 Temp      | 27 degrees C      | ok
DIMM 3 Temp      | no reading        | ns
DIMM 4 Temp      | 29 degrees C      | ok
DIMM 5 Temp      | 30 degrees C      | ok
DIMM 6 Temp      | 29 degrees C      | ok
DIMM 7 Temp      | 29 degrees C      | ok
DIMM 8 Temp      | 29 degrees C      | ok
DIMM 9 Temp      | 29 degrees C      | ok
DIMM 10 Temp     | no reading        | ns
DIMM 11 Temp     | 27 degrees C      | ok
DIMM 12 Temp     | no reading        | ns

Then check the fans rpm (previously 18000 front & 15000 rear)
ipmitool sdr | grep -i fan

Fan Mismatch     | 0x00              | ok
Fan 1 Front Tach | 7168 RPM          | ok
Fan 2 Front Tach | 7168 RPM          | ok
Fan 3 Front Tach | 7168 RPM          | ok
Fan 4 Front Tach | 7168 RPM          | ok
Fan 5 Front Tach | 7168 RPM          | ok
Fan 6 Front Tach | 7168 RPM          | ok
Fan 7 Front Tach | 0 RPM             | ok
Fan 1 Rear Tach  | 6048 RPM          | ok
Fan 2 Rear Tach  | 5940 RPM          | ok
Fan 3 Rear Tach  | 6048 RPM          | ok
Fan 4 Rear Tach  | 5940 RPM          | ok
Fan 5 Rear Tach  | 5940 RPM          | ok
Fan 6 Rear Tach  | 5940 RPM          | ok
Fan 7 Rear Tach  | 0 RPM             | ok
Sys Fan Pwr      | 6 Watts           | ok