High Fan Speed Issue on Lenovo ThinkSystem Servers

ThierryEscande

Issue quick summary

Symptoms

On some Lenovo servers, DIMM temperature reading reports errors which results in higher fan speed.

Known impacted systems

ThinkSystem SR635 V3
ThinkSystem SR655 V3

Possible solutions (but not possible/recommended)

Enable `CONFIG_X86_AMD_PLATFORM_DEVICE` in kernel configuration

By enabling this driver, the hardware device is correctly initialized and this solution fixed the issue. Unfortunately, this is not a viable solution.

Main kernel package

Ideally, enabling CONFIG_X86_AMD_PLATFORM_DEVICE in XCP-ng main kernel would solve the issue but this is actually not possible as this also enables CONFIG_PINCTRL. Enabling CONFIG_PINCTRL modifies the device internal kernel structure, resulting in kernel ABI changes. Such changes in the kernel ABI would break all external driver packages.

Alternate kernel package

CONFIG_X86_AMD_PLATFORM_DEVICE is now enable in the kernel-alt package for XCP-ng 8.2 and 8.3 since we don't really care about ABI stability for this kernel.

But this kernel is not meant to be used in production as it is less tested and no external driver package can be used with it. Therefore, even if it fixes the issue, using the kernel-alt package is highly discouraged.

Actual workaround

Blacklist `i2c_designware_platform` driver

From my understanding, without CONFIG_X86_AMD_PLATFORM_DEVICE enabled in the kernel, some hardware initializations are missing resulting in incorrect values reported by the i2c_designware_platform driver. Preventing this driver from being probed by the kernel seems to solve the issue.

Since this driver is built-in in the XCP-ng kernel and not compiled as a module, the blacklist method (shared by @Riven in this post) is as follow:

Edit /etc/grub-efi.cfg and add initcall_blacklist=dw_i2c_init_driver to the line module2 /boot/vmlinuz-4.19-xen in the XCP-ng menu entry.

...
menuentry 'XCP-ng' {
        search --label --set root root-pxdcvt
        multiboot2 /boot/xen.gz dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
        module2 /boot/vmlinuz-4.19-xen root=LABEL=root-pxdcvt ro nolvm hpet=disable rd.auto console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver
        module2 /boot/initrd-4.19-xen.img
}
...

Run grub-mkconfig and reboot.

Lenovo support

Lenovo is aware and is discussing with Citrix about this issue. As a reference, the Lenovo support entry can be found here.

RIX_IT

@ThierryEscande
Can confirm.

Applied the workaround to both Servers and upgraded to the latest XCC (KAX338N) & UEFI (KAE122D).
Temps are displayed correctly and the Servers are whisper quiet.

A huge thanks to anyone involved!

rmaclachlan

XenServer has a fix in their latest release for this - https://docs.xenserver.com/en-us/xenserver/8/whats-new/normal

These updates fix the following issues:

    The fan on Lenovo AMD systems is always at full speed and DIMM temperatures are not reported correctly by the BMC.

Does this mean the same fix can be implemented into xcpng 8.3?

olivierlambert

Yes, we'll evaluate how they fixed it without changing the whole ABI, which will cause to rebuild all the drivers.

vichutcheson

@olivierlambert

I found this forum while doing some research on the Lenovo fan speed issue, we have a few hundred MX630 & MX650 V3 Nodes.

Not if this applies for your situation - but you can use/upgrade the BMC to version 4.5.1 XCC Firmware and it should resolve your issue. We also found that upgrading to the "best recipe" for these nodes also fixed the issue. The latest "best recipe" has XCC version 5.1 and it also resolves the issue.

I've included the link to the download - you can upload it through the BMC portal and then update the XCC firmware.

https://datacentersupport.lenovo.com/us/en/downloads/DS569380

Hope this helps! FYI - we didn't have to adjust the UEFI firmware version.

-Vic

stormi

@vichutcheson We'll publish a fix anyway.

hrv3e

@stormi when will this be out? thank you

@stormi said in High Fan Speed Issue on Lenovo ThinkSystem Servers:

@vichutcheson We'll publish a fix anyway.

gduperrey

@hrv3e We currently have a kernel version, including a patch for this, waiting for testing. As soon as we have validated the rpm within our test environment, we will be able to make it available for validation through the xcp-ng-testing repo.

DeOccultist

@gduperrey Is this alternate kernel fix for the fans already released with the November patches for XCP-ng 8.2?
https://xcp-ng.org/blog/2024/11/15/november-2024-security-and-maintenance-update-for-xcp-ng-8-2-lts/

🔧 Update for alternate kernel

Backport of a fix to correct cooling fan rotation speed on some Lenovo servers.

gduperrey

@DeOccultist Yes kernel-alt already includes this patch, but if you are not using it already, we do not recommend doing so as it will possibly have other impacts on your infrastructure by using it.

Our tests were completed overnight. We should be able to offer you the "normal" kernel, integrating this patch, available for validation tests in the xcp-ng-testing repo in the coming hours.

gduperrey

The kernel RPM integrating the patch is now available on the xcp-ng-testing repo. So you can already upgrade and use it. To do this, here are the commands:

yum clean metadata --enablerepo=xcp-ng-testing
yum update kernel --enablerepo=xcp-ng-testing
reboot

It will be released to all our users in an upcoming security update or update train. We don't have a date yet.

olivierlambert

\o/

Andrew

@gduperrey (unrelated to this fan issue) I loaded the new standard 8.2 testing kernel on my NUC11 and it seems to boot a little faster and also no longer complains about some APIC devices.