High Fan Speed Issue on Lenovo ThinkSystem Servers
-
Thanks for the feedback everyone. Let's see what happens next
-
@Riven said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
grep CONFIG_X86_AMD_PLATFORM_DEVICE /boot/config-4.19.0+1 # CONFIG_X86_AMD_PLATFORM_DEVICE is not setA rebuilt kernel with this enabled might be all is needed.
Hi,
We're building kernel-alt packages for both XCP-ng 8.2 and 8.3 with X86_AMD_PLATFORM_DEVICE enabled. We'll notify you when they are available for testing.
-
The
kernel-altpackage withX86_AMD_PLATFORM_DEVICEenabled is available for testing.This is for XCP-ng 8.2.1 only for now.
Use yum to install it:
yum --enablerepo=xcp-ng-testing install kernel-altReboot and select
XCP-ng kernel-alt 4.19.265at the grub screen. -
@ThierryEscande said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
The
kernel-altpackage withX86_AMD_PLATFORM_DEVICEenabled is available for testing.This is for XCP-ng 8.2.1 only for now.
Use yum to install it:
yum --enablerepo=xcp-ng-testing install kernel-altReboot and select
XCP-ng kernel-alt 4.19.265at the grub screen.It fixes the memory temp reading, and therefore fan controls - BUT I no longer have any networking.
Reboot to std kernel and its back just fine.
-
Kernel config differences from existing std (4.19.0+1) to testing (4.19.265)
Network interfaces we have are Intel(R) E810-DA2 10/25GbE SFP28 2-Port
# Linux/x86 4.19.0+1 Kernel Configuration | # Linux/x86 4.19.265 Kernel Configuration > CONFIG_CC_HAS_ASM_GOTO=y > # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set > CONFIG_SYSTEM_DATA_VERIFICATION=y # CONFIG_X86_AMD_PLATFORM_DEVICE is not set | CONFIG_X86_AMD_PLATFORM_DEVICE=y > CONFIG_X86_INTEL_TSX_MODE_OFF=y > # CONFIG_X86_INTEL_TSX_MODE_ON is not set > # CONFIG_X86_INTEL_TSX_MODE_AUTO is not set # CONFIG_DELL_RBU is not set | CONFIG_DELL_RBU=m > CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y > CONFIG_ARCH_USE_MEMREMAP_PROT=y CONFIG_ASN1=m | CONFIG_ASN1=y > CONFIG_UNIX_SCM=y # CONFIG_WIRELESS is not set | CONFIG_WIRELESS=y > CONFIG_CFG80211=m > # CONFIG_NL80211_TESTMODE is not set > # CONFIG_CFG80211_DEVELOPER_WARNINGS is not set > # CONFIG_CFG80211_CERTIFICATION_ONUS is not set > CONFIG_CFG80211_REQUIRE_SIGNED_REGDB=y > CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y > CONFIG_CFG80211_DEFAULT_PS=y > # CONFIG_CFG80211_DEBUGFS is not set > CONFIG_CFG80211_CRDA_SUPPORT=y > # CONFIG_CFG80211_WEXT is not set > # CONFIG_MAC80211 is not set > CONFIG_MAC80211_STA_HASH_MAX_SIZE=0 # CONFIG_FW_LOADER_USER_HELPER is not set | CONFIG_FW_LOADER_USER_HELPER=y > # CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set # CONFIG_BLK_DEV_SR_VENDOR is not set < # CONFIG_ASIX_PHY is not set | # CONFIG_AX88796B_PHY is not set # CONFIG_WLAN is not set | CONFIG_WLAN=y > # CONFIG_WIRELESS_WDS is not set > CONFIG_WLAN_VENDOR_ADMTEK=y > CONFIG_WLAN_VENDOR_ATH=y > # CONFIG_ATH_DEBUG is not set > # CONFIG_ATH5K_PCI is not set > # CONFIG_ATH6KL is not set > # CONFIG_WIL6210 is not set > CONFIG_WLAN_VENDOR_ATMEL=y > # CONFIG_ATMEL is not set > CONFIG_WLAN_VENDOR_BROADCOM=y > # CONFIG_BRCMFMAC is not set > CONFIG_WLAN_VENDOR_CISCO=y > # CONFIG_AIRO is not set > CONFIG_WLAN_VENDOR_INTEL=y > # CONFIG_IPW2100 is not set > # CONFIG_IPW2200 is not set > CONFIG_WLAN_VENDOR_INTERSIL=y > # CONFIG_HOSTAP is not set > # CONFIG_HERMES is not set > # CONFIG_PRISM54 is not set > CONFIG_WLAN_VENDOR_MARVELL=y > # CONFIG_LIBERTAS is not set > # CONFIG_MWIFIEX is not set > CONFIG_WLAN_VENDOR_MEDIATEK=y > CONFIG_WLAN_VENDOR_RALINK=y > CONFIG_WLAN_VENDOR_REALTEK=y > CONFIG_WLAN_VENDOR_RSI=y > CONFIG_WLAN_VENDOR_ST=y > CONFIG_WLAN_VENDOR_TI=y > CONFIG_WLAN_VENDOR_ZYDAS=y > # CONFIG_USB_ZD1201 is not set > CONFIG_WLAN_VENDOR_QUANTENNA=y > # CONFIG_QTNFMAC_PEARL_PCIE is not set > # CONFIG_USB_NET_RNDIS_WLAN is not set > # CONFIG_THUNDERBOLT_NET is not set # CONFIG_NVM is not set < > CONFIG_LDISC_AUTOLOAD=y # CONFIG_R3964 is not set < > CONFIG_RANDOM_TRUST_BOOTLOADER=y # CONFIG_PINCTRL is not set | CONFIG_PINCTRL=y > # CONFIG_DEBUG_PINCTRL is not set > # CONFIG_PINCTRL_AMD is not set > # CONFIG_PINCTRL_MCP23S08 is not set > # CONFIG_PINCTRL_SX150X is not set > # CONFIG_PINCTRL_BAYTRAIL is not set > # CONFIG_PINCTRL_CHERRYVIEW is not set > # CONFIG_PINCTRL_BROXTON is not set > # CONFIG_PINCTRL_CANNONLAKE is not set > # CONFIG_PINCTRL_CEDARFORK is not set > # CONFIG_PINCTRL_DENVERTON is not set > # CONFIG_PINCTRL_GEMINILAKE is not set > # CONFIG_PINCTRL_ICELAKE is not set > # CONFIG_PINCTRL_LEWISBURG is not set > # CONFIG_PINCTRL_SUNRISEPOINT is not set > # CONFIG_BATTERY_RT5033 is not set CONFIG_DRM_I915_GVT_XENGT=m < CONFIG_FB=y < CONFIG_FIRMWARE_EDID=y < > CONFIG_FB=y > CONFIG_FIRMWARE_EDID=y CONFIG_VGACON_SOFT_SCROLLBACK=y < CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64 < # CONFIG_VGACON_SOFT_SCROLLBACK_PERSISTENT_ENABLE_BY_DEFAULT < # CONFIG_USB_RIO500 is not set < # CONFIG_THUNDERBOLT is not set | CONFIG_THUNDERBOLT=m # CONFIG_NTFS_RW is not set | CONFIG_NTFS_RW=y # CONFIG_IMA_TEMPLATE is not set < CONFIG_CRYPTO_AKCIPHER=m | CONFIG_CRYPTO_AKCIPHER=y CONFIG_CRYPTO_RSA=m | CONFIG_CRYPTO_RSA=y # CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE is not set | CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y > CONFIG_X509_CERTIFICATE_PARSER=y > CONFIG_PKCS7_MESSAGE_PARSER=y > # CONFIG_PKCS7_TEST_KEY is not set > # CONFIG_SIGNED_PE_FILE_VERIFICATION is not set # CONFIG_SYSTEM_TRUSTED_KEYRING is not set | CONFIG_SYSTEM_TRUSTED_KEYRING=y > CONFIG_SYSTEM_TRUSTED_KEYS="" > # CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set > # CONFIG_SECONDARY_TRUSTED_KEYRING is not set CONFIG_OID_REGISTRY=m | CONFIG_OID_REGISTRY=y -
Looks like the "ice" driver is never initialized/loaded, which is what the NIC requires.
Also appears to be some IPMI_SI errors
-
Tested 4.19.265.1 which also has no networking for me, so its not the most recent update that killed it.
-
4.19.265-1 & -2 (havent tried earlier ones)
# dmesg | grep -i intel Intel GenuineIntel ioatdma: Intel(R) QuickData Technology Driver 4.00 # dmesg | grep eth xen_netfront: Initialising Xen virtual ethernet driver4.19.0+1
# dmesg | grep -i intel Intel GenuineIntel ioatdma: Intel(R) QuickData Technology Driver 4.00 ice: Intel(R) Ethernet Connection E800 Series Linux Driver - version 1.6.4 ice: Copyright (C) 2018-2021, Intel Corporation. # dmesg | grep eth xen_netfront: Initialising Xen virtual ethernet driver ice 0000:41:00.0 side-1974-eth0: renamed from eth0 ice 0000:41:00.1 side-1128-eth1: renamed from eth1 ice 0000:81:00.0 side-3431-eth2: renamed from eth2 ice 0000:81:00.1 side-597-eth3: renamed from eth3 ice 0000:41:00.0 eth0: renamed from side-1974-eth0 ice 0000:41:00.1 eth1: renamed from side-1128-eth1 ice 0000:81:00.0 eth2: renamed from side-3431-eth2 ice 0000:81:00.1 eth3: renamed from side-597-eth3 device eth0 entered promiscuous mode device eth2 entered promiscuous mode ice 0000:81:00.0 eth2: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth2 ice 0000:41:00.0 eth0: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth0 device eth1 entered promiscuous mode device eth3 entered promiscuous mode ice 0000:41:00.1 eth1: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth1 ice 0000:81:00.1 eth3: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth3 8021q: adding VLAN 0 to HW filter on device eth2 8021q: adding VLAN 0 to HW filter on device eth2 8021q: adding VLAN 0 to HW filter on device eth3 8021q: adding VLAN 0 to HW filter on device eth3 8021q: adding VLAN 0 to HW filter on device eth1 8021q: adding VLAN 0 to HW filter on device eth1 8021q: adding VLAN 0 to HW filter on device eth0 8021q: adding VLAN 0 to HW filter on device eth0 -
Thierry is on holidays for now
It is more likely due to the fact the ice driver is either coming from the
intel-iceorintel-ice-altwhich are both for the main kernel, the -alt one being a more up to date version, and not as the confusing name could imply, for thekernel-alt.The
kernel-altis in any case a debugging and testing kernel and not meant for production, but your tests confirmed that this seems to fix the fan speed issue, now we can discuss internally about what to do. -
@bleader I've done exactly the same on my ThinkSystem SR665 V3, BMC Version
3.20 (Build ID: KAX334O), UEFI Version 4.20 (Build ID: KAE120J), LXPM Version 4.12 (Build ID: GNL114G) and it worked;# Found which driver to blacklist from kernel init fgrep i2c /boot/System.map-4.19.0+1 | grep init [10:10 xcp-ng-host1 ~]# fgrep i2c /boot/System.map-4.19.0+1 | grep init ffffffff815172f0 T drm_i2c_encoder_init ffffffff815574c0 T __regmap_init_i2c ffffffff81557510 T __devm_regmap_init_i2c ffffffff815d0700 t i2c_dw_init_master ffffffff815d14f0 t i2c_dw_init_slave ffffffff81ea9b40 r __ksymtab_drm_i2c_encoder_init ffffffff81eb0570 r __ksymtab___devm_regmap_init_i2c ffffffff81eb0838 r __ksymtab___regmap_init_i2c ffffffff81edc1a9 r __kstrtab_drm_i2c_encoder_init ffffffff81edffcd r __kstrtab___devm_regmap_init_i2c ffffffff81edffe4 r __kstrtab___regmap_init_i2c ffffffff8248156d t i2c_init ffffffff82481b65 t dw_i2c_init_driver ffffffff8255fe48 t __initcall_i2c_init2 ffffffff8255ffa8 t __initcall_dw_i2c_init_driver4In /etc/grub-efi.cfg I added initcall_blacklist=dw_i2c_init_driver and ran grub-mkconfig then rebooted.
terminal_input serial console terminal_output serial console set default=0 set timeout=5 menuentry 'XCP-ng' { search --label --set root root-pxdcvt multiboot2 /boot/xen.gz dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 module2 /boot/vmlinuz-4.19-xen root=LABEL=root-pxdcvt ro nolvm hpet=disable rd.auto console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver module2 /boot/initrd-4.19-xen.img }Run grub-mkconfig then rebooted
grub-mkconfigI first checked the current system temperatures and fan speeds (configured at fix RPM);
[10:17 xcp-ng-host1 ~]# ipmitool sdr | grep -i temp Ambient Temp | 30 degrees C | ok Exhaust Temp | 34 degrees C | ok CPU 1 Temp | 39 degrees C | ok CPU 2 Temp | no reading | ns DIMM 1 Temp | no reading | ns DIMM 2 Temp | no reading | ns DIMM 3 Temp | no reading | ns DIMM 4 Temp | no reading | ns DIMM 5 Temp | no reading | ns DIMM 6 Temp | 0 degrees C | ok DIMM 7 Temp | 0 degrees C | ok DIMM 8 Temp | no reading | ns DIMM 9 Temp | no reading | ns DIMM 10 Temp | no reading | ns DIMM 11 Temp | no reading | ns DIMM 12 Temp | no reading | ns DIMM 13 Temp | no reading | ns DIMM 14 Temp | no reading | ns DIMM 15 Temp | no reading | ns DIMM 16 Temp | no reading | ns DIMM 17 Temp | no reading | ns DIMM 18 Temp | no reading | ns DIMM 19 Temp | no reading | ns DIMM 20 Temp | no reading | ns DIMM 21 Temp | no reading | ns DIMM 22 Temp | no reading | ns DIMM 23 Temp | no reading | ns DIMM 24 Temp | no reading | ns PCIe 1 OverTemp | 0x00 | ok PCIe 2 OverTemp | 0x00 | ok PCIe 3 OverTemp | 0x00 | ok OCP OverTemp | 0x00 | ok [10:18 xcp-ng-host1 ~]# ipmitool sdr | grep -i fan Fan Mismatch | 0x00 | ok Fan 1 Front Tach | 6642 RPM | ok Fan 2 Front Tach | 6642 RPM | ok Fan 3 Front Tach | 6724 RPM | ok Fan 4 Front Tach | 6560 RPM | ok Fan 5 Front Tach | 6642 RPM | ok Fan 6 Tach | 0 RPM | ok Fan 1 Rear Tach | 6225 RPM | ok Fan 2 Rear Tach | 6150 RPM | ok Fan 3 Rear Tach | 6300 RPM | ok Fan 4 Rear Tach | 6150 RPM | ok Fan 5 Rear Tach | 6300 RPM | ok Sys Fan Pwr | 18 Watts | okI followed by a reboot of the Xclarity BMC controller and the new readings are;
[10:23 xcp-ng-host1 ~]# ipmitool sdr | grep -i temp Ambient Temp | 30 degrees C | ok Exhaust Temp | 34 degrees C | ok CPU 1 Temp | 40 degrees C | ok CPU 2 Temp | no reading | ns DIMM 1 Temp | no reading | ns DIMM 2 Temp | no reading | ns DIMM 3 Temp | no reading | ns DIMM 4 Temp | no reading | ns DIMM 5 Temp | no reading | ns DIMM 6 Temp | 38 degrees C | ok DIMM 7 Temp | 37 degrees C | ok DIMM 8 Temp | no reading | ns DIMM 9 Temp | no reading | ns DIMM 10 Temp | no reading | ns DIMM 11 Temp | no reading | ns DIMM 12 Temp | no reading | ns DIMM 13 Temp | no reading | ns DIMM 14 Temp | no reading | ns DIMM 15 Temp | no reading | ns DIMM 16 Temp | no reading | ns DIMM 17 Temp | no reading | ns DIMM 18 Temp | no reading | ns DIMM 19 Temp | no reading | ns DIMM 20 Temp | no reading | ns DIMM 21 Temp | no reading | ns DIMM 22 Temp | no reading | ns DIMM 23 Temp | no reading | ns DIMM 24 Temp | no reading | ns PCIe 1 OverTemp | 0x00 | ok PCIe 2 OverTemp | 0x00 | ok PCIe 3 OverTemp | 0x00 | ok OCP OverTemp | 0x00 | ok [10:26 xcp-ng-host1 ~]# ipmitool sdr | grep -i fan Fan Mismatch | 0x00 | ok Fan 1 Front Tach | 8528 RPM | ok Fan 2 Front Tach | 8446 RPM | ok Fan 3 Front Tach | 8446 RPM | ok Fan 4 Front Tach | 8610 RPM | ok Fan 5 Front Tach | 8446 RPM | ok Fan 6 Tach | 0 RPM | ok Fan 1 Rear Tach | 7950 RPM | ok Fan 2 Rear Tach | 7950 RPM | ok Fan 3 Rear Tach | 8025 RPM | ok Fan 4 Rear Tach | 7950 RPM | ok Fan 5 Rear Tach | 7875 RPM | ok Sys Fan Pwr | 24 Watts | ok -
Issue quick summary
Symptoms
On some Lenovo servers, DIMM temperature reading reports errors which results in higher fan speed.
Known impacted systems
- ThinkSystem SR635 V3
- ThinkSystem SR655 V3
Possible solutions (but not possible/recommended)
Enable
CONFIG_X86_AMD_PLATFORM_DEVICEin kernel configurationBy enabling this driver, the hardware device is correctly initialized and this solution fixed the issue. Unfortunately, this is not a viable solution.
Main kernel package
Ideally, enabling
CONFIG_X86_AMD_PLATFORM_DEVICEin XCP-ng main kernel would solve the issue but this is actually not possible as this also enablesCONFIG_PINCTRL. EnablingCONFIG_PINCTRLmodifies thedeviceinternal kernel structure, resulting in kernel ABI changes. Such changes in the kernel ABI would break all external driver packages.Alternate kernel package
CONFIG_X86_AMD_PLATFORM_DEVICEis now enable in thekernel-altpackage for XCP-ng 8.2 and 8.3 since we don't really care about ABI stability for this kernel.But this kernel is not meant to be used in production as it is less tested and no external driver package can be used with it. Therefore, even if it fixes the issue, using the
kernel-altpackage is highly discouraged.Actual workaround
Blacklist
i2c_designware_platformdriverFrom my understanding, without
CONFIG_X86_AMD_PLATFORM_DEVICEenabled in the kernel, some hardware initializations are missing resulting in incorrect values reported by thei2c_designware_platformdriver. Preventing this driver from being probed by the kernel seems to solve the issue.Since this driver is built-in in the XCP-ng kernel and not compiled as a module, the blacklist method (shared by @Riven in this post) is as follow:
- Edit
/etc/grub-efi.cfgand addinitcall_blacklist=dw_i2c_init_driverto the linemodule2 /boot/vmlinuz-4.19-xenin theXCP-ngmenu entry.
... menuentry 'XCP-ng' { search --label --set root root-pxdcvt multiboot2 /boot/xen.gz dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 module2 /boot/vmlinuz-4.19-xen root=LABEL=root-pxdcvt ro nolvm hpet=disable rd.auto console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver module2 /boot/initrd-4.19-xen.img } ...- Run
grub-mkconfigand reboot.
Lenovo support
Lenovo is aware and is discussing with Citrix about this issue. As a reference, the Lenovo support entry can be found here.
-
@ThierryEscande
Can confirm.Applied the workaround to both Servers and upgraded to the latest XCC (KAX338N) & UEFI (KAE122D).
Temps are displayed correctly and the Servers are whisper quiet.A huge thanks to anyone involved!
-
XenServer has a fix in their latest release for this - https://docs.xenserver.com/en-us/xenserver/8/whats-new/normal
These updates fix the following issues: The fan on Lenovo AMD systems is always at full speed and DIMM temperatures are not reported correctly by the BMC.Does this mean the same fix can be implemented into xcpng 8.3?
-
Yes, we'll evaluate how they fixed it without changing the whole ABI, which will cause to rebuild all the drivers.
-
I found this forum while doing some research on the Lenovo fan speed issue, we have a few hundred MX630 & MX650 V3 Nodes.
Not if this applies for your situation - but you can use/upgrade the BMC to version 4.5.1 XCC Firmware and it should resolve your issue. We also found that upgrading to the "best recipe" for these nodes also fixed the issue. The latest "best recipe" has XCC version 5.1 and it also resolves the issue.
I've included the link to the download - you can upload it through the BMC portal and then update the XCC firmware.
https://datacentersupport.lenovo.com/us/en/downloads/DS569380
Hope this helps! FYI - we didn't have to adjust the UEFI firmware version.
-Vic
-
@vichutcheson We'll publish a fix anyway.
-
G gduperrey referenced this topic
-
@stormi when will this be out? thank you
@stormi said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
@vichutcheson We'll publish a fix anyway.
-
@hrv3e We currently have a kernel version, including a patch for this, waiting for testing. As soon as we have validated the rpm within our test environment, we will be able to make it available for validation through the
xcp-ng-testingrepo. -
@gduperrey Is this alternate kernel fix for the fans already released with the November patches for XCP-ng 8.2?
https://xcp-ng.org/blog/2024/11/15/november-2024-security-and-maintenance-update-for-xcp-ng-8-2-lts/🔧 Update for alternate kernel Backport of a fix to correct cooling fan rotation speed on some Lenovo servers. -
@DeOccultist Yes
kernel-altalready includes this patch, but if you are not using it already, we do not recommend doing so as it will possibly have other impacts on your infrastructure by using it.Our tests were completed overnight. We should be able to offer you the "normal"
kernel, integrating this patch, available for validation tests in thexcp-ng-testingrepo in the coming hours.
