High Fan Speed Issue on Lenovo ThinkSystem Servers
-
@rfx77
I opened a support ticket with Lenovo but still no luck since XenServer 8 is a Vendor Certified OS and not a Lenovo Supported OS.Vendor Certified: The server has met the OS Partner Certified/Logo program requirements, however, hardware device driver updates may not be available beyond the inbox OS device drivers. As a result support is limited and customers should strongly consider this when determining which OS version to install. The OS vendor provides OS support with inbox drivers only. Lenovo only provides hardware level support, and will work with hardware partners and OS vendors in a best-effort capacity to close any compatibility issues customers may experience with their installation. Lenovo will work with the OS provider to submit hardware compatibility (HCL) test suite results so that the supported solution can be posted to the respective OS providers HCL site.The SR645 v3 has different UEFI firmware versions to the SR635 v3, it looks like v1.25 for SR645 v3 is equivilent to v1.43 for SR635 v3
================================================== Version 1.25, Build ID KAE106Y [Critical] Release date: [May/ 2023] Release Ref: Genoa Wave2 EAR602 [AMD 2S V3] ================================================== Support Systems: Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B This UEFI supports AMD processor: AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.3 1.0 Prerequisites and dependencies None 2.0 Fixes Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page. Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior. 3.0 Enhancements None 4.0 Limitations None ================================================== Version 1.43, Build ID KAE110O [Critical] Release date: [May/ 2023] Release Ref: Genoa Wave2 EAR602 [AMD 1S V3] ================================================== Support Systems: Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F This UEFI supports AMD processor: AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.4 1.0 Prerequisites and dependencies None 2.0 Fixes Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page. Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior. 3.0 Enhancements None 4.0 Limitations NoneWhat I want to know though is where to download the older version of the UEFI firmware? I can only see the latest version on the Lenovo website.
-
@DeOccultist
That's pretty much what I was expecting from Lenovo at this point. Sad, but thanks a lot for trying!You can find the older versions here:
Scroll all the way to the bottom to find the Button that says "Show Previous Versions":
-
Don't forget to save them somewhere, we'd be happy to store them on our side and have a dedicated section to download it if needed.
-
@RIX_IT Thank you for the tips.
Just for the record, I downgraded the SR645 v3 to UEFI v1.25 KAE106Y and the fans are now spinning at 9000RPM and the fan speed adjusts accordingly depending on the CPU load so I think this is the correct behaviour.
But once upgrade to version UEFI 2.10 KAE112N for the SR645 v3, the fan spins up to 20000RPM with unbearable sound. This also occurs with the latest firmware 4.20 KAE120J.
I opened a ticket with XenServer 8 Trial Edition bugs portal and they finally replied with "This issue is on our radar. We're currently developing a solution. To avoid duplicates, we'll archive this ticket for now"
Not sure when or how it will be fixed though. -
Thanks again for your effort!
That gives at least a little hope for it to be fixed in the future.Perhaps @olivierlambert could give us some insides on whether the changes can or will be implemented if they're fixed from XenServer's side?
Ā
I did also find this:
XenServer 8 cannot acquire the DIMM temperature and fan speed is too fast on the AMD platform - Lenovo ThinkSystemLooks like Lenovo finally acknowledged that there is a problem. They even provided a Workaround for this.
I'll see if I can test it. It might take some time because we already have our servers running in production, we just couldn't wait any longer for a solution.
-
Olivier is currently on holidays, so I'll try to answer: it will mostly depend where the fix is implemented by XenServer, they do have some closed source parts, if that fix goes in such a part, we won't be able to get it back. If it is implemented in some of the open source part, we should be able to get it and integrate the change on our side.
Let's hope the fix ends up on the right side
-
@ThierryEscande @bleader @olivierlambert Hope this helps if XenServer don't come to the party.
Given the new information about the I2C-DesignWare driver - I've disabled it on one of my SR635V3 systems, and I can read the DIMM temps now
Lenovo ThinkSystem SR635 V3, BIOS KAE118M-4.11
USE AT YOUR OWN RISK - I have not tested this in production, or found out what else it might effect
However given the "workaround" provided by Lenovo for XenServer - it should be OK.The driver is built into the kernel instead of a module, so had to be blacklisted slightly differently.
Found which driver to blacklist from kernel init.
fgrep i2c /boot/System.map-4.19.0+1 | grep initffffffff815172f0 T drm_i2c_encoder_init ffffffff815574c0 T __regmap_init_i2c ffffffff81557510 T __devm_regmap_init_i2c ffffffff815d0700 t i2c_dw_init_master ffffffff815d14f0 t i2c_dw_init_slave ffffffff81ea9b40 r __ksymtab_drm_i2c_encoder_init ffffffff81eb0570 r __ksymtab___devm_regmap_init_i2c ffffffff81eb0838 r __ksymtab___regmap_init_i2c ffffffff81edc1a9 r __kstrtab_drm_i2c_encoder_init ffffffff81edffcd r __kstrtab___devm_regmap_init_i2c ffffffff81edffe4 r __kstrtab___regmap_init_i2c ffffffff8248156d t i2c_init ffffffff82481b65 t dw_i2c_init_driver ffffffff8255fe48 t __initcall_i2c_init2 ffffffff8255ffa8 t __initcall_dw_i2c_init_driver4Guessed dw_i2c_init_driver as the most likely candidate
In /etc/grub-efi.cfg I added initcall_blacklist=dw_i2c_init_driver and ran grub-mkconfig then rebooted.
menuentry 'XCP-ng' { search --label --set root root-ltcoyb multiboot2 /boot/xen.gz watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 dom0_mem=16384M,max:16384M module2 /boot/vmlinuz-4.19-xen root=LABEL=root-ltcoyb ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver module2 /boot/initrd-4.19-xen.img }Restarted the BMC to be sure, and checked. Previously these reported 0 degrees C for populated DIMM slots.
ipmitool sdr | grep -i tempAmbient Temp | 18 degrees C | ok Exhaust Temp | 35 degrees C | ok CPU 1 Temp | 31 degrees C | ok DIMM 1 Temp | no reading | ns DIMM 2 Temp | 27 degrees C | ok DIMM 3 Temp | no reading | ns DIMM 4 Temp | 29 degrees C | ok DIMM 5 Temp | 30 degrees C | ok DIMM 6 Temp | 29 degrees C | ok DIMM 7 Temp | 29 degrees C | ok DIMM 8 Temp | 29 degrees C | ok DIMM 9 Temp | 29 degrees C | ok DIMM 10 Temp | no reading | ns DIMM 11 Temp | 27 degrees C | ok DIMM 12 Temp | no reading | nsThen check the fans rpm (previously 18000 front & 15000 rear)
ipmitool sdr | grep -i fanFan Mismatch | 0x00 | ok Fan 1 Front Tach | 7168 RPM | ok Fan 2 Front Tach | 7168 RPM | ok Fan 3 Front Tach | 7168 RPM | ok Fan 4 Front Tach | 7168 RPM | ok Fan 5 Front Tach | 7168 RPM | ok Fan 6 Front Tach | 7168 RPM | ok Fan 7 Front Tach | 0 RPM | ok Fan 1 Rear Tach | 6048 RPM | ok Fan 2 Rear Tach | 5940 RPM | ok Fan 3 Rear Tach | 6048 RPM | ok Fan 4 Rear Tach | 5940 RPM | ok Fan 5 Rear Tach | 5940 RPM | ok Fan 6 Rear Tach | 5940 RPM | ok Fan 7 Rear Tach | 0 RPM | ok Sys Fan Pwr | 6 Watts | ok
-
Hi,
Does it solve the fan speed issue? If yes, might worth digging more.
-
@olivierlambert said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
Hi,
Does it solve the fan speed issue? If yes, might worth digging more.
Yes absolutely - fans are operating as per normal/expectation now.
According to https://support.lenovo.com/ie/en/solutions/tt2254-xenserver-8-cannot-acquire-the-dimm-temperature-and-fan-speed-is-too-fast-on-the-amd-platform-lenovo-thinksystem this is a work around.
The main problem is:
CONFIG_X86_AMD_PLATFORM_DEVICE is a configuration setting in Linux kernel which would affect initial system HW setting. In this case, due to CONFIG_X86_AMD_PLATFORM_DEVICE is not set, the clock source of AMDI0010 (the device inside CPU) can't be registered which results in incorrect I2C command output from it and causes the DIMM temperature retrieval failure in XCC side.
The system fan speed is automatically varies based on DIMM temperature readings, and when the DIMM temperature being error, the fan speed will be tuned to high rotation. -
grep CONFIG_X86_AMD_PLATFORM_DEVICE /boot/config-4.19.0+1 # CONFIG_X86_AMD_PLATFORM_DEVICE is not setA rebuilt kernel with this enabled might be all is needed.
-
@RIX_IT
I've been informed by XenServer trial support that Lenovo has filed the ticket LEN-605 for this issue, it's probably a private issue tracker though.
They'll let me know once it is fixed. -
Thanks for the feedback everyone. Let's see what happens next
-
@Riven said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
grep CONFIG_X86_AMD_PLATFORM_DEVICE /boot/config-4.19.0+1 # CONFIG_X86_AMD_PLATFORM_DEVICE is not setA rebuilt kernel with this enabled might be all is needed.
Hi,
We're building kernel-alt packages for both XCP-ng 8.2 and 8.3 with X86_AMD_PLATFORM_DEVICE enabled. We'll notify you when they are available for testing.
-
The
kernel-altpackage withX86_AMD_PLATFORM_DEVICEenabled is available for testing.This is for XCP-ng 8.2.1 only for now.
Use yum to install it:
yum --enablerepo=xcp-ng-testing install kernel-altReboot and select
XCP-ng kernel-alt 4.19.265at the grub screen. -
@ThierryEscande said in High Fan Speed Issue on Lenovo ThinkSystem Servers:
The
kernel-altpackage withX86_AMD_PLATFORM_DEVICEenabled is available for testing.This is for XCP-ng 8.2.1 only for now.
Use yum to install it:
yum --enablerepo=xcp-ng-testing install kernel-altReboot and select
XCP-ng kernel-alt 4.19.265at the grub screen.It fixes the memory temp reading, and therefore fan controls - BUT I no longer have any networking.
Reboot to std kernel and its back just fine.
-
Kernel config differences from existing std (4.19.0+1) to testing (4.19.265)
Network interfaces we have are Intel(R) E810-DA2 10/25GbE SFP28 2-Port
# Linux/x86 4.19.0+1 Kernel Configuration | # Linux/x86 4.19.265 Kernel Configuration > CONFIG_CC_HAS_ASM_GOTO=y > # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set > CONFIG_SYSTEM_DATA_VERIFICATION=y # CONFIG_X86_AMD_PLATFORM_DEVICE is not set | CONFIG_X86_AMD_PLATFORM_DEVICE=y > CONFIG_X86_INTEL_TSX_MODE_OFF=y > # CONFIG_X86_INTEL_TSX_MODE_ON is not set > # CONFIG_X86_INTEL_TSX_MODE_AUTO is not set # CONFIG_DELL_RBU is not set | CONFIG_DELL_RBU=m > CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y > CONFIG_ARCH_USE_MEMREMAP_PROT=y CONFIG_ASN1=m | CONFIG_ASN1=y > CONFIG_UNIX_SCM=y # CONFIG_WIRELESS is not set | CONFIG_WIRELESS=y > CONFIG_CFG80211=m > # CONFIG_NL80211_TESTMODE is not set > # CONFIG_CFG80211_DEVELOPER_WARNINGS is not set > # CONFIG_CFG80211_CERTIFICATION_ONUS is not set > CONFIG_CFG80211_REQUIRE_SIGNED_REGDB=y > CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y > CONFIG_CFG80211_DEFAULT_PS=y > # CONFIG_CFG80211_DEBUGFS is not set > CONFIG_CFG80211_CRDA_SUPPORT=y > # CONFIG_CFG80211_WEXT is not set > # CONFIG_MAC80211 is not set > CONFIG_MAC80211_STA_HASH_MAX_SIZE=0 # CONFIG_FW_LOADER_USER_HELPER is not set | CONFIG_FW_LOADER_USER_HELPER=y > # CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set # CONFIG_BLK_DEV_SR_VENDOR is not set < # CONFIG_ASIX_PHY is not set | # CONFIG_AX88796B_PHY is not set # CONFIG_WLAN is not set | CONFIG_WLAN=y > # CONFIG_WIRELESS_WDS is not set > CONFIG_WLAN_VENDOR_ADMTEK=y > CONFIG_WLAN_VENDOR_ATH=y > # CONFIG_ATH_DEBUG is not set > # CONFIG_ATH5K_PCI is not set > # CONFIG_ATH6KL is not set > # CONFIG_WIL6210 is not set > CONFIG_WLAN_VENDOR_ATMEL=y > # CONFIG_ATMEL is not set > CONFIG_WLAN_VENDOR_BROADCOM=y > # CONFIG_BRCMFMAC is not set > CONFIG_WLAN_VENDOR_CISCO=y > # CONFIG_AIRO is not set > CONFIG_WLAN_VENDOR_INTEL=y > # CONFIG_IPW2100 is not set > # CONFIG_IPW2200 is not set > CONFIG_WLAN_VENDOR_INTERSIL=y > # CONFIG_HOSTAP is not set > # CONFIG_HERMES is not set > # CONFIG_PRISM54 is not set > CONFIG_WLAN_VENDOR_MARVELL=y > # CONFIG_LIBERTAS is not set > # CONFIG_MWIFIEX is not set > CONFIG_WLAN_VENDOR_MEDIATEK=y > CONFIG_WLAN_VENDOR_RALINK=y > CONFIG_WLAN_VENDOR_REALTEK=y > CONFIG_WLAN_VENDOR_RSI=y > CONFIG_WLAN_VENDOR_ST=y > CONFIG_WLAN_VENDOR_TI=y > CONFIG_WLAN_VENDOR_ZYDAS=y > # CONFIG_USB_ZD1201 is not set > CONFIG_WLAN_VENDOR_QUANTENNA=y > # CONFIG_QTNFMAC_PEARL_PCIE is not set > # CONFIG_USB_NET_RNDIS_WLAN is not set > # CONFIG_THUNDERBOLT_NET is not set # CONFIG_NVM is not set < > CONFIG_LDISC_AUTOLOAD=y # CONFIG_R3964 is not set < > CONFIG_RANDOM_TRUST_BOOTLOADER=y # CONFIG_PINCTRL is not set | CONFIG_PINCTRL=y > # CONFIG_DEBUG_PINCTRL is not set > # CONFIG_PINCTRL_AMD is not set > # CONFIG_PINCTRL_MCP23S08 is not set > # CONFIG_PINCTRL_SX150X is not set > # CONFIG_PINCTRL_BAYTRAIL is not set > # CONFIG_PINCTRL_CHERRYVIEW is not set > # CONFIG_PINCTRL_BROXTON is not set > # CONFIG_PINCTRL_CANNONLAKE is not set > # CONFIG_PINCTRL_CEDARFORK is not set > # CONFIG_PINCTRL_DENVERTON is not set > # CONFIG_PINCTRL_GEMINILAKE is not set > # CONFIG_PINCTRL_ICELAKE is not set > # CONFIG_PINCTRL_LEWISBURG is not set > # CONFIG_PINCTRL_SUNRISEPOINT is not set > # CONFIG_BATTERY_RT5033 is not set CONFIG_DRM_I915_GVT_XENGT=m < CONFIG_FB=y < CONFIG_FIRMWARE_EDID=y < > CONFIG_FB=y > CONFIG_FIRMWARE_EDID=y CONFIG_VGACON_SOFT_SCROLLBACK=y < CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64 < # CONFIG_VGACON_SOFT_SCROLLBACK_PERSISTENT_ENABLE_BY_DEFAULT < # CONFIG_USB_RIO500 is not set < # CONFIG_THUNDERBOLT is not set | CONFIG_THUNDERBOLT=m # CONFIG_NTFS_RW is not set | CONFIG_NTFS_RW=y # CONFIG_IMA_TEMPLATE is not set < CONFIG_CRYPTO_AKCIPHER=m | CONFIG_CRYPTO_AKCIPHER=y CONFIG_CRYPTO_RSA=m | CONFIG_CRYPTO_RSA=y # CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE is not set | CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y > CONFIG_X509_CERTIFICATE_PARSER=y > CONFIG_PKCS7_MESSAGE_PARSER=y > # CONFIG_PKCS7_TEST_KEY is not set > # CONFIG_SIGNED_PE_FILE_VERIFICATION is not set # CONFIG_SYSTEM_TRUSTED_KEYRING is not set | CONFIG_SYSTEM_TRUSTED_KEYRING=y > CONFIG_SYSTEM_TRUSTED_KEYS="" > # CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set > # CONFIG_SECONDARY_TRUSTED_KEYRING is not set CONFIG_OID_REGISTRY=m | CONFIG_OID_REGISTRY=y -
Looks like the "ice" driver is never initialized/loaded, which is what the NIC requires.
Also appears to be some IPMI_SI errors
-
Tested 4.19.265.1 which also has no networking for me, so its not the most recent update that killed it.
-
4.19.265-1 & -2 (havent tried earlier ones)
# dmesg | grep -i intel Intel GenuineIntel ioatdma: Intel(R) QuickData Technology Driver 4.00 # dmesg | grep eth xen_netfront: Initialising Xen virtual ethernet driver4.19.0+1
# dmesg | grep -i intel Intel GenuineIntel ioatdma: Intel(R) QuickData Technology Driver 4.00 ice: Intel(R) Ethernet Connection E800 Series Linux Driver - version 1.6.4 ice: Copyright (C) 2018-2021, Intel Corporation. # dmesg | grep eth xen_netfront: Initialising Xen virtual ethernet driver ice 0000:41:00.0 side-1974-eth0: renamed from eth0 ice 0000:41:00.1 side-1128-eth1: renamed from eth1 ice 0000:81:00.0 side-3431-eth2: renamed from eth2 ice 0000:81:00.1 side-597-eth3: renamed from eth3 ice 0000:41:00.0 eth0: renamed from side-1974-eth0 ice 0000:41:00.1 eth1: renamed from side-1128-eth1 ice 0000:81:00.0 eth2: renamed from side-3431-eth2 ice 0000:81:00.1 eth3: renamed from side-597-eth3 device eth0 entered promiscuous mode device eth2 entered promiscuous mode ice 0000:81:00.0 eth2: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth2 ice 0000:41:00.0 eth0: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth0 device eth1 entered promiscuous mode device eth3 entered promiscuous mode ice 0000:41:00.1 eth1: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth1 ice 0000:81:00.1 eth3: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None 8021q: adding VLAN 0 to HW filter on device eth3 8021q: adding VLAN 0 to HW filter on device eth2 8021q: adding VLAN 0 to HW filter on device eth2 8021q: adding VLAN 0 to HW filter on device eth3 8021q: adding VLAN 0 to HW filter on device eth3 8021q: adding VLAN 0 to HW filter on device eth1 8021q: adding VLAN 0 to HW filter on device eth1 8021q: adding VLAN 0 to HW filter on device eth0 8021q: adding VLAN 0 to HW filter on device eth0 -
Thierry is on holidays for now
It is more likely due to the fact the ice driver is either coming from the
intel-iceorintel-ice-altwhich are both for the main kernel, the -alt one being a more up to date version, and not as the confusing name could imply, for thekernel-alt.The
kernel-altis in any case a debugging and testing kernel and not meant for production, but your tests confirmed that this seems to fix the fan speed issue, now we can discuss internally about what to do.
