XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    High Fan Speed Issue on Lenovo ThinkSystem Servers

    Scheduled Pinned Locked Moved Hardware
    101 Posts 15 Posters 46.3k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L Offline
      LennertvdBerg @bleader
      last edited by

      @bleader I've done exactly the same on my ThinkSystem SR665 V3, BMC Version
      3.20 (Build ID: KAX334O), UEFI Version 4.20 (Build ID: KAE120J), LXPM Version 4.12 (Build ID: GNL114G) and it worked;

      # Found which driver to blacklist from kernel init
      fgrep i2c /boot/System.map-4.19.0+1 | grep init
      [10:10 xcp-ng-host1 ~]# fgrep i2c /boot/System.map-4.19.0+1 | grep init
      ffffffff815172f0 T drm_i2c_encoder_init
      ffffffff815574c0 T __regmap_init_i2c
      ffffffff81557510 T __devm_regmap_init_i2c
      ffffffff815d0700 t i2c_dw_init_master
      ffffffff815d14f0 t i2c_dw_init_slave
      ffffffff81ea9b40 r __ksymtab_drm_i2c_encoder_init
      ffffffff81eb0570 r __ksymtab___devm_regmap_init_i2c
      ffffffff81eb0838 r __ksymtab___regmap_init_i2c
      ffffffff81edc1a9 r __kstrtab_drm_i2c_encoder_init
      ffffffff81edffcd r __kstrtab___devm_regmap_init_i2c
      ffffffff81edffe4 r __kstrtab___regmap_init_i2c
      ffffffff8248156d t i2c_init
      ffffffff82481b65 t dw_i2c_init_driver
      ffffffff8255fe48 t __initcall_i2c_init2
      ffffffff8255ffa8 t __initcall_dw_i2c_init_driver4
      

      In /etc/grub-efi.cfg I added initcall_blacklist=dw_i2c_init_driver and ran grub-mkconfig then rebooted.

      terminal_input serial console
      terminal_output serial console
      set default=0
      set timeout=5
      menuentry 'XCP-ng' {
              search --label --set root root-pxdcvt
              multiboot2 /boot/xen.gz dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
              module2 /boot/vmlinuz-4.19-xen root=LABEL=root-pxdcvt ro nolvm hpet=disable rd.auto console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver
              module2 /boot/initrd-4.19-xen.img
      }
      

      Run grub-mkconfig then rebooted

      grub-mkconfig
      

      I first checked the current system temperatures and fan speeds (configured at fix RPM);

      [10:17 xcp-ng-host1 ~]# ipmitool sdr | grep -i temp
      Ambient Temp     | 30 degrees C      | ok
      Exhaust Temp     | 34 degrees C      | ok
      CPU 1 Temp       | 39 degrees C      | ok
      CPU 2 Temp       | no reading        | ns
      DIMM 1 Temp      | no reading        | ns
      DIMM 2 Temp      | no reading        | ns
      DIMM 3 Temp      | no reading        | ns
      DIMM 4 Temp      | no reading        | ns
      DIMM 5 Temp      | no reading        | ns
      DIMM 6 Temp      | 0 degrees C       | ok
      DIMM 7 Temp      | 0 degrees C       | ok
      DIMM 8 Temp      | no reading        | ns
      DIMM 9 Temp      | no reading        | ns
      DIMM 10 Temp     | no reading        | ns
      DIMM 11 Temp     | no reading        | ns
      DIMM 12 Temp     | no reading        | ns
      DIMM 13 Temp     | no reading        | ns
      DIMM 14 Temp     | no reading        | ns
      DIMM 15 Temp     | no reading        | ns
      DIMM 16 Temp     | no reading        | ns
      DIMM 17 Temp     | no reading        | ns
      DIMM 18 Temp     | no reading        | ns
      DIMM 19 Temp     | no reading        | ns
      DIMM 20 Temp     | no reading        | ns
      DIMM 21 Temp     | no reading        | ns
      DIMM 22 Temp     | no reading        | ns
      DIMM 23 Temp     | no reading        | ns
      DIMM 24 Temp     | no reading        | ns
      PCIe 1 OverTemp  | 0x00              | ok
      PCIe 2 OverTemp  | 0x00              | ok
      PCIe 3 OverTemp  | 0x00              | ok
      OCP OverTemp     | 0x00              | ok
      [10:18 xcp-ng-host1 ~]# ipmitool sdr | grep -i fan
      Fan Mismatch     | 0x00              | ok
      Fan 1 Front Tach | 6642 RPM          | ok
      Fan 2 Front Tach | 6642 RPM          | ok
      Fan 3 Front Tach | 6724 RPM          | ok
      Fan 4 Front Tach | 6560 RPM          | ok
      Fan 5 Front Tach | 6642 RPM          | ok
      Fan 6 Tach       | 0 RPM             | ok
      Fan 1 Rear Tach  | 6225 RPM          | ok
      Fan 2 Rear Tach  | 6150 RPM          | ok
      Fan 3 Rear Tach  | 6300 RPM          | ok
      Fan 4 Rear Tach  | 6150 RPM          | ok
      Fan 5 Rear Tach  | 6300 RPM          | ok
      Sys Fan Pwr      | 18 Watts          | ok
      

      I followed by a reboot of the Xclarity BMC controller and the new readings are;

      [10:23 xcp-ng-host1 ~]# ipmitool sdr | grep -i temp
      Ambient Temp     | 30 degrees C      | ok
      Exhaust Temp     | 34 degrees C      | ok
      CPU 1 Temp       | 40 degrees C      | ok
      CPU 2 Temp       | no reading        | ns
      DIMM 1 Temp      | no reading        | ns
      DIMM 2 Temp      | no reading        | ns
      DIMM 3 Temp      | no reading        | ns
      DIMM 4 Temp      | no reading        | ns
      DIMM 5 Temp      | no reading        | ns
      DIMM 6 Temp      | 38 degrees C      | ok
      DIMM 7 Temp      | 37 degrees C      | ok
      DIMM 8 Temp      | no reading        | ns
      DIMM 9 Temp      | no reading        | ns
      DIMM 10 Temp     | no reading        | ns
      DIMM 11 Temp     | no reading        | ns
      DIMM 12 Temp     | no reading        | ns
      DIMM 13 Temp     | no reading        | ns
      DIMM 14 Temp     | no reading        | ns
      DIMM 15 Temp     | no reading        | ns
      DIMM 16 Temp     | no reading        | ns
      DIMM 17 Temp     | no reading        | ns
      DIMM 18 Temp     | no reading        | ns
      DIMM 19 Temp     | no reading        | ns
      DIMM 20 Temp     | no reading        | ns
      DIMM 21 Temp     | no reading        | ns
      DIMM 22 Temp     | no reading        | ns
      DIMM 23 Temp     | no reading        | ns
      DIMM 24 Temp     | no reading        | ns
      PCIe 1 OverTemp  | 0x00              | ok
      PCIe 2 OverTemp  | 0x00              | ok
      PCIe 3 OverTemp  | 0x00              | ok
      OCP OverTemp     | 0x00              | ok
      [10:26 xcp-ng-host1 ~]# ipmitool sdr | grep -i fan
      Fan Mismatch     | 0x00              | ok
      Fan 1 Front Tach | 8528 RPM          | ok
      Fan 2 Front Tach | 8446 RPM          | ok
      Fan 3 Front Tach | 8446 RPM          | ok
      Fan 4 Front Tach | 8610 RPM          | ok
      Fan 5 Front Tach | 8446 RPM          | ok
      Fan 6 Tach       | 0 RPM             | ok
      Fan 1 Rear Tach  | 7950 RPM          | ok
      Fan 2 Rear Tach  | 7950 RPM          | ok
      Fan 3 Rear Tach  | 8025 RPM          | ok
      Fan 4 Rear Tach  | 7950 RPM          | ok
      Fan 5 Rear Tach  | 7875 RPM          | ok
      Sys Fan Pwr      | 24 Watts          | ok
      
      1 Reply Last reply Reply Quote 1
      • T Offline
        ThierryEscande Vates 🪐 XCP-ng Team
        last edited by ThierryEscande

        Issue quick summary

        Symptoms

        On some Lenovo servers, DIMM temperature reading reports errors which results in higher fan speed.

        Known impacted systems

        • ThinkSystem SR635 V3
        • ThinkSystem SR655 V3

        Possible solutions (but not possible/recommended)

        Enable CONFIG_X86_AMD_PLATFORM_DEVICE in kernel configuration

        By enabling this driver, the hardware device is correctly initialized and this solution fixed the issue. Unfortunately, this is not a viable solution.

        Main kernel package

        Ideally, enabling CONFIG_X86_AMD_PLATFORM_DEVICE in XCP-ng main kernel would solve the issue but this is actually not possible as this also enables CONFIG_PINCTRL. Enabling CONFIG_PINCTRL modifies the device internal kernel structure, resulting in kernel ABI changes. Such changes in the kernel ABI would break all external driver packages.

        Alternate kernel package

        CONFIG_X86_AMD_PLATFORM_DEVICE is now enable in the kernel-alt package for XCP-ng 8.2 and 8.3 since we don't really care about ABI stability for this kernel.

        But this kernel is not meant to be used in production as it is less tested and no external driver package can be used with it. Therefore, even if it fixes the issue, using the kernel-alt package is highly discouraged.

        Actual workaround

        Blacklist i2c_designware_platform driver

        From my understanding, without CONFIG_X86_AMD_PLATFORM_DEVICE enabled in the kernel, some hardware initializations are missing resulting in incorrect values reported by the i2c_designware_platform driver. Preventing this driver from being probed by the kernel seems to solve the issue.

        Since this driver is built-in in the XCP-ng kernel and not compiled as a module, the blacklist method (shared by @Riven in this post) is as follow:

        • Edit /etc/grub-efi.cfg and add initcall_blacklist=dw_i2c_init_driver to the line module2 /boot/vmlinuz-4.19-xen in the XCP-ng menu entry.
        ...
        menuentry 'XCP-ng' {
                search --label --set root root-pxdcvt
                multiboot2 /boot/xen.gz dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
                module2 /boot/vmlinuz-4.19-xen root=LABEL=root-pxdcvt ro nolvm hpet=disable rd.auto console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver
                module2 /boot/initrd-4.19-xen.img
        }
        ...
        
        • Run grub-mkconfig and reboot.

        Lenovo support

        Lenovo is aware and is discussing with Citrix about this issue. As a reference, the Lenovo support entry can be found here.

        RIX_ITR 1 Reply Last reply Reply Quote 6
        • RIX_ITR Offline
          RIX_IT @ThierryEscande
          last edited by

          @ThierryEscande
          Can confirm.

          Applied the workaround to both Servers and upgraded to the latest XCC (KAX338N) & UEFI (KAE122D).
          Temps are displayed correctly and the Servers are whisper quiet.

          A huge thanks to anyone involved!

          1 Reply Last reply Reply Quote 2
          • R Offline
            rmaclachlan
            last edited by

            XenServer has a fix in their latest release for this - https://docs.xenserver.com/en-us/xenserver/8/whats-new/normal

            These updates fix the following issues:
            
                The fan on Lenovo AMD systems is always at full speed and DIMM temperatures are not reported correctly by the BMC.
            

            Does this mean the same fix can be implemented into xcpng 8.3?

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Yes, we'll evaluate how they fixed it without changing the whole ABI, which will cause to rebuild all the drivers.

              V 1 Reply Last reply Reply Quote 0
              • V Offline
                vichutcheson @olivierlambert
                last edited by

                @olivierlambert

                I found this forum while doing some research on the Lenovo fan speed issue, we have a few hundred MX630 & MX650 V3 Nodes.

                Not if this applies for your situation - but you can use/upgrade the BMC to version 4.5.1 XCC Firmware and it should resolve your issue. We also found that upgrading to the "best recipe" for these nodes also fixed the issue. The latest "best recipe" has XCC version 5.1 and it also resolves the issue.

                I've included the link to the download - you can upload it through the BMC portal and then update the XCC firmware.

                https://datacentersupport.lenovo.com/us/en/downloads/DS569380

                Hope this helps! FYI - we didn't have to adjust the UEFI firmware version.

                -Vic

                stormiS 1 Reply Last reply Reply Quote 2
                • stormiS Offline
                  stormi Vates 🪐 XCP-ng Team @vichutcheson
                  last edited by

                  @vichutcheson We'll publish a fix anyway.

                  H 1 Reply Last reply Reply Quote 0
                  • gduperreyG gduperrey referenced this topic on
                  • H Offline
                    hrv3e @stormi
                    last edited by

                    @stormi when will this be out? thank you

                    @stormi said in High Fan Speed Issue on Lenovo ThinkSystem Servers:

                    @vichutcheson We'll publish a fix anyway.

                    gduperreyG 1 Reply Last reply Reply Quote 0
                    • gduperreyG Offline
                      gduperrey Vates 🪐 XCP-ng Team @hrv3e
                      last edited by

                      @hrv3e We currently have a kernel version, including a patch for this, waiting for testing. As soon as we have validated the rpm within our test environment, we will be able to make it available for validation through the xcp-ng-testing repo.

                      D 1 Reply Last reply Reply Quote 0
                      • D Offline
                        DeOccultist @gduperrey
                        last edited by

                        @gduperrey Is this alternate kernel fix for the fans already released with the November patches for XCP-ng 8.2?
                        https://xcp-ng.org/blog/2024/11/15/november-2024-security-and-maintenance-update-for-xcp-ng-8-2-lts/

                        🔧 Update for alternate kernel
                        
                        Backport of a fix to correct cooling fan rotation speed on some Lenovo servers.
                        
                        gduperreyG 1 Reply Last reply Reply Quote 0
                        • gduperreyG Offline
                          gduperrey Vates 🪐 XCP-ng Team @DeOccultist
                          last edited by

                          @DeOccultist Yes kernel-alt already includes this patch, but if you are not using it already, we do not recommend doing so as it will possibly have other impacts on your infrastructure by using it.

                          Our tests were completed overnight. We should be able to offer you the "normal" kernel, integrating this patch, available for validation tests in the xcp-ng-testing repo in the coming hours.

                          1 Reply Last reply Reply Quote 1
                          • gduperreyG Offline
                            gduperrey Vates 🪐 XCP-ng Team
                            last edited by

                            The kernel RPM integrating the patch is now available on the xcp-ng-testing repo. So you can already upgrade and use it. To do this, here are the commands:

                            yum clean metadata --enablerepo=xcp-ng-testing
                            yum update kernel --enablerepo=xcp-ng-testing
                            reboot
                            

                            It will be released to all our users in an upcoming security update or update train. We don't have a date yet.

                            A 1 Reply Last reply Reply Quote 4
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              \o/

                              1 Reply Last reply Reply Quote 0
                              • A Offline
                                Andrew Top contributor @gduperrey
                                last edited by

                                @gduperrey (unrelated to this fan issue) I loaded the new standard 8.2 testing kernel on my NUC11 and it seems to boot a little faster and also no longer complains about some APIC devices.

                                1 Reply Last reply Reply Quote 0
                                • gduperreyG gduperrey referenced this topic on
                                • gduperreyG gduperrey referenced this topic on
                                • First post
                                  Last post