XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    High Fan Speed Issue on Lenovo ThinkSystem Servers

    Scheduled Pinned Locked Moved Hardware
    101 Posts 15 Posters 43.2k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • L Offline
      LennertvdBerg @Gheppy
      last edited by

      @Gheppy I just installed Ubuntu 22.044 LTS with kernel 5.15.0-102-generic just to test if there could be anything like a 'vendor lock'. Using Ubuntu I just see my memory temperatures and all my fan speeds are around 6000 rpm. So it really seems to be something with XCP and Lenovo.

      1 Reply Last reply Reply Quote 0
      • GheppyG Offline
        Gheppy
        last edited by Gheppy

        Try to install lm-sensors and see what it detects.
        It seems to me that the kernel it does not have drivers for IPMI BMC KCS.
        My fans have 8000 - 8100 rpm

        yum -y install lm_sensors
        sensors -v
        sensors-detect
        sensors
        
        L 1 Reply Last reply Reply Quote 0
        • L Offline
          LennertvdBerg @Gheppy
          last edited by

          @Gheppy I've just reinstalled xcp-ng-8.3.0-beta2 after my Ubuntu experiment and installed lm_sensors. The output is indeed:

          Driver `to-be-written':
            * ISA bus, address 0xcc0
              Chip `IPMI BMC KCS' (confidence: 8)
          
          Note: there is no driver for IPMI BMC KCS yet.
          Check http://www.lm-sensors.org/wiki/Devices for updates.
          
          No modules to load, skipping modules configuration.
          
          Unloading i2c-dev... OK
          Unloading cpuid... OK
          

          The complete output is:
          Screenshot 2024-04-09 at 11.57.06.png
          Screenshot 2024-04-09 at 11.57.21.png

          What will be the solution for this?

          1 Reply Last reply Reply Quote 0
          • GheppyG Offline
            Gheppy
            last edited by

            We need to talk to the XCP-ng team and see if they can provide the module for the xen kernel.

            1 Reply Last reply Reply Quote 0
            • L Offline
              LennertvdBerg @olivierlambert
              last edited by

              @olivierlambert can you help us with providing the module for the xen kernel, which @Gheppy is talking about?

              R 1 Reply Last reply Reply Quote 0
              • R Offline
                rmaclachlan @LennertvdBerg
                last edited by rmaclachlan

                @LennertvdBerg The above message is indicating there is no driver written for IPMI BMC KCS for the LM_Sensors application, not that a kernel module is missing in XCPNG.

                You can read more about this on the lm_sensors github issue https://github.com/lm-sensors/lm-sensors/issues/69

                If you wish to view sensor data in XPCNG you can do so through IPMI still using ipmitool

                ipmitool sensor
                

                This will list the sensors in the server.

                DIMM 1           | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
                DIMM 1 Temp      | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 2           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 2 Temp      | 23.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 3           | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
                DIMM 3 Temp      | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 4           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 4 Temp      | 25.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 5           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 5 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 6           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 6 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 7           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 7 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 8           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 8 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 9           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 9 Temp      | 25.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 10          | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
                DIMM 10 Temp     | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
                DIMM 11          | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                DIMM 11 Temp     | 24.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                

                The issue is when upgrading the UEFI we start seeing the sensor data read NA for the RAM modules which spins up the fans on the server, I don't know how to determine what on the OS is causing that but it sounds like something is trying to read that information and is locking up the sensor.

                deric created this issue in lm-sensors/lm-sensors

                closed Driver for IPMI BMC KCS #69

                L 1 Reply Last reply Reply Quote 0
                • L Offline
                  LennertvdBerg @rmaclachlan
                  last edited by LennertvdBerg

                  @rmaclachlan Thanks. I'm also unsure how we can determine what in the OS is causing this issue. Are there other installations or modifications we could try to help isolate the problem, such as another Linux distribution with the same kernel, to see if it's a kernel-related issue? @gduperrey or @olivierlambert any suggestions how we can help the team with identifying this?

                  T 1 Reply Last reply Reply Quote 0
                  • olivierlambertO Online
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    I have no idea, sorry. Hopefully someone could tell us what we need to add in XCP-ng to fix it.

                    1 Reply Last reply Reply Quote 0
                    • T Offline
                      ThierryEscande Vates 🪐 XCP-ng Team @LennertvdBerg
                      last edited by

                      @LennertvdBerg To verify that the IPMI drivers are not messing up with the firmware, you could try to blacklist the IPMI modules.
                      Create the file /etc/modprobe.d/blacklist-ipmi.conf containing the following and reboot

                      blacklist ipmi_si
                      blacklist ipmi_devintf
                      blacklist ipmi_msghandler
                      

                      Actually add a line for whatever modules the command lsmod | grep ipmi gives you

                      L 1 Reply Last reply Reply Quote 1
                      • L Offline
                        LennertvdBerg @ThierryEscande
                        last edited by

                        @ThierryEscande . When I do lsmod | grep ipmiI get the following results

                        ipmi_si                65536  0 
                        ipmi_devintf           20480  0 
                        ipmi_msghandler        61440  2 ipmi_devintf,ipmi_si
                        

                        So, I created the file with vi /etc/modprobe.d/blacklist-ipmi.conf and added the following:

                        blacklist ipmi_si
                        blacklist ipmi_devintf
                        blacklist ipmi_msghandler
                        

                        I saved the file and rebooted the system using shutdown -r now. However, I still don't see the memory temperatures in Xclarity, and the server's fans are still running at over 13,000 RPM. The system is running XCP-NG 8.3 beta 2 with kernel 4.19.0+1.

                        1 Reply Last reply Reply Quote 0
                        • T Offline
                          ThierryEscande Vates 🪐 XCP-ng Team
                          last edited by

                          We have a new kernel-alt release available for testing purpose. This is only for XCP-ng 8.2.1 for now. The package for XCP-ng 8.3 should land soon.

                          ⚠ THIS IS A TEMPORARY REPO FOR TESTING PURPOSE ONLY ⚠

                          To install this kernel-alt package:

                          • Edit the xcp-ng.repo file
                          # vi /etc/yum.repos.d/xcp-ng.repo
                          
                          • Add the following lines at the end of the file:
                          [xcp-ng-tescande]
                          name=XCP-ng 8.2 tescande User Repository
                          baseurl=https://koji.xcp-ng.org/repos/user/8/8.2/tescande1/x86_64/
                          enabled=0
                          gpgcheck=0
                          priority=1
                          
                          • Install the kernel-alt package
                          # yum --enablerepo=xcp-ng-tescande install kernel-alt
                          
                          • Reboot and select XCP-ng kernel-alt 4.19.309 at the grub screen.
                          RIX_ITR L 2 Replies Last reply Reply Quote 0
                          • RIX_ITR Offline
                            RIX_IT @ThierryEscande
                            last edited by

                            @ThierryEscande
                            I've upgraded to the kernel-alt 4.19.309 while still using the old UEFI (kae110k 1.41). Currently the IPMI modules are not blacklisted in the modprobe config.

                            The fan speeds remain stuck at 9k RPMs, consistent with both the stable and previous alt kernel versions.

                            I also attempted to upgrade again to the newest UEFI (kae118m 4.11) but didn't notice any discernible difference in fan behavior - here it's still around 13k RPMs.
                            The ipmitool output also didn't change from the one @rmaclachlan provided.

                            If there are any other suggestions for testing or specific logs you'd like me to provide, please let me know.

                            T 1 Reply Last reply Reply Quote 0
                            • T Offline
                              ThierryEscande Vates 🪐 XCP-ng Team @RIX_IT
                              last edited by

                              @RIX_IT Thanks for your feedback.

                              Can you please share the output of dmesg and xl dmesg ?

                              RIX_ITR 1 Reply Last reply Reply Quote 0
                              • T Offline
                                ThierryEscande Vates 🪐 XCP-ng Team
                                last edited by

                                Even if this will probably be same as with XCP-ng 8.2, the kernel-alt package is available for 8.3.

                                Same steps as in my previous post, except for the repo URL in xcp-ng.repo:

                                [xcp-ng-tescande]
                                name=XCP-ng 8.3 tescande User Repository
                                baseurl=https://koji.xcp-ng.org/repos/user/8/8.3/tescande2/x86_64/
                                enabled=0
                                gpgcheck=0
                                priority=1
                                
                                1 Reply Last reply Reply Quote 0
                                • T Offline
                                  ThierryEscande Vates 🪐 XCP-ng Team
                                  last edited by

                                  And for those on XCP-ng 8.3, it might be worth giving a try to Xen 4.17. You'll find all the necessary information on how to upgrade in the thread Xen 4.17 on XCP-ng 8.3.

                                  L 1 Reply Last reply Reply Quote 0
                                  • L Offline
                                    LennertvdBerg @ThierryEscande
                                    last edited by

                                    @ThierryEscande I'm experiencing difficulties with installing the kernel-alt package on my system. Currently, I am using XCP-ng version 8.3.0-beta2. It appears that there might be a problem with updategrub.py. Any guidance on how to resolve this would be greatly appreciated.

                                    I've update xcp-ng.repo and this is my output of yum --enablerepo=xcp-ng-tescande install kernel-alt:

                                    [20:12 xcp-ng-test1 ~]# yum --enablerepo=xcp-ng-tescande install kernel-alt
                                    Loaded plugins: fastestmirror
                                    Loading mirror speeds from cached hostfile
                                    Excluding mirror: updates.xcp-ng.org
                                     * xcp-ng-base: mirrors.xcp-ng.org
                                    Excluding mirror: updates.xcp-ng.org
                                     * xcp-ng-updates: mirrors.xcp-ng.org
                                    Resolving Dependencies
                                    --> Running transaction check
                                    ---> Package kernel-alt.x86_64 0:4.19.309-1.0.lenovotest.2.xcpng8.3 will be installed
                                    --> Finished Dependency Resolution
                                    
                                    Dependencies Resolved
                                    
                                    ===========================================================================================================================================================================================================================================================================================
                                     Package                                                       Arch                                                      Version                                                                                  Repository                                                          Size
                                    ===========================================================================================================================================================================================================================================================================================
                                    Installing:
                                     kernel-alt                                                    x86_64                                                    4.19.309-1.0.lenovotest.2.xcpng8.3                                                       xcp-ng-tescande                                                     30 M
                                    
                                    Transaction Summary
                                    ===========================================================================================================================================================================================================================================================================================
                                    Install  1 Package
                                    
                                    Total download size: 30 M
                                    Installed size: 154 M
                                    Is this ok [y/d/N]: y
                                    Downloading packages:
                                    kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64.rpm                                                                                                                                                                                                            |  30 MB  00:00:01     
                                    Running transaction check
                                    Running transaction test
                                    Transaction test succeeded
                                    Running transaction
                                      Installing : kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64                                                                                                                                                                                                                    1/1 
                                    /var/tmp/rpm-tmp.l9rbzO: line 9: /opt/xensource/bin/updategrub.py: No such file or directory
                                    warning: %post(kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64) scriptlet failed, exit status 127
                                    Non-fatal POSTIN scriptlet failure in rpm package kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64
                                      Verifying  : kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64                                                                                                                                                                                                                    1/1 
                                    
                                    Installed:
                                      kernel-alt.x86_64 0:4.19.309-1.0.lenovotest.2.xcpng8.3                    
                                    
                                    1 Reply Last reply Reply Quote 0
                                    • L Offline
                                      LennertvdBerg @ThierryEscande
                                      last edited by

                                      @ThierryEscande I've updated to Xen 4.17 and it seems the upgrade went fine:

                                      host                   : xcp-ng-test1
                                      release                : 4.19.0+1
                                      version                : #1 SMP Wed Jan 24 17:19:11 CET 2024
                                      machine                : x86_64
                                      nr_cpus                : 64
                                      max_cpu_id             : 63
                                      nr_nodes               : 1
                                      cores_per_socket       : 32
                                      threads_per_core       : 2
                                      cpu_mhz                : 3245.126
                                      hw_caps                : 178bf3ff:7efa320b:2e500800:244037ff:0000000f:f1bf97a9:00405fce:00000780
                                      virt_caps              : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2
                                      total_memory           : 130850
                                      free_memory            : 121721
                                      sharing_freed_memory   : 0
                                      sharing_used_memory    : 0
                                      outstanding_claims     : 0
                                      free_cpus              : 0
                                      xen_major              : 4
                                      xen_minor              : 17
                                      xen_extra              : .3-3
                                      xen_version            : 4.17.3-3
                                      xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
                                      xen_scheduler          : credit
                                      xen_pagesize           : 4096
                                      platform_params        : virt_start=0xffff800000000000
                                      xen_changeset          : $Format:%H$, pq ???
                                      xen_commandline        : dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
                                      cc_compiler            : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
                                      cc_compile_by          : mockbuild
                                      cc_compile_domain      : [unknown]
                                      cc_compile_date        : Wed Feb 28 10:12:19 CET 2024
                                      build_id               : 9a011a28e29a21a7643376b36aec959253587d42
                                      xend_config_format     : 4
                                      

                                      However, the issues with the fan speeds and missing memory temperature readings still persist. 😕

                                      1 Reply Last reply Reply Quote 0
                                      • RIX_ITR Offline
                                        RIX_IT @ThierryEscande
                                        last edited by

                                        @ThierryEscande
                                        Of course. See the attached files:

                                        dmseg.txt
                                        xl_dmseg.txt

                                        What I also did notice was that networking isn't working at all with the 4.19.309 kernel. Settings were the same as before and I couldn't even ping the gateway. I tried an emergency reset and reconfiguring, but still no luck. Upon booting with the stable kernel, it worked again as expected. But that's an issue for another day, I guess.

                                        1 Reply Last reply Reply Quote 0
                                        • R Offline
                                          Riven
                                          last edited by Riven

                                          I have recently purchased four SR635v3 servers with AMD EPYC 9354P cpu.

                                          I managed to get some info from Lenovo about how to manually change the fan speeds.

                                          How to get the existing fan speeds. If this fails, check IPMI settings in the BMC.

                                          [09:10 xcp01 ~]# ipmitool sdr list | grep -i fan
                                          Fan Mismatch     | 0x00              | ok
                                          Fan 1 Front Tach | 19072 RPM         | ok
                                          Fan 2 Front Tach | 19072 RPM         | ok
                                          Fan 3 Front Tach | 18944 RPM         | ok
                                          Fan 4 Front Tach | 19072 RPM         | ok
                                          Fan 5 Front Tach | 18944 RPM         | ok
                                          Fan 6 Front Tach | 19072 RPM         | ok
                                          Fan 7 Front Tach | 0 RPM             | ok
                                          Fan 1 Rear Tach  | 15768 RPM         | ok
                                          Fan 2 Rear Tach  | 15768 RPM         | ok
                                          Fan 3 Rear Tach  | 15876 RPM         | ok
                                          Fan 4 Rear Tach  | 15876 RPM         | ok
                                          Fan 5 Rear Tach  | 15768 RPM         | ok
                                          Fan 6 Rear Tach  | 15768 RPM         | ok
                                          Fan 7 Rear Tach  | 0 RPM             | ok
                                          Sys Fan Pwr      | 48 Watts          | ok
                                          

                                          To change the fans to a fixed percentage - note the decimal "35" this is the percentage of maximum fan speed to set to.

                                          ipmitool raw 0x3a 0x07 0xff 35 0x01
                                          

                                          After a short while for the fans to spin down.

                                          Fan Mismatch     | 0x00              | ok
                                          Fan 1 Front Tach | 12288 RPM         | ok
                                          Fan 2 Front Tach | 12160 RPM         | ok
                                          Fan 3 Front Tach | 12160 RPM         | ok
                                          Fan 4 Front Tach | 12288 RPM         | ok
                                          Fan 5 Front Tach | 12160 RPM         | ok
                                          Fan 6 Front Tach | 12288 RPM         | ok
                                          Fan 7 Front Tach | 0 RPM             | ok
                                          Fan 1 Rear Tach  | 10152 RPM         | ok
                                          Fan 2 Rear Tach  | 10152 RPM         | ok
                                          Fan 3 Rear Tach  | 10260 RPM         | ok
                                          Fan 4 Rear Tach  | 10152 RPM         | ok
                                          Fan 5 Rear Tach  | 10260 RPM         | ok
                                          Fan 6 Rear Tach  | 10152 RPM         | ok
                                          Fan 7 Rear Tach  | 0 RPM             | ok
                                          Sys Fan Pwr      | 15 Watts          | ok
                                          

                                          To revert to auto-fan control you either need to restart the BMC/XCC controller which isn't system impacting.

                                          Please keep in mind this was stressed to me that its NOT SUPPORTED/RECOMMENDED - and could cause thermal damage to internal components!!

                                          We only have the M.2 mirror kit with two small disks, no other raid cards or internal disks.
                                          If you have RAID and internal disks you might want to fix your fans to a higher speed.

                                          Someone with scripting knowledge could also monitor a few temps and adjust the IPMI fan control accordingly.
                                          This could even be managed externally via IPMI over LAN for multiple servers.

                                          R 1 Reply Last reply Reply Quote 2
                                          • R Offline
                                            Riven @Riven
                                            last edited by

                                            Forgot to mention - I pointed Lenovo at this thread, but got the usual "OS isn't supported, find an alternative"

                                            1 Reply Last reply Reply Quote 2
                                            • First post
                                              Last post