XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    High Fan Speed Issue on Lenovo ThinkSystem Servers

    Scheduled Pinned Locked Moved Hardware
    101 Posts 15 Posters 43.2k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • gduperreyG Offline
      gduperrey Vates 🪐 XCP-ng Team
      last edited by

      You can find information about kernel-alt here: https://docs.xcp-ng.org/installation/hardware/#-alternate-kernel

      L 1 Reply Last reply Reply Quote 0
      • RIX_ITR Offline
        RIX_IT @bleader
        last edited by

        @bleader

        Just did that on 8.2 (kernel-alt.x86_64 0:4.19.265-1.xcpng8.2), not the testing one yet.

        Got a few errors on startup, not related afaik, but still fyi:
        ab1830d1-ca13-4f4a-8805-2d78058889c3-image.png

        That didn't make a difference as it seems:
        05da9d95-b548-4a77-a6d4-2871c3ae3432-image.png

        I'm on UEFI 1.41, could try the most recent somewhat later once people leave their offices.

        R 1 Reply Last reply Reply Quote 0
        • L Offline
          LennertvdBerg @gduperrey
          last edited by LennertvdBerg

          @gduperrey said in High Fan Speed Issue on Lenovo ThinkSystem Servers:

          https://docs.xcp-ng.org/installation/hardware/#-alternate-kernel

          I tried installing the kernel-alt using

          yum install kernel-alt
          

          However I receive the following error;

          python: can't open file '/opt/xensource/bin/updategrub.py': [Errno 2] No such file or directory
          warning: %postun(kernel-alt-4.19.227-5.xcpng8.3.x86_64) scriptlet failed, exit status 2
          

          kernel-alt failure.png

          1 Reply Last reply Reply Quote 0
          • R Offline
            rmaclachlan @RIX_IT
            last edited by

            @RIX_IT I tried kernel-alt before rolling back the uefi and it didn't help. Also tried the new Xen beta stormi posted (4.19 was it?) in case something was added but that didn't fix it either.

            Thanks for trying the new Lenovo fw - I saw it come out and was going to test when I had time but now I don't need to!

            1 Reply Last reply Reply Quote 1
            • GheppyG Offline
              Gheppy
              last edited by

              Did it come fully equipped?
              Did you install Lenovo signed HDD in it? Or you have compatibles HDD.
              If you have compatibility, try to put a HDD with (original) Lenovo firmware.
              I had the same problem with an Lenovo ThinkSystem SR630, I had installed a Lenovo non-firmware HDD, all of them. I bought one with Lenovo firmware, I installed and the problem disappeared.

              R L 2 Replies Last reply Reply Quote 0
              • R Offline
                rmaclachlan @Gheppy
                last edited by

                @Gheppy Our SR635v3 are running factory installed Lenovo hardware, although I did test swapping out the Broadcom 57504 OCP NIC with Intel X710 but the fan issue persisted. Thanks for the suggestion!

                1 Reply Last reply Reply Quote 0
                • L Offline
                  LennertvdBerg @Gheppy
                  last edited by

                  @Gheppy I just installed Ubuntu 22.044 LTS with kernel 5.15.0-102-generic just to test if there could be anything like a 'vendor lock'. Using Ubuntu I just see my memory temperatures and all my fan speeds are around 6000 rpm. So it really seems to be something with XCP and Lenovo.

                  1 Reply Last reply Reply Quote 0
                  • GheppyG Offline
                    Gheppy
                    last edited by Gheppy

                    Try to install lm-sensors and see what it detects.
                    It seems to me that the kernel it does not have drivers for IPMI BMC KCS.
                    My fans have 8000 - 8100 rpm

                    yum -y install lm_sensors
                    sensors -v
                    sensors-detect
                    sensors
                    
                    L 1 Reply Last reply Reply Quote 0
                    • L Offline
                      LennertvdBerg @Gheppy
                      last edited by

                      @Gheppy I've just reinstalled xcp-ng-8.3.0-beta2 after my Ubuntu experiment and installed lm_sensors. The output is indeed:

                      Driver `to-be-written':
                        * ISA bus, address 0xcc0
                          Chip `IPMI BMC KCS' (confidence: 8)
                      
                      Note: there is no driver for IPMI BMC KCS yet.
                      Check http://www.lm-sensors.org/wiki/Devices for updates.
                      
                      No modules to load, skipping modules configuration.
                      
                      Unloading i2c-dev... OK
                      Unloading cpuid... OK
                      

                      The complete output is:
                      Screenshot 2024-04-09 at 11.57.06.png
                      Screenshot 2024-04-09 at 11.57.21.png

                      What will be the solution for this?

                      1 Reply Last reply Reply Quote 0
                      • GheppyG Offline
                        Gheppy
                        last edited by

                        We need to talk to the XCP-ng team and see if they can provide the module for the xen kernel.

                        1 Reply Last reply Reply Quote 0
                        • L Offline
                          LennertvdBerg @olivierlambert
                          last edited by

                          @olivierlambert can you help us with providing the module for the xen kernel, which @Gheppy is talking about?

                          R 1 Reply Last reply Reply Quote 0
                          • R Offline
                            rmaclachlan @LennertvdBerg
                            last edited by rmaclachlan

                            @LennertvdBerg The above message is indicating there is no driver written for IPMI BMC KCS for the LM_Sensors application, not that a kernel module is missing in XCPNG.

                            You can read more about this on the lm_sensors github issue https://github.com/lm-sensors/lm-sensors/issues/69

                            If you wish to view sensor data in XPCNG you can do so through IPMI still using ipmitool

                            ipmitool sensor
                            

                            This will list the sensors in the server.

                            DIMM 1           | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
                            DIMM 1 Temp      | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 2           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 2 Temp      | 23.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 3           | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
                            DIMM 3 Temp      | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 4           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 4 Temp      | 25.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 5           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 5 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 6           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 6 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 7           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 7 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 8           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 8 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 9           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 9 Temp      | 25.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 10          | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
                            DIMM 10 Temp     | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            DIMM 11          | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
                            DIMM 11 Temp     | 24.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
                            

                            The issue is when upgrading the UEFI we start seeing the sensor data read NA for the RAM modules which spins up the fans on the server, I don't know how to determine what on the OS is causing that but it sounds like something is trying to read that information and is locking up the sensor.

                            deric created this issue in lm-sensors/lm-sensors

                            closed Driver for IPMI BMC KCS #69

                            L 1 Reply Last reply Reply Quote 0
                            • L Offline
                              LennertvdBerg @rmaclachlan
                              last edited by LennertvdBerg

                              @rmaclachlan Thanks. I'm also unsure how we can determine what in the OS is causing this issue. Are there other installations or modifications we could try to help isolate the problem, such as another Linux distribution with the same kernel, to see if it's a kernel-related issue? @gduperrey or @olivierlambert any suggestions how we can help the team with identifying this?

                              T 1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                I have no idea, sorry. Hopefully someone could tell us what we need to add in XCP-ng to fix it.

                                1 Reply Last reply Reply Quote 0
                                • T Offline
                                  ThierryEscande Vates 🪐 XCP-ng Team @LennertvdBerg
                                  last edited by

                                  @LennertvdBerg To verify that the IPMI drivers are not messing up with the firmware, you could try to blacklist the IPMI modules.
                                  Create the file /etc/modprobe.d/blacklist-ipmi.conf containing the following and reboot

                                  blacklist ipmi_si
                                  blacklist ipmi_devintf
                                  blacklist ipmi_msghandler
                                  

                                  Actually add a line for whatever modules the command lsmod | grep ipmi gives you

                                  L 1 Reply Last reply Reply Quote 1
                                  • L Offline
                                    LennertvdBerg @ThierryEscande
                                    last edited by

                                    @ThierryEscande . When I do lsmod | grep ipmiI get the following results

                                    ipmi_si                65536  0 
                                    ipmi_devintf           20480  0 
                                    ipmi_msghandler        61440  2 ipmi_devintf,ipmi_si
                                    

                                    So, I created the file with vi /etc/modprobe.d/blacklist-ipmi.conf and added the following:

                                    blacklist ipmi_si
                                    blacklist ipmi_devintf
                                    blacklist ipmi_msghandler
                                    

                                    I saved the file and rebooted the system using shutdown -r now. However, I still don't see the memory temperatures in Xclarity, and the server's fans are still running at over 13,000 RPM. The system is running XCP-NG 8.3 beta 2 with kernel 4.19.0+1.

                                    1 Reply Last reply Reply Quote 0
                                    • T Offline
                                      ThierryEscande Vates 🪐 XCP-ng Team
                                      last edited by

                                      We have a new kernel-alt release available for testing purpose. This is only for XCP-ng 8.2.1 for now. The package for XCP-ng 8.3 should land soon.

                                      ⚠ THIS IS A TEMPORARY REPO FOR TESTING PURPOSE ONLY ⚠

                                      To install this kernel-alt package:

                                      • Edit the xcp-ng.repo file
                                      # vi /etc/yum.repos.d/xcp-ng.repo
                                      
                                      • Add the following lines at the end of the file:
                                      [xcp-ng-tescande]
                                      name=XCP-ng 8.2 tescande User Repository
                                      baseurl=https://koji.xcp-ng.org/repos/user/8/8.2/tescande1/x86_64/
                                      enabled=0
                                      gpgcheck=0
                                      priority=1
                                      
                                      • Install the kernel-alt package
                                      # yum --enablerepo=xcp-ng-tescande install kernel-alt
                                      
                                      • Reboot and select XCP-ng kernel-alt 4.19.309 at the grub screen.
                                      RIX_ITR L 2 Replies Last reply Reply Quote 0
                                      • RIX_ITR Offline
                                        RIX_IT @ThierryEscande
                                        last edited by

                                        @ThierryEscande
                                        I've upgraded to the kernel-alt 4.19.309 while still using the old UEFI (kae110k 1.41). Currently the IPMI modules are not blacklisted in the modprobe config.

                                        The fan speeds remain stuck at 9k RPMs, consistent with both the stable and previous alt kernel versions.

                                        I also attempted to upgrade again to the newest UEFI (kae118m 4.11) but didn't notice any discernible difference in fan behavior - here it's still around 13k RPMs.
                                        The ipmitool output also didn't change from the one @rmaclachlan provided.

                                        If there are any other suggestions for testing or specific logs you'd like me to provide, please let me know.

                                        T 1 Reply Last reply Reply Quote 0
                                        • T Offline
                                          ThierryEscande Vates 🪐 XCP-ng Team @RIX_IT
                                          last edited by

                                          @RIX_IT Thanks for your feedback.

                                          Can you please share the output of dmesg and xl dmesg ?

                                          RIX_ITR 1 Reply Last reply Reply Quote 0
                                          • T Offline
                                            ThierryEscande Vates 🪐 XCP-ng Team
                                            last edited by

                                            Even if this will probably be same as with XCP-ng 8.2, the kernel-alt package is available for 8.3.

                                            Same steps as in my previous post, except for the repo URL in xcp-ng.repo:

                                            [xcp-ng-tescande]
                                            name=XCP-ng 8.3 tescande User Repository
                                            baseurl=https://koji.xcp-ng.org/repos/user/8/8.3/tescande2/x86_64/
                                            enabled=0
                                            gpgcheck=0
                                            priority=1
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post