High Fan Speed Issue on Lenovo ThinkSystem Servers
-
@ThierryEscande I'm experiencing difficulties with installing the
kernel-altpackage on my system. Currently, I am using XCP-ng version 8.3.0-beta2. It appears that there might be a problem with updategrub.py. Any guidance on how to resolve this would be greatly appreciated.I've update
xcp-ng.repoand this is my output ofyum --enablerepo=xcp-ng-tescande install kernel-alt:[20:12 xcp-ng-test1 ~]# yum --enablerepo=xcp-ng-tescande install kernel-alt Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org Resolving Dependencies --> Running transaction check ---> Package kernel-alt.x86_64 0:4.19.309-1.0.lenovotest.2.xcpng8.3 will be installed --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================================================================================================================================================================================== Package Arch Version Repository Size =========================================================================================================================================================================================================================================================================================== Installing: kernel-alt x86_64 4.19.309-1.0.lenovotest.2.xcpng8.3 xcp-ng-tescande 30 M Transaction Summary =========================================================================================================================================================================================================================================================================================== Install 1 Package Total download size: 30 M Installed size: 154 M Is this ok [y/d/N]: y Downloading packages: kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64.rpm | 30 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64 1/1 /var/tmp/rpm-tmp.l9rbzO: line 9: /opt/xensource/bin/updategrub.py: No such file or directory warning: %post(kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64) scriptlet failed, exit status 127 Non-fatal POSTIN scriptlet failure in rpm package kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64 Verifying : kernel-alt-4.19.309-1.0.lenovotest.2.xcpng8.3.x86_64 1/1 Installed: kernel-alt.x86_64 0:4.19.309-1.0.lenovotest.2.xcpng8.3 -
@ThierryEscande I've updated to
Xen 4.17and it seems the upgrade went fine:host : xcp-ng-test1 release : 4.19.0+1 version : #1 SMP Wed Jan 24 17:19:11 CET 2024 machine : x86_64 nr_cpus : 64 max_cpu_id : 63 nr_nodes : 1 cores_per_socket : 32 threads_per_core : 2 cpu_mhz : 3245.126 hw_caps : 178bf3ff:7efa320b:2e500800:244037ff:0000000f:f1bf97a9:00405fce:00000780 virt_caps : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2 total_memory : 130850 free_memory : 121721 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 xen_major : 4 xen_minor : 17 xen_extra : .3-3 xen_version : 4.17.3-3 xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : $Format:%H$, pq ??? xen_commandline : dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 cc_compiler : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1) cc_compile_by : mockbuild cc_compile_domain : [unknown] cc_compile_date : Wed Feb 28 10:12:19 CET 2024 build_id : 9a011a28e29a21a7643376b36aec959253587d42 xend_config_format : 4However, the issues with the fan speeds and missing memory temperature readings still persist.
-
@ThierryEscande
Of course. See the attached files:What I also did notice was that networking isn't working at all with the 4.19.309 kernel. Settings were the same as before and I couldn't even ping the gateway. I tried an emergency reset and reconfiguring, but still no luck. Upon booting with the stable kernel, it worked again as expected. But that's an issue for another day, I guess.
-
I have recently purchased four SR635v3 servers with AMD EPYC 9354P cpu.
I managed to get some info from Lenovo about how to manually change the fan speeds.
How to get the existing fan speeds. If this fails, check IPMI settings in the BMC.
[09:10 xcp01 ~]# ipmitool sdr list | grep -i fan Fan Mismatch | 0x00 | ok Fan 1 Front Tach | 19072 RPM | ok Fan 2 Front Tach | 19072 RPM | ok Fan 3 Front Tach | 18944 RPM | ok Fan 4 Front Tach | 19072 RPM | ok Fan 5 Front Tach | 18944 RPM | ok Fan 6 Front Tach | 19072 RPM | ok Fan 7 Front Tach | 0 RPM | ok Fan 1 Rear Tach | 15768 RPM | ok Fan 2 Rear Tach | 15768 RPM | ok Fan 3 Rear Tach | 15876 RPM | ok Fan 4 Rear Tach | 15876 RPM | ok Fan 5 Rear Tach | 15768 RPM | ok Fan 6 Rear Tach | 15768 RPM | ok Fan 7 Rear Tach | 0 RPM | ok Sys Fan Pwr | 48 Watts | okTo change the fans to a fixed percentage - note the decimal "35" this is the percentage of maximum fan speed to set to.
ipmitool raw 0x3a 0x07 0xff 35 0x01After a short while for the fans to spin down.
Fan Mismatch | 0x00 | ok Fan 1 Front Tach | 12288 RPM | ok Fan 2 Front Tach | 12160 RPM | ok Fan 3 Front Tach | 12160 RPM | ok Fan 4 Front Tach | 12288 RPM | ok Fan 5 Front Tach | 12160 RPM | ok Fan 6 Front Tach | 12288 RPM | ok Fan 7 Front Tach | 0 RPM | ok Fan 1 Rear Tach | 10152 RPM | ok Fan 2 Rear Tach | 10152 RPM | ok Fan 3 Rear Tach | 10260 RPM | ok Fan 4 Rear Tach | 10152 RPM | ok Fan 5 Rear Tach | 10260 RPM | ok Fan 6 Rear Tach | 10152 RPM | ok Fan 7 Rear Tach | 0 RPM | ok Sys Fan Pwr | 15 Watts | okTo revert to auto-fan control you either need to restart the BMC/XCC controller which isn't system impacting.
Please keep in mind this was stressed to me that its NOT SUPPORTED/RECOMMENDED - and could cause thermal damage to internal components!!
We only have the M.2 mirror kit with two small disks, no other raid cards or internal disks.
If you have RAID and internal disks you might want to fix your fans to a higher speed.Someone with scripting knowledge could also monitor a few temps and adjust the IPMI fan control accordingly.
This could even be managed externally via IPMI over LAN for multiple servers. -
Forgot to mention - I pointed Lenovo at this thread, but got the usual "OS isn't supported, find an alternative"
-
Well, it would be more productive if they could only tell which component and version is needed to correctly handle their own fans…
-
@olivierlambert would there be a way after GRUB to walk step by step through the boot and see where it goes wrong?
-
I had some time to test today so I upgraded the FW on a server, disabled IPMI as suggested above in the kernel and the fans remained spun up.
I disabled ACPI in grub (acpi=off) and the fans didn't spin up but then dom0 failed to fully load so that isn't great lol
Is it just a matter of the ACPI kernel driver being outdated? I'm not sure how to check that
-
@rmaclachlan If it's an ACPI issue and since Lenovo doesn't seem to be very cooperative, one could try to downgrade the firmware to a working version (i.e. one that runs fans at normal speed) and dump the ACPI table. Then upgrade to the latest firmware, dump the ACPI tables again, and then compare them.
I don't know ACPI much but I can have a look if you can share them.
The ACPI tools should be already installed on any XCP-ng host.
- Dump the ACPI tables in binary format
Do so in an empty folder as this produces numerous files
# acpidump -b- Decompile the
dsdt.datfile
# iasl -e ssdt*.dat -d dsdt.dat- Do the same operations for both firmwares and share the
dsdt.dslfiles.
The files are pretty big so don't hesitate to compress them before sharing.
- Dump the ACPI tables in binary format
-
-
@rmaclachlan Thanks for the files. Did not see anything obvious at first sight.
I forgot to ask you for ssdt files too. Would it be possible to do the same with these files ?
iasl -d ssdt*.dat(I hope you kept the old firmware ones somewhere, otherwise don't bother to downgrade again. Just share the new firmware ssdt files)
-
@ThierryEscande I kept all the files from the acpidump from both new and old fw. I've ran that on both sets of acpi dumps which produced quite a few dsl files (one per ssdt) so I've just zipped both folders for you here:
-
@rmaclachlan Thanks a lot. Unfortunately I did not find any evidence of what could be wrong from the ACPI tables.
It obviously does not come from the IPMI devices as there is no modification in this area.
So without help from Lenovo it will be difficult for us to go further. If you manage to get Lenovo involved one way or another we will be happy to collaborate and help.
-
@ThierryEscande Has anyone made any progress on this? @Riven you got contact details at Lenovo for contacting regarding this?
-
@LennertvdBerg no update on that, sorry. As said before, it will be hard to tell what's going on without feedback from Lenovo.
-
@LennertvdBerg
We already tried getting into contact with Lenovo a while ago. But like I already stated, they weren't able to escalate the ticket because of the unsupported OS. That's the same response that @Riven got.Maybe you could drop Lenovo a ticket as well and point them to this thread. Let's see if it helps if more people report this issue. Otherwise we seem to be pretty much out of Luck.
We have one of our two servers now in production running the old UEFI, Sound Level is not great, but bearable.
Still definitely far from an optimal solution. -
@RIX_IT I've dropped today a ticket as well, hoping them to realise it would be beneficial for all parties if they could help solving this.
-
@LennertvdBerg Lenovo didn't want to provide any support. However, they just published a new UEFI/BIOS. Not sure if this is going to fix things;
-
Nothing that seems relevant in the changelog.
================================================== Version 4.20, Build ID KAE120J [Critical] Release date: [Jun/ 2024] Release Ref: FW Agile 24-2 [AMD 2S V3][AMD 1S V3] ================================================== Support Systems: Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D This UEFI supports AMD processor: [SR665v3][SR645v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo","Genoa-X"), AGESA GenoaPI-SP5_1.0.0.C [SR655v3][SR635v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo"), AGESA GenoaPI-SP5_1.0.0.C 1.0 Prerequisites and dependencies Supported "Continuous" option for boot order through the IPMI command or redfish interface -- this feature also needs the related support at XCC side through version KAX334O-3.20. Prerequisite build is XCC Version KAX322Z-2.31. 2.0 Fixes [SR655v3] Fixed the issue that the PCIe slot 7 of ThinkSystem SR655 x8/x8 PCIe Riser3 could not detect PCIe device rightly. [SR665v3][SR645v3][SR655v3][SR635v3] Added more MMIO reservation padding to support ThinkSystem 2.5" U.3 PM1743 NVME SSD hot add. [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that NVMe SSD (connects to backplane) speed was degraded to Gen4. [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that assert tag of SMBIOS type 3 shows incorrectly in ESXi 7.0 U3n and later version. [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that power-on password could not be entered rightly when SOL was activate via IPMI. [SR665v3][SR645v3][SR655v3][SR635v3] Removed the option of "Disable" for "System Settings > Processors > 3D V-Cache" in System Setup Utility and Processors.3DV-Cache to OneCLI. [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "System Settings > Processors > MONITOR/MWAIT" is not grey out when "System Settings > Processors > Global C-state Control" is "Enabled". [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would hang if executing "Diagnostics > Memory Test > Advanced Memory Test" on LXPM when ThinkSystem 2.5 U.3 6500 ION 30.72TB Read Intensive NVMe PCIe 4.0 x4 HS SSD is attached. (TT2149) [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would not load default when the number of consecutive failed POST attempts reaches the POST Attempts Limit. [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "Processors > Number of Enabled CPU Cores Per Socket" can not configured the value smaller than 16. [SR665v3][SR645v3][SR655v3][SR635v3] Moved NVMe devices information from "System Settings > Storage" to "System Settings > NVMe" in System Setup Utility. 3.0 Enhancements [SR665v3][SR645v3][SR655v3][SR635v3] Integrated AMD GenoaPI version 1.0.0.C - Updated uCode to Genoa B1: 0x0A101148 Genoa B2: 0x0A101248 Bergamo A1: 0x0AA00116 Bergamo A2: 0x0AA00215 - Address CVE-2024-21978 (Medium), CVE-2024-21980 (High), CVE-2023-31355(High), CVE-2023-31315(High) [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Security > Trusted Platform Module >TPM 2.0 > Hide TPM from OS" in System Setup Utility and TrustedComputingGroup.HideTPMfromOS to OneCLI. [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Periodic Directory Rinse" in System Setup Utility and Processors.PeriodicDirectoryRinse to OneCLI. [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Probe Filter Organization" in System Setup Utility and Processors.ProbeFilterOrganization to OneCLI. [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > P-State" in System Setup Utility and Processors.P-State to OneCLI. [SR665v3][SR645v3][SR655v3][SR635v3] Changed the default value of "System Settings > Devices and I/O Ports > PCIe Ten Bit Tag" in System Setup Utility and DevicesandIOPorts.PCIeTenBitTagSupport in OneCLI from "Disabled" to "Enabled" [SR665v3][SR645v3][SR655v3][SR635v3] Supported "Continuous" option for boot order through the IPMI command or redfish interface. [SR665v3][SR645v3][SR655v3][SR635v3] Changed minimum memory frequency from 3200 Mhz to 3600 Mhz. -
@LennertvdBerg I just tested the new UEFI combined with the new XCC Firmware. Also installed all pending XCP-ng updates while being at it.
XCC Firmware didn't make any difference, so this one should be updated as Lenovo addressed quite a few CVE's.However, the new UEFI is still not the solution to our problem, just as @Riven already figured by the changelogs. Fans are still sitting at around 12-13k RPMs. Also the DIMM Temps are still not shown in XCC.
