@ThierryEscande

4.19.265-1 & -2 (havent tried earlier ones)

# dmesg | grep -i intel
   Intel GenuineIntel
 ioatdma: Intel(R) QuickData Technology Driver 4.00
# dmesg | grep eth
 xen_netfront: Initialising Xen virtual ethernet driver

4.19.0+1

# dmesg | grep -i intel
   Intel GenuineIntel
 ioatdma: Intel(R) QuickData Technology Driver 4.00
 ice: Intel(R) Ethernet Connection E800 Series Linux Driver - version 1.6.4
 ice: Copyright (C) 2018-2021, Intel Corporation.

# dmesg | grep eth
 xen_netfront: Initialising Xen virtual ethernet driver
 ice 0000:41:00.0 side-1974-eth0: renamed from eth0
 ice 0000:41:00.1 side-1128-eth1: renamed from eth1
 ice 0000:81:00.0 side-3431-eth2: renamed from eth2
 ice 0000:81:00.1 side-597-eth3: renamed from eth3
 ice 0000:41:00.0 eth0: renamed from side-1974-eth0
 ice 0000:41:00.1 eth1: renamed from side-1128-eth1
 ice 0000:81:00.0 eth2: renamed from side-3431-eth2
 ice 0000:81:00.1 eth3: renamed from side-597-eth3
 device eth0 entered promiscuous mode
 device eth2 entered promiscuous mode
 ice 0000:81:00.0 eth2: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None
 8021q: adding VLAN 0 to HW filter on device eth2
 ice 0000:41:00.0 eth0: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None
 8021q: adding VLAN 0 to HW filter on device eth0
 device eth1 entered promiscuous mode
 device eth3 entered promiscuous mode
 ice 0000:41:00.1 eth1: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None
 8021q: adding VLAN 0 to HW filter on device eth1
 ice 0000:81:00.1 eth3: NIC Link is up 10 Gbps Full Duplex, Requested FEC: RS-FEC, Negotiated FEC: NONE, Autoneg Advertised: Off, Autoneg Negotiated: False, Flow Control: None
 8021q: adding VLAN 0 to HW filter on device eth3
 8021q: adding VLAN 0 to HW filter on device eth2
 8021q: adding VLAN 0 to HW filter on device eth2
 8021q: adding VLAN 0 to HW filter on device eth3
 8021q: adding VLAN 0 to HW filter on device eth3
 8021q: adding VLAN 0 to HW filter on device eth1
 8021q: adding VLAN 0 to HW filter on device eth1
 8021q: adding VLAN 0 to HW filter on device eth0
 8021q: adding VLAN 0 to HW filter on device eth0

Tested 4.19.265.1 which also has no networking for me, so its not the most recent update that killed it.

Looks like the "ice" driver is never initialized/loaded, which is what the NIC requires.

Also appears to be some IPMI_SI errors

Kernel config differences from existing std (4.19.0+1) to testing (4.19.265)

Network interfaces we have are Intel(R) E810-DA2 10/25GbE SFP28 2-Port

# Linux/x86 4.19.0+1 Kernel Configuration                     | # Linux/x86 4.19.265 Kernel Configuration
                                                              > CONFIG_CC_HAS_ASM_GOTO=y
                                                              > # CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
                                                              > CONFIG_SYSTEM_DATA_VERIFICATION=y
# CONFIG_X86_AMD_PLATFORM_DEVICE is not set                   | CONFIG_X86_AMD_PLATFORM_DEVICE=y
                                                              > CONFIG_X86_INTEL_TSX_MODE_OFF=y
                                                              > # CONFIG_X86_INTEL_TSX_MODE_ON is not set
                                                              > # CONFIG_X86_INTEL_TSX_MODE_AUTO is not set
# CONFIG_DELL_RBU is not set                                  | CONFIG_DELL_RBU=m
                                                              > CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
                                                              > CONFIG_ARCH_USE_MEMREMAP_PROT=y
CONFIG_ASN1=m                                                 | CONFIG_ASN1=y
                                                              > CONFIG_UNIX_SCM=y
# CONFIG_WIRELESS is not set                                  | CONFIG_WIRELESS=y
                                                              > CONFIG_CFG80211=m
                                                              > # CONFIG_NL80211_TESTMODE is not set
                                                              > # CONFIG_CFG80211_DEVELOPER_WARNINGS is not set
                                                              > # CONFIG_CFG80211_CERTIFICATION_ONUS is not set
                                                              > CONFIG_CFG80211_REQUIRE_SIGNED_REGDB=y
                                                              > CONFIG_CFG80211_USE_KERNEL_REGDB_KEYS=y
                                                              > CONFIG_CFG80211_DEFAULT_PS=y
                                                              > # CONFIG_CFG80211_DEBUGFS is not set
                                                              > CONFIG_CFG80211_CRDA_SUPPORT=y
                                                              > # CONFIG_CFG80211_WEXT is not set
                                                              > # CONFIG_MAC80211 is not set
                                                              > CONFIG_MAC80211_STA_HASH_MAX_SIZE=0
# CONFIG_FW_LOADER_USER_HELPER is not set                     | CONFIG_FW_LOADER_USER_HELPER=y
                                                              > # CONFIG_FW_LOADER_USER_HELPER_FALLBACK is not set
# CONFIG_BLK_DEV_SR_VENDOR is not set                         <
# CONFIG_ASIX_PHY is not set                                  | # CONFIG_AX88796B_PHY is not set
# CONFIG_WLAN is not set                                      | CONFIG_WLAN=y
                                                              > # CONFIG_WIRELESS_WDS is not set
                                                              > CONFIG_WLAN_VENDOR_ADMTEK=y
                                                              > CONFIG_WLAN_VENDOR_ATH=y
                                                              > # CONFIG_ATH_DEBUG is not set
                                                              > # CONFIG_ATH5K_PCI is not set
                                                              > # CONFIG_ATH6KL is not set
                                                              > # CONFIG_WIL6210 is not set
                                                              > CONFIG_WLAN_VENDOR_ATMEL=y
                                                              > # CONFIG_ATMEL is not set
                                                              > CONFIG_WLAN_VENDOR_BROADCOM=y
                                                              > # CONFIG_BRCMFMAC is not set
                                                              > CONFIG_WLAN_VENDOR_CISCO=y
                                                              > # CONFIG_AIRO is not set
                                                              > CONFIG_WLAN_VENDOR_INTEL=y
                                                              > # CONFIG_IPW2100 is not set
                                                              > # CONFIG_IPW2200 is not set
                                                              > CONFIG_WLAN_VENDOR_INTERSIL=y
                                                              > # CONFIG_HOSTAP is not set
                                                              > # CONFIG_HERMES is not set
                                                              > # CONFIG_PRISM54 is not set
                                                              > CONFIG_WLAN_VENDOR_MARVELL=y
                                                              > # CONFIG_LIBERTAS is not set
                                                              > # CONFIG_MWIFIEX is not set
                                                              > CONFIG_WLAN_VENDOR_MEDIATEK=y
                                                              > CONFIG_WLAN_VENDOR_RALINK=y
                                                              > CONFIG_WLAN_VENDOR_REALTEK=y
                                                              > CONFIG_WLAN_VENDOR_RSI=y
                                                              > CONFIG_WLAN_VENDOR_ST=y
                                                              > CONFIG_WLAN_VENDOR_TI=y
                                                              > CONFIG_WLAN_VENDOR_ZYDAS=y
                                                              > # CONFIG_USB_ZD1201 is not set
                                                              > CONFIG_WLAN_VENDOR_QUANTENNA=y
                                                              > # CONFIG_QTNFMAC_PEARL_PCIE is not set
                                                              > # CONFIG_USB_NET_RNDIS_WLAN is not set
                                                              > # CONFIG_THUNDERBOLT_NET is not set
# CONFIG_NVM is not set                                       <
                                                              > CONFIG_LDISC_AUTOLOAD=y
# CONFIG_R3964 is not set                                     <
                                                              > CONFIG_RANDOM_TRUST_BOOTLOADER=y
# CONFIG_PINCTRL is not set                                   | CONFIG_PINCTRL=y
                                                              > # CONFIG_DEBUG_PINCTRL is not set
                                                              > # CONFIG_PINCTRL_AMD is not set
                                                              > # CONFIG_PINCTRL_MCP23S08 is not set
                                                              > # CONFIG_PINCTRL_SX150X is not set
                                                              > # CONFIG_PINCTRL_BAYTRAIL is not set
                                                              > # CONFIG_PINCTRL_CHERRYVIEW is not set
                                                              > # CONFIG_PINCTRL_BROXTON is not set
                                                              > # CONFIG_PINCTRL_CANNONLAKE is not set
                                                              > # CONFIG_PINCTRL_CEDARFORK is not set
                                                              > # CONFIG_PINCTRL_DENVERTON is not set
                                                              > # CONFIG_PINCTRL_GEMINILAKE is not set
                                                              > # CONFIG_PINCTRL_ICELAKE is not set
                                                              > # CONFIG_PINCTRL_LEWISBURG is not set
                                                              > # CONFIG_PINCTRL_SUNRISEPOINT is not set
                                                              > # CONFIG_BATTERY_RT5033 is not set
CONFIG_DRM_I915_GVT_XENGT=m                                   <
CONFIG_FB=y                                                   <
CONFIG_FIRMWARE_EDID=y                                        <
                                                              > CONFIG_FB=y
                                                              > CONFIG_FIRMWARE_EDID=y
CONFIG_VGACON_SOFT_SCROLLBACK=y                               <
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=64                         <
# CONFIG_VGACON_SOFT_SCROLLBACK_PERSISTENT_ENABLE_BY_DEFAULT  <
# CONFIG_USB_RIO500 is not set                                <
# CONFIG_THUNDERBOLT is not set                               | CONFIG_THUNDERBOLT=m
# CONFIG_NTFS_RW is not set                                   | CONFIG_NTFS_RW=y
# CONFIG_IMA_TEMPLATE is not set                              <
CONFIG_CRYPTO_AKCIPHER=m                                      | CONFIG_CRYPTO_AKCIPHER=y
CONFIG_CRYPTO_RSA=m                                           | CONFIG_CRYPTO_RSA=y
# CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE is not set             | CONFIG_ASYMMETRIC_PUBLIC_KEY_SUBTYPE=y
                                                              > CONFIG_X509_CERTIFICATE_PARSER=y
                                                              > CONFIG_PKCS7_MESSAGE_PARSER=y
                                                              > # CONFIG_PKCS7_TEST_KEY is not set
                                                              > # CONFIG_SIGNED_PE_FILE_VERIFICATION is not set
# CONFIG_SYSTEM_TRUSTED_KEYRING is not set                    | CONFIG_SYSTEM_TRUSTED_KEYRING=y
                                                              > CONFIG_SYSTEM_TRUSTED_KEYS=""
                                                              > # CONFIG_SYSTEM_EXTRA_CERTIFICATE is not set
                                                              > # CONFIG_SECONDARY_TRUSTED_KEYRING is not set
CONFIG_OID_REGISTRY=m                                         | CONFIG_OID_REGISTRY=y

@ThierryEscande said in High Fan Speed Issue on Lenovo ThinkSystem Servers:

The kernel-alt package with X86_AMD_PLATFORM_DEVICE enabled is available for testing.

This is for XCP-ng 8.2.1 only for now.

Use yum to install it:

yum --enablerepo=xcp-ng-testing install kernel-alt

Reboot and select XCP-ng kernel-alt 4.19.265 at the grub screen.

It fixes the memory temp reading, and therefore fan controls - BUT I no longer have any networking.
Reboot to std kernel and its back just fine.

grep CONFIG_X86_AMD_PLATFORM_DEVICE /boot/config-4.19.0+1
# CONFIG_X86_AMD_PLATFORM_DEVICE is not set

A rebuilt kernel with this enabled might be all is needed.

@olivierlambert said in High Fan Speed Issue on Lenovo ThinkSystem Servers:

Hi,

Does it solve the fan speed issue? If yes, might worth digging more.

Yes absolutely - fans are operating as per normal/expectation now.

According to https://support.lenovo.com/ie/en/solutions/tt2254-xenserver-8-cannot-acquire-the-dimm-temperature-and-fan-speed-is-too-fast-on-the-amd-platform-lenovo-thinksystem this is a work around.

The main problem is:

CONFIG_X86_AMD_PLATFORM_DEVICE is a configuration setting in Linux kernel which would affect initial system HW setting. In this case, due to CONFIG_X86_AMD_PLATFORM_DEVICE is not set, the clock source of AMDI0010 (the device inside CPU) can't be registered which results in incorrect I2C command output from it and causes the DIMM temperature retrieval failure in XCC side.
The system fan speed is automatically varies based on DIMM temperature readings, and when the DIMM temperature being error, the fan speed will be tuned to high rotation.

@ThierryEscande @bleader @olivierlambert Hope this helps if XenServer don't come to the party.
Given the new information about the I2C-DesignWare driver - I've disabled it on one of my SR635V3 systems, and I can read the DIMM temps now

Lenovo ThinkSystem SR635 V3, BIOS KAE118M-4.11

USE AT YOUR OWN RISK - I have not tested this in production, or found out what else it might effect
However given the "workaround" provided by Lenovo for XenServer - it should be OK.

The driver is built into the kernel instead of a module, so had to be blacklisted slightly differently.

Found which driver to blacklist from kernel init.
fgrep i2c /boot/System.map-4.19.0+1 | grep init

ffffffff815172f0 T drm_i2c_encoder_init
ffffffff815574c0 T __regmap_init_i2c
ffffffff81557510 T __devm_regmap_init_i2c
ffffffff815d0700 t i2c_dw_init_master
ffffffff815d14f0 t i2c_dw_init_slave
ffffffff81ea9b40 r __ksymtab_drm_i2c_encoder_init
ffffffff81eb0570 r __ksymtab___devm_regmap_init_i2c
ffffffff81eb0838 r __ksymtab___regmap_init_i2c
ffffffff81edc1a9 r __kstrtab_drm_i2c_encoder_init
ffffffff81edffcd r __kstrtab___devm_regmap_init_i2c
ffffffff81edffe4 r __kstrtab___regmap_init_i2c
ffffffff8248156d t i2c_init
ffffffff82481b65 t dw_i2c_init_driver
ffffffff8255fe48 t __initcall_i2c_init2
ffffffff8255ffa8 t __initcall_dw_i2c_init_driver4

Guessed dw_i2c_init_driver as the most likely candidate

In /etc/grub-efi.cfg I added initcall_blacklist=dw_i2c_init_driver and ran grub-mkconfig then rebooted.

menuentry 'XCP-ng' {
        search --label --set root root-ltcoyb
        multiboot2 /boot/xen.gz watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 dom0_mem=16384M,max:16384M
        module2 /boot/vmlinuz-4.19-xen root=LABEL=root-ltcoyb ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles initcall_blacklist=dw_i2c_init_driver
        module2 /boot/initrd-4.19-xen.img
}

Restarted the BMC to be sure, and checked. Previously these reported 0 degrees C for populated DIMM slots.
ipmitool sdr | grep -i temp

Ambient Temp     | 18 degrees C      | ok
Exhaust Temp     | 35 degrees C      | ok
CPU 1 Temp       | 31 degrees C      | ok
DIMM 1 Temp      | no reading        | ns
DIMM 2 Temp      | 27 degrees C      | ok
DIMM 3 Temp      | no reading        | ns
DIMM 4 Temp      | 29 degrees C      | ok
DIMM 5 Temp      | 30 degrees C      | ok
DIMM 6 Temp      | 29 degrees C      | ok
DIMM 7 Temp      | 29 degrees C      | ok
DIMM 8 Temp      | 29 degrees C      | ok
DIMM 9 Temp      | 29 degrees C      | ok
DIMM 10 Temp     | no reading        | ns
DIMM 11 Temp     | 27 degrees C      | ok
DIMM 12 Temp     | no reading        | ns

Then check the fans rpm (previously 18000 front & 15000 rear)
ipmitool sdr | grep -i fan

Fan Mismatch     | 0x00              | ok
Fan 1 Front Tach | 7168 RPM          | ok
Fan 2 Front Tach | 7168 RPM          | ok
Fan 3 Front Tach | 7168 RPM          | ok
Fan 4 Front Tach | 7168 RPM          | ok
Fan 5 Front Tach | 7168 RPM          | ok
Fan 6 Front Tach | 7168 RPM          | ok
Fan 7 Front Tach | 0 RPM             | ok
Fan 1 Rear Tach  | 6048 RPM          | ok
Fan 2 Rear Tach  | 5940 RPM          | ok
Fan 3 Rear Tach  | 6048 RPM          | ok
Fan 4 Rear Tach  | 5940 RPM          | ok
Fan 5 Rear Tach  | 5940 RPM          | ok
Fan 6 Rear Tach  | 5940 RPM          | ok
Fan 7 Rear Tach  | 0 RPM             | ok
Sys Fan Pwr      | 6 Watts           | ok

Nothing that seems relevant in the changelog.

==================================================
Version 4.20, Build ID KAE120J [Critical]
Release date: [Jun/ 2024]
Release Ref: FW Agile 24-2 [AMD 2S V3][AMD 1S V3]    
==================================================

Support Systems: 
  Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H
  Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F
  Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B
  Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D

This UEFI supports AMD processor:
 [SR665v3][SR645v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo","Genoa-X"), AGESA GenoaPI-SP5_1.0.0.C
 [SR655v3][SR635v3]support: AMD EPYC 9004 Series processor (formerly codenamed "Genoa","Bergamo"), AGESA GenoaPI-SP5_1.0.0.C
 
1.0 Prerequisites and dependencies
 Supported "Continuous" option for boot order through the IPMI command or redfish interface -- this feature also needs the related support at XCC side through version KAX334O-3.20.
 Prerequisite build is XCC Version KAX322Z-2.31.

2.0 Fixes
  [SR655v3] Fixed the issue that the PCIe slot 7 of ThinkSystem SR655 x8/x8 PCIe Riser3 could not detect PCIe device rightly.
  [SR665v3][SR645v3][SR655v3][SR635v3] Added more MMIO reservation padding to support ThinkSystem 2.5" U.3 PM1743 NVME SSD hot add.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that NVMe SSD (connects to backplane) speed was degraded to Gen4.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that assert tag of SMBIOS type 3 shows incorrectly in ESXi 7.0 U3n and later version.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that power-on password could not be entered rightly when SOL was activate via IPMI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Removed the option of "Disable" for "System Settings > Processors > 3D V-Cache" in System Setup Utility and Processors.3DV-Cache to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "System Settings > Processors > MONITOR/MWAIT" is not grey out when "System Settings > Processors > Global C-state Control" is "Enabled".
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would hang if executing "Diagnostics > Memory Test > Advanced Memory Test" on LXPM when ThinkSystem 2.5 U.3 6500 ION 30.72TB Read Intensive NVMe PCIe 4.0 x4 HS SSD is attached. (TT2149)
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that system would not load default when the number of consecutive failed POST attempts reaches the POST Attempts Limit.
  [SR665v3][SR645v3][SR655v3][SR635v3] Fixed the issue that "Processors > Number of Enabled CPU Cores Per Socket" can not configured the value smaller than 16.
  [SR665v3][SR645v3][SR655v3][SR635v3] Moved NVMe devices information from "System Settings > Storage" to "System Settings > NVMe" in System Setup Utility.

3.0 Enhancements
  [SR665v3][SR645v3][SR655v3][SR635v3] Integrated AMD GenoaPI version 1.0.0.C
  - Updated uCode to
  Genoa B1: 0x0A101148
  Genoa B2: 0x0A101248
  Bergamo A1: 0x0AA00116
  Bergamo A2: 0x0AA00215
  - Address CVE-2024-21978 (Medium), CVE-2024-21980 (High), CVE-2023-31355(High), CVE-2023-31315(High) 
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Security > Trusted Platform Module >TPM 2.0 > Hide TPM from OS" in System Setup Utility and TrustedComputingGroup.HideTPMfromOS to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Periodic Directory Rinse" in System Setup Utility and Processors.PeriodicDirectoryRinse to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > Probe Filter Organization" in System Setup Utility and Processors.ProbeFilterOrganization to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Exposed "System Settings > Processors > P-State" in System Setup Utility and Processors.P-State to OneCLI.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed the default value of "System Settings > Devices and I/O Ports > PCIe Ten Bit Tag" in System Setup Utility and DevicesandIOPorts.PCIeTenBitTagSupport in OneCLI from  "Disabled" to "Enabled" 
  [SR665v3][SR645v3][SR655v3][SR635v3] Supported "Continuous" option for boot order through the IPMI command or redfish interface.
  [SR665v3][SR645v3][SR655v3][SR635v3] Changed minimum memory frequency from 3200 Mhz to 3600 Mhz.

@olivierlambert - Yes, so when we do a V2V there is only 1 VIF listed to put the migrated VIFs in to, even if the VM has 5 VIFs

Is it possible to query each vmx and allow us to select a 'network' for each VIF that it has?

Any plans to check how many vNICs there are and allow us to map each one correctly.

All of our vmware VMs have multiple vNICs and are in different port-groups.

Its not a major hassle to change them post-migration - just a nice-to-have.

Forgot to mention - I pointed Lenovo at this thread, but got the usual "OS isn't supported, find an alternative"

I have recently purchased four SR635v3 servers with AMD EPYC 9354P cpu.

I managed to get some info from Lenovo about how to manually change the fan speeds.

How to get the existing fan speeds. If this fails, check IPMI settings in the BMC.

[09:10 xcp01 ~]# ipmitool sdr list | grep -i fan
Fan Mismatch     | 0x00              | ok
Fan 1 Front Tach | 19072 RPM         | ok
Fan 2 Front Tach | 19072 RPM         | ok
Fan 3 Front Tach | 18944 RPM         | ok
Fan 4 Front Tach | 19072 RPM         | ok
Fan 5 Front Tach | 18944 RPM         | ok
Fan 6 Front Tach | 19072 RPM         | ok
Fan 7 Front Tach | 0 RPM             | ok
Fan 1 Rear Tach  | 15768 RPM         | ok
Fan 2 Rear Tach  | 15768 RPM         | ok
Fan 3 Rear Tach  | 15876 RPM         | ok
Fan 4 Rear Tach  | 15876 RPM         | ok
Fan 5 Rear Tach  | 15768 RPM         | ok
Fan 6 Rear Tach  | 15768 RPM         | ok
Fan 7 Rear Tach  | 0 RPM             | ok
Sys Fan Pwr      | 48 Watts          | ok

To change the fans to a fixed percentage - note the decimal "35" this is the percentage of maximum fan speed to set to.

ipmitool raw 0x3a 0x07 0xff 35 0x01

After a short while for the fans to spin down.

Fan Mismatch     | 0x00              | ok
Fan 1 Front Tach | 12288 RPM         | ok
Fan 2 Front Tach | 12160 RPM         | ok
Fan 3 Front Tach | 12160 RPM         | ok
Fan 4 Front Tach | 12288 RPM         | ok
Fan 5 Front Tach | 12160 RPM         | ok
Fan 6 Front Tach | 12288 RPM         | ok
Fan 7 Front Tach | 0 RPM             | ok
Fan 1 Rear Tach  | 10152 RPM         | ok
Fan 2 Rear Tach  | 10152 RPM         | ok
Fan 3 Rear Tach  | 10260 RPM         | ok
Fan 4 Rear Tach  | 10152 RPM         | ok
Fan 5 Rear Tach  | 10260 RPM         | ok
Fan 6 Rear Tach  | 10152 RPM         | ok
Fan 7 Rear Tach  | 0 RPM             | ok
Sys Fan Pwr      | 15 Watts          | ok

To revert to auto-fan control you either need to restart the BMC/XCC controller which isn't system impacting.

Please keep in mind this was stressed to me that its NOT SUPPORTED/RECOMMENDED - and could cause thermal damage to internal components!!

We only have the M.2 mirror kit with two small disks, no other raid cards or internal disks.
If you have RAID and internal disks you might want to fix your fans to a higher speed.

Someone with scripting knowledge could also monitor a few temps and adjust the IPMI fan control accordingly.
This could even be managed externally via IPMI over LAN for multiple servers.