On the Pool > Advanced tab you can select a backup network so if you created a backup network for a specific NIC you could set it there

@RIX_IT I tried kernel-alt before rolling back the uefi and it didn't help. Also tried the new Xen beta stormi posted (4.19 was it?) in case something was added but that didn't fix it either.

Thanks for trying the new Lenovo fw - I saw it come out and was going to test when I had time but now I don't need to!

@RIX_IT I rolled my XCC and UEFI back to last May and the fans finally quieted down.

BMC Version
2.10 (Build ID: KAX318R)
UEFI Version
1.41 (Build ID: KAE110K)

I noticed when the fans spun up the XCC page was showing multiple DIMMs temp as NA and when I checked

ipmitool sensor

in the XCPNG terminal the DIMM # Temp was also 0 when the fans were up. I'm assuming the old kernel XCPNG runs is causing some havoc with the UEFI Lenovo has on these servers and is preventing that sensor from being read so XCC freaks out and spins the fans up because it assumes the temp on that sensor is really high. No idea why the old XCC/UEFI seem to work though.

@bleader My older UEFI fix didn't work long as the fans were spun up again this morning. The servers are nice and quiet when booting to AlmaLinux or like RIX_IT mentioned Windows Server so could this be an issue with Kernal 4.19 on XCP not playing well with new EPYC chipsets?

@bleader yeah I'm not sure how the OS could affect that either and certainly haven't seen that in the past...very odd.

I am on 8.3 beta 2. I haven't tested with 8.2

Let me know if you figure this out, I've been fighting with this issue for the last couple days as well. We have SR635v3 servers with AMD EPYC 9354P processors. After GRUB the fans spin down, and the kernel begins to load, once I see efi: EFI_MEMMAP is not enabled the fans spin back up.

In the interim the fix has been to roll back UEFI firmware back to 1.43 which loses some microcode updates but at least our fans aren't screaming at 16k RPM in an office environment.

Our main pool upgraded with no issues with the ISO. Same as the above posters though, master doesn't ask for network config but other hosts in the pool do...thankfully all the bonds and networking came up without issue on them.

Will attempt updating our other locations after hours but I don't foresee any problems

Not sure if this is a known issue, I couldn't find it on Github but maybe I was looking in the wrong spot. Is anyone else's network Bond speeds 0? I thought I had seen a fix a month ago for this in a release but now I can't find that changelog

Same here (ticket is in).

Anyone having issues with NFS SR performance too? Since the update my VMs are transferring about 140MB/s vs before the update we were seeing speeds around 440MB/s.

I'm using "dd if=/dev/zero of=tmp bs=1G oflag=dsync count=1" to test, if I SSH onto the host I can get full speed to the NFS mount but from within a VM it is much slower, this means backups are also crawling since XOA can't read the VHD very quickly.

iSCSI performance seems unaffected.

Thank you @florent for all your help! We got the VM to import now, I will try the other failed VM outside business hours but I expect it will work now as well!

XenServer has a fix in their latest release for this - https://docs.xenserver.com/en-us/xenserver/8/whats-new/normal

These updates fix the following issues:

    The fan on Lenovo AMD systems is always at full speed and DIMM temperatures are not reported correctly by the BMC.

Does this mean the same fix can be implemented into xcpng 8.3?

Our main pool upgraded with no issues with the ISO. Same as the above posters though, master doesn't ask for network config but other hosts in the pool do...thankfully all the bonds and networking came up without issue on them.

Will attempt updating our other locations after hours but I don't foresee any problems

Seems to be running fine on our 12 hosts, iSCSI came up fine too with all our paths.

@ThierryEscande I kept all the files from the acpidump from both new and old fw. I've ran that on both sets of acpi dumps which produced quite a few dsl files (one per ssdt) so I've just zipped both folders for you here:

ACPI-Lenovo.zip

Thanks having a look! I've dumped the ACPI before and after firmware updates and linked them below for you.

New FW

Old FW

I had some time to test today so I upgraded the FW on a server, disabled IPMI as suggested above in the kernel and the fans remained spun up.

I disabled ACPI in grub (acpi=off) and the fans didn't spin up but then dom0 failed to fully load so that isn't great lol

Is it just a matter of the ACPI kernel driver being outdated? I'm not sure how to check that

@LennertvdBerg The above message is indicating there is no driver written for IPMI BMC KCS for the LM_Sensors application, not that a kernel module is missing in XCPNG.

You can read more about this on the lm_sensors github issue https://github.com/lm-sensors/lm-sensors/issues/69

If you wish to view sensor data in XPCNG you can do so through IPMI still using ipmitool

ipmitool sensor

This will list the sensors in the server.

DIMM 1           | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
DIMM 1 Temp      | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 2           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 2 Temp      | 23.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 3           | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
DIMM 3 Temp      | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 4           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 4 Temp      | 25.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 5           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 5 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 6           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 6 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 7           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 7 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 8           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 8 Temp      | 26.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 9           | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 9 Temp      | 25.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 10          | 0x0        | discrete   | 0x0080| na        | na        | na        | na        | na        | na
DIMM 10 Temp     | na         | degrees C  | na    | na        | na        | na        | 85.000    | 87.000    | 91.000
DIMM 11          | 0x0        | discrete   | 0x4080| na        | na        | na        | na        | na        | na
DIMM 11 Temp     | 24.000     | degrees C  | ok    | na        | na        | na        | 85.000    | 87.000    | 91.000

The issue is when upgrading the UEFI we start seeing the sensor data read NA for the RAM modules which spins up the fans on the server, I don't know how to determine what on the OS is causing that but it sounds like something is trying to read that information and is locking up the sensor.

deric created this issue in lm-sensors/lm-sensors

closed Driver for IPMI BMC KCS #69

@Gheppy Our SR635v3 are running factory installed Lenovo hardware, although I did test swapping out the Broadcom 57504 OCP NIC with Intel X710 but the fan issue persisted. Thanks for the suggestion!

@RIX_IT I tried kernel-alt before rolling back the uefi and it didn't help. Also tried the new Xen beta stormi posted (4.19 was it?) in case something was added but that didn't fix it either.

Thanks for trying the new Lenovo fw - I saw it come out and was going to test when I had time but now I don't need to!

@RIX_IT I rolled my XCC and UEFI back to last May and the fans finally quieted down.

BMC Version
2.10 (Build ID: KAX318R)
UEFI Version
1.41 (Build ID: KAE110K)

I noticed when the fans spun up the XCC page was showing multiple DIMMs temp as NA and when I checked

ipmitool sensor

in the XCPNG terminal the DIMM # Temp was also 0 when the fans were up. I'm assuming the old kernel XCPNG runs is causing some havoc with the UEFI Lenovo has on these servers and is preventing that sensor from being read so XCC freaks out and spins the fans up because it assumes the temp on that sensor is really high. No idea why the old XCC/UEFI seem to work though.