@RIX_IT
I've been informed by XenServer trial support that Lenovo has filed the ticket LEN-605 for this issue, it's probably a private issue tracker though.
They'll let me know once it is fixed.

@johnarvid That error looks like the error that I've been getting... https://xcp-ng.org/forum/topic/4353/delta-backup-failed-invalid-argument

I've switched back to full backups for now.
I'm still investigating and troubleshooting though so no solutions yet...

@nraynaud @olivierlambert Thanks guys!

Thanks for the tip.

I was using node v8.11.4 and had issues compiling during the upgrade then upgraded to the latest stable node version which wasn't v8.x.

I upgraded to v8.16.2 now and it runs fine. Thanks.

@gduperrey Is this alternate kernel fix for the fans already released with the November patches for XCP-ng 8.2?
https://xcp-ng.org/blog/2024/11/15/november-2024-security-and-maintenance-update-for-xcp-ng-8-2-lts/

🔧 Update for alternate kernel

Backport of a fix to correct cooling fan rotation speed on some Lenovo servers.

@RIX_IT
I've been informed by XenServer trial support that Lenovo has filed the ticket LEN-605 for this issue, it's probably a private issue tracker though.
They'll let me know once it is fixed.

@RIX_IT Thank you for the tips.

Just for the record, I downgraded the SR645 v3 to UEFI v1.25 KAE106Y and the fans are now spinning at 9000RPM and the fan speed adjusts accordingly depending on the CPU load so I think this is the correct behaviour.

But once upgrade to version UEFI 2.10 KAE112N for the SR645 v3, the fan spins up to 20000RPM with unbearable sound. This also occurs with the latest firmware 4.20 KAE120J.

I opened a ticket with XenServer 8 Trial Edition bugs portal and they finally replied with "This issue is on our radar. We're currently developing a solution. To avoid duplicates, we'll archive this ticket for now"
Not sure when or how it will be fixed though.

@rfx77
I opened a support ticket with Lenovo but still no luck since XenServer 8 is a Vendor Certified OS and not a Lenovo Supported OS.

Vendor Certified: The server has met the OS Partner Certified/Logo program requirements, however, hardware device driver updates may not be available beyond the inbox OS device drivers. As a result support is limited and customers should strongly consider this when determining which OS version to install. The OS vendor provides OS support with inbox drivers only. Lenovo only provides hardware level support, and will work with hardware partners and OS vendors in a best-effort capacity to close any compatibility issues customers may experience with their installation. Lenovo will work with the OS provider to submit hardware compatibility (HCL) test suite results so that the supported solution can be posted to the respective OS providers HCL site.

The SR645 v3 has different UEFI firmware versions to the SR635 v3, it looks like v1.25 for SR645 v3 is equivilent to v1.43 for SR635 v3

==================================================
Version 1.25, Build ID KAE106Y [Critical]
Release date: [May/ 2023]
Release Ref: Genoa Wave2 EAR602 [AMD 2S V3]
==================================================

Support Systems: 
  Lenovo ThinkSystem SR645v3 Server, Machine Types 7D9C/7D9D
  Lenovo ThinkSystem SR665v3 Server, Machine Types 7D9A/7D9B
  
This UEFI supports AMD processor:
  AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.3

1.0 Prerequisites and dependencies
  None
 
2.0 Fixes
  Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page.
  Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior.
 
3.0 Enhancements
 None

4.0 Limitations 
  None


==================================================
Version 1.43, Build ID KAE110O [Critical]
Release date: [May/ 2023]
Release Ref: Genoa Wave2 EAR602 [AMD 1S V3]
==================================================

Support Systems: 
  Lenovo ThinkSystem SR635v3 Server, Machine Types 7D9G/7D9H
  Lenovo ThinkSystem SR655v3 Server, Machine Types 7D9E/7D9F

This UEFI supports AMD processor:
    AMD EPYC 9004 Series processor (formerly codenamed "Genoa"), AGESA GenoaPI-SP5_1.0.0.4
  
1.0 Prerequisites and dependencies
  None

2.0 Fixes
   Fixed the issue that configuring OperatingModes.ChooseOperatingMode to "Maximum Performance" from OOB did not match the same configuration in UEFI F1 Setup page.
   Addressed AMD's recent Genoa processor errata that in rare scenarios can result in #UD, #PF, or other unexpected system behavior.
  
3.0 Enhancements
   None
	
4.0 Limitations 
  None

What I want to know though is where to download the older version of the UEFI firmware? I can only see the latest version on the Lenovo website.

@rfx77
That's what I thought of too and tried booting into the Xenserver 8 iso XenServer8_2024-06-03.iso.
Even booting with the ISO installer will cause the fans to spin up abnormally fast, up to 20,000RPM.

@RIX_IT

I just noticed this issue today when I installed XCP-ng 8.2.1 on a ThinkSystem SR645 v3 with UEFI KAE120J-4.20.

The fans are spinning at high 16,800 to 20,224 RPM the high pitch sound is unbearably annoying.

I changed operating mode in the UEFI settings from maximum performance to maximum efficiency. It made no difference fans are still spinning as fast as before.

@JoyceBabu said in windows 11 Support:

JoyceBabu
20 days ago

@okynnor I installed Win11 after disabling TPM, RAM and SafeMode checks by adding the DWORD32 keys BypassTPMCheck, BypassRAMCheck and BypassSecureBootCheck under HKEY_LOCAL_MACHINE\System\Setup\LabConfig.

@okynnor Full details here: https://www.bleepingcomputer.com/news/microsoft/how-to-bypass-the-windows-11-tpm-20-requirement/

Xen drivers from windows updates didn't install properly for all device. So I installed Xen Tools from here https://downloads.xenserver.com/vm-tools-windows/9.3.1/managementagentx64.msi and works fine.

@Darkbeldin I checked the xensource.log on the server and found the error message "failed with exception Server_error(INVALID_DEVICE, [ autodetect ])" during export, similar to the post here: https://xcp-ng.org/forum/topic/4319/invalid_device-autodetect-during-backup

I checked the XO dashboard health and found 256 "Pool Metadata Backup" VDIs attached to Control Domain.

Not sure how or why the "Pool Metadata Backup" got stuck attached to Control Domain but I guess it has reached the 256 limit. I "Forget" all of them and full backup seems to be working now.

For the record, it wasn't anything to do with snapshots. The VM was OFF so I guess it doesn't need to take a snapshot before transfer hence the log doesn't mention snapshot.

Hi @Darkbeldin

I disabled compression, the XVA is now 37KB

Normally it should say snapshot above transfer but I don't see it for some reason.

When trying to restore the backup, I get IMPORT_ERROR_PREMATURE_EOF().

I'm curently running XO (5.107.1) community from source and have had scheduled backups for one of the XCP-ng server 8.0.0 regularly without issues previously with XO version 5.102.1. I originally wondered if it has something to do with the new version of XO but I rolled back to 5.98.1 and still have the same issue.

I am now getting successful backups but only in kilobytes for all the VMs on this particular XCP-ng server. Not sure if it has anything to do with it but I noticed in the gui showing the status of the job changing quickly from Started -> Interrupted -> Sucessful.

     transfer
        Start: Dec 6, 2022, 02:18:05 PM
        End: Dec 6, 2022, 02:18:42 PM
        Duration: a few seconds
        Size: 5.62 KiB
        Speed: 154.26 B/s
    Start: Dec 6, 2022, 02:18:05 PM
    End: Dec 6, 2022, 02:18:42 PM
    Duration: a few seconds

Start: Dec 6, 2022, 02:18:03 PM
End: Dec 6, 2022, 02:18:42 PM
Duration: a few seconds
Type: full

Backup config is Full backup with Zstd compression and normal snapshot, SMB remote. If I include VMs from the problematic XCP-ng server and another in the backup job. It executes correctly and succesfully for one of them and the problematic server backup is only 5.63KB.

Nothing else has changed that I'm aware of, I have rebooted the XCP-ng server, XO server, and the remote storage, tries specifying a different remote storage point. I also create a new VM backup job but still the same. Other XCP-ng server backups from XO are working normally.

What else can I check? Does it have something to do with an error during snapshot? I doesn't look like it created the snapshot before transferring the 5KB.

Thanks.