Applied latest candidate test updates to 3 x Dell OptiPlex 7040 (i7-6700, 48GB, 10Gbps-attached TrueNAS shared-storage) pool. Update process was error-free and successful. Everything appears to be working normally following the update.

@stormi Installed all of the test updates on my three-host home-lab this weekend. Similar configuration to @gskger 3 x Dell OptiPlex 7040 SFF hosts and home-built FreeNAS server with separate physical 1Gb networks for management, storage and migration. I call it my "Tiny Cluster" due to its diminutive footprint. I use it for configuration prototyping. Intel VPRO AMT on Xen hosts and storage server enables headless console operation using MeshCommander (think poor man's iDRAC). All updates were installed without issue. Backups and restores seem to work just fine. Of special interest to me was the UEFI Secure Boot capabilities. Installed the x64 dbx.auth from uefi.org (I presume since XCP-ng is 64-bit that that was the correct choice. Probably should be made explicit in the instructions.) Seems to work perfectly. I tested with Windows 10-20H2 and Windows 10-21H1. Also tested with RHEL 8.4 which has built-in support for secure boot (Microsoft-signed bootloader shim) and that too "just works." The varstore-ls <VM-uuid> command shows PK, KEK, dbx and db in the store as expected. Stops unsigned bootloader as expected on unsupported OSes. Looks great! Thank you for all of the work you've put into it. I suspect designing and building emulated system firmware is not for the faint of heart . . . Very impressive!

Latest version 8.3 candidate updates installed and are working fine on three-host home lab pool. Received a couple of repo errors for a certain mirror, but yum tried another mirror and it completed successfully. After updates were applied, performed live migrations between hosts with no problems and updated a Windows 11 Version 24H2 VM to the November 2024 cumulative update without problems. (VM is currently running Citrix Tools 9.3.2 without issues.)

Hi All. Yes, this is a very annoying problem that I've also experienced after a fresh Windows 11-24H2 install on XCP-ng 8.3 fully production-patched to date. I am accessing my Windows 11 VM console via a Windows 11-24H2 physical client PC using latest Firefox browser. The keyboard and mouse attached to my laptop via a Dell DisplayLink D3100 USB3 dock are a standard wired Logitech mouse with scroll and a wired Logitech keyboard. The XCP-ng 8.3 host is managed via XO from source (XOS) on the latest commit (66e67) as of yesterday 2/16/2025. XOS lives in an AlmaLinux 8.10 VM built with @ronivay 's superb installation script.

After some Googling around, this frozen mouse issue appears to have occurred in other hypervisors too. It looks to be a Windows problem rather than an XCP-ng 8.3/XO/QEMU problem. (I see you smiling @olivierlambert ).

I can't guarantee this technique will work for everyone, but after a day, I am no longer experiencing the mouse failure.

What appears to be happening is that the Windows Plug-and-Play (PNP) mouse driver configuration is getting borked due to multiple triggerings of PNP. During the first-boot of the VM post-installation, it finds the original emulated hardware. Following the installation of the Citrix management agent 9.4, it performs additional device configuration that doesn't always go well. In the device manager, click view, show hidden devices to see any phantom devices that I generally remove so as to keep everything as clean and pristine as possible.

This Windows registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class{4D36E96F-E325-11CE-BFC1-08002BE10318} is your friend.

You must make sure that the value mouclass is the only value in the UpperFilters key of the above device class hive. As a general preventative against Windows oddities, I changed the value and then changed it back to mouclass to force the registry editor to rewrite the hive. You should also delete the mouse instance details folders 0000, 0001 etc. These should get deleted for you when you remove the mouse devices from the device manager. Windows will recreate those during the reboot.

Random aside: another thing I like to do is to change the VM's UEFI OVMF display settings to 1280x960 in the Tiano UEFI firmware. This allows me to see the entire VM on my 1920x1080 HD monitor when Firefox is in full-screen mode, XO console scale set to 100%, and the Windows VM display resolution also set to 1280x960. This is intended to prevent weird visual scaling anomalies.

The following image is my device manager after the fix. When the mouse was malfunctioning, the system had only created the PS/2 mouse device. The HID-compliant mouse was created after deleting the original PS/2 mouse device and the failed USB Universal Host Controller devices in device manager. Following this, scan for new devices to recreate what is missing and reboot the VM so that those devices get registered and initialized correctly.

Some additional screenshots of the mouse instance registry hive values:

Applying the latest XCP-ng 8.3 RC2 "xen-*" and intel-microcode updates from the candidate repository worked great here as well. Everything appears to be running well.

Latest updates over ISO-installed 8.3 RC2 worked fine for me. I did experience one host in my three-host pool to which no VMs could be migrated. After looking at the networking from bash in DOM0, it showed that both 10G ports for the storage and migration networks were DOWN. These ports are on a genuine IBM-branded Intel X540-T2 card I bought used on eBay so it might have gone bad. Since the card has worked well for some time, I figured it couldn't hurt to re-seat it in the PCIe slot. Sure enough, that fixed it. Moral of the story: check the mundane stuff first; it's not always the fault of new updates.

Upgraded 3-node pool (home lab: Dell OptiPlex 7040 SFF x 3) from 8.3 RC1 to 8.3 RC2 using bootable ISO. It worked perfectly for me. As others have noted, it does ask you to select the management interface when upgrading the slave nodes. Once you do that, it automatically populates all of the previously configured network parameters for that host so you are really only confirming the existing values. The OptiPlex 7040's (i7-6700) all have Intel VPro AMT so they are running headless. The MeshCommander program is used to access the VPro console on each host. A DisplayPort display emulator dongle is needed to keep the integrated-GPU active in order to be able to see the console and firmware setup screens with this configuration. It's effectively a poor man's iDRAC. So far, everything is working well on 8.3 RC2.

@stormi All new patches applied fine. No apparent problems identified so far.

@beshleman I tried the latest testing update @stormi published with the updated SB support and it does indeed work properly including allowing installation of Windows Update KB4535680 on Server 2019 as previously cited. Also--a big thank you for adding the default parameter values for the improved secureboot-certs install command. Less is more. Very nice!

@beshleman So, after yum --enablerepo=base install python-requests on each of my hosts, secureboot-certs install default default default latest works perfectly. (Cool that it installs certs to each host in the pool with one invocation from any pool host.) Interesting that it doesn't install the three files to /var/lib/uefistored until you secure boot a vm on each host. I went looking for them and was initially confused because they were only written to the pool db. Serves me right for looking under the hood! Looks like XCP-ng secure boot is ready for prime time. Great job!

Latest urgent updates installed on 3-node Intel pool. Everything is working as before including the pre-production code "no stats" issue, but that still resolves following xe-toolstack-restart command. Since it is currently Microsoft patch Tuesday here, the latest Windows 11-24H2 2025-05 cumulative update was installed to VM along with various Linux VM updates and live VM host migrations. All working well including latest af03c Xen Orchestra from source (XOS).

@Greg_E Maybe it only affects Intel? That would be a new wrinkle. Perhaps other AMD users can confirm if the latest AlmaLinux 8 kernel runs fine on their AMD XCP-ng 8.3 hosts? Just read the above referenced RedHat bug report here: https://access.redhat.com/solutions/7116307 Sure enough, it says Xen + Intel in problem description. Check it out.

@Greg_E Wondering what might be the difference? I am using the XenServer Linux tools version 8.4.0-1 and BIOS boot firmware.

Latest test updates applied to 3-host Dell OptiPlex 7040 SFF (i7-6700) pool. Upgrade went smoothly and everything appears to be working normally. I did notice the flatlined stats issue mentioned previously, but the xe-toolstack-restart seemed to bring those back to life. Thanks @Greg_E and @abudef for catching and addressing that.

This is just a "me too" reply to indicate that I am also experiencing the boot failure immediately after upgrading AlmaLinux 8.10 to the latest kernel 4.18.0-553.50.1. Hopefully, the kernel fix will get integrated soon into the affected and popular RedHat 8 derivatives so that Alma and Rocky 8 et al. can continue to run on our favorite hypervisor. This is a bad one.

@piotrlotr1 Will definitely use the trick if needed in the future. Normally, changing the network for a running VM is not something we typically need to do. Thanks!

All, Just to let you know that I have observed this same behavior on Windows 11 24H2 with the latest Citrix drivers. After changing the network, the CPU spikes to 100% and the VM becomes unusable. In that case, the only fix is to perform a forced reboot. As others have noted here, sometimes the VM will continue to run okay, but the CPU spike failure happens more often than not.

Latest test updates were applied to three-node home lab pool (3 x Dell OptiPlex SFF 7040 i7-6700/48GB RAM/256GB NVMe boot drive/TrueNAS Core 10Gbps-attached NFS shared storage) without incident. VMs appear to running normally, live migration between hosts works fine and no apparent problems so far.

Hi All. Yes, this is a very annoying problem that I've also experienced after a fresh Windows 11-24H2 install on XCP-ng 8.3 fully production-patched to date. I am accessing my Windows 11 VM console via a Windows 11-24H2 physical client PC using latest Firefox browser. The keyboard and mouse attached to my laptop via a Dell DisplayLink D3100 USB3 dock are a standard wired Logitech mouse with scroll and a wired Logitech keyboard. The XCP-ng 8.3 host is managed via XO from source (XOS) on the latest commit (66e67) as of yesterday 2/16/2025. XOS lives in an AlmaLinux 8.10 VM built with @ronivay 's superb installation script.

After some Googling around, this frozen mouse issue appears to have occurred in other hypervisors too. It looks to be a Windows problem rather than an XCP-ng 8.3/XO/QEMU problem. (I see you smiling @olivierlambert ).

I can't guarantee this technique will work for everyone, but after a day, I am no longer experiencing the mouse failure.

What appears to be happening is that the Windows Plug-and-Play (PNP) mouse driver configuration is getting borked due to multiple triggerings of PNP. During the first-boot of the VM post-installation, it finds the original emulated hardware. Following the installation of the Citrix management agent 9.4, it performs additional device configuration that doesn't always go well. In the device manager, click view, show hidden devices to see any phantom devices that I generally remove so as to keep everything as clean and pristine as possible.

This Windows registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class{4D36E96F-E325-11CE-BFC1-08002BE10318} is your friend.

You must make sure that the value mouclass is the only value in the UpperFilters key of the above device class hive. As a general preventative against Windows oddities, I changed the value and then changed it back to mouclass to force the registry editor to rewrite the hive. You should also delete the mouse instance details folders 0000, 0001 etc. These should get deleted for you when you remove the mouse devices from the device manager. Windows will recreate those during the reboot.

Random aside: another thing I like to do is to change the VM's UEFI OVMF display settings to 1280x960 in the Tiano UEFI firmware. This allows me to see the entire VM on my 1920x1080 HD monitor when Firefox is in full-screen mode, XO console scale set to 100%, and the Windows VM display resolution also set to 1280x960. This is intended to prevent weird visual scaling anomalies.

The following image is my device manager after the fix. When the mouse was malfunctioning, the system had only created the PS/2 mouse device. The HID-compliant mouse was created after deleting the original PS/2 mouse device and the failed USB Universal Host Controller devices in device manager. Following this, scan for new devices to recreate what is missing and reboot the VM so that those devices get registered and initialized correctly.

Some additional screenshots of the mouse instance registry hive values:

Applied latest candidate test updates to 3 x Dell OptiPlex 7040 (i7-6700, 48GB, 10Gbps-attached TrueNAS shared-storage) pool. Update process was error-free and successful. Everything appears to be working normally following the update.