Hello,

We're glad to announce a new Alpha version 9.0.9000.0 of our XCP-ng Windows PV Drivers for Windows 10 1607 or newer, and Windows Server 2016 or newer. Note: this driver release is for testing purposes only. Not for production use!

The release includes new 9.1-series drivers from Xen Project with new features and stability improvements. We also included a XenClean tool for cleanly removing any existing Xen PV drivers, whether they are installed via XCP-ng, Citrix or Windows Update. Feel free to give it a try!

We're looking to finalize the drivers and produce a signed, Windows-ready installation package. Your feedback will help us greatly in reaching this goal.

You'll find the new release on XCP-ng GitHub. Below are the release notes:

Before Installing

• This driver release is for testing purposes only. Not for production use!
• Make backups/snapshots before installing!
• The drivers in this package are testsigned and require enabling testsigning mode. Disable Secure Boot, then run the included script testsign/install.ps1 as Administrator to configure Windows and install the necessary signer certificates. Your VM will reboot automatically.
• If you encounter installation/uninstallation errors, please try again with one of the following commands:

For installing:
msiexec.exe /i XenDrivers-x64.msi /l*vx install.log

For uninstalling:
msiexec.exe /x XenDrivers-x64.msi /l*vx uninstall.log

Please include this log along with the file C:\Windows\INF\setupapi.dev.log in your bug report. These files will help us troubleshoot any installation issues.

Changes since 8.2.2-beta

• This release is based on upstream 9.1-series drivers, which includes the PV Mouse/Keyboard driver and PV Console Driver. You can access the PV console with the following command from a XCP-ng host:

xl console -t pv <vmname>

• New installer with clean uninstallation and multiple install/uninstall safety checks.
• New XenClean utility for cleanly removing XCP-ng and Citrix drivers.
• Multiple driver stability fixes.
• Volume Shadow Service Provider is no longer included. (XCP-ng 8.1 and newer no longer support quiescent snapshots)
• Management Agent is currently not included. Xen Orchestra may show "Management agent not detected", but shutdown and reboot features will work normally. We're looking to bring back features of the management agent in the next releases.

Help / Community Support

Please report any errors or BSOD you encounter during testing of this release. Your feedback is very appreciated.

edit: Before using XenClean or XenBootFix, please read carefully the instructions on the XCP-ng documentation website!

Hello all,

Version 9.0.9030 of the new Windows PV drivers has been released.

This release brings multiple driver stability fixes, a new Rust-based Xen Guest Agent and the XenBootFix boot repair tool.

This driver release is for testing purposes only and not for production use. It requires putting the system into testsign mode. (only applies to the drivers themselves, not XenBootFix and XenClean; you can use the two tools separately without any configuration)

Download the release here: https://github.com/xcp-ng/win-pv-drivers/releases/tag/v9.0.9030

Release notes are below:

Before Installing

• This driver release is for testing purposes only. Not for production use!
• Make backups/snapshots before installing!
• The drivers in this package are testsigned and require enabling testsigning mode. Disable Secure Boot, then run the included script testsign/install.ps1 as Administrator to configure Windows and install the necessary signer certificates. Your VM will reboot automatically.
• If you're running version 9.0.9000.0, we're interested in hearing about your upgrade experience to 9.0.9030.

Changes since 9.0.9000.0

• NEW: Includes the new Rust-based Xen Guest Agent for resource and IP address reporting.
• NEW: Includes the new XenBootFix boot repair tool for VMs rendered unbootable by any Xen drivers (Xen PV, XCP-ng, Citrix, etc.)
• NOTICE: Xen PV disk drivers are now disabled by default on new installations. The default emulated NVMe driver will be used instead. Existing installations will not be affected.
• Fixes: Many more installation, uninstallation and upgrade safety checks are now built into the installer.
• Fixes: Various improvements to XenClean for more complete removal of driver services.
• Fixes: xenvbd: Xen PV disk drives now report as SSD drives to avoid unnecessary defragmentation (commit)
• Fixes: xenvif: Fix various VM hanging and crashing issues when changing VM network in Xen Orchestra (commit 1, commit 2)
• Debug symbols are now included with the installation package.

Changes since 8.2.2-beta

• This release is based on upstream 9.1-series drivers, which includes the PV Mouse/Keyboard driver and PV Console Driver. You can access the PV console with the following command from a XCP-ng host:

xl console -t pv <vmname>

• New installer with clean uninstallation and multiple install/uninstall safety checks.
• New XenClean utility for cleanly removing XCP-ng and Citrix drivers.
• Multiple driver stability fixes.
• Volume Shadow Service Provider is no longer included. (XCP-ng 8.1 and newer no longer support quiescent snapshots)
• Management Agent is currently not included. Xen Orchestra may show "Management agent not detected", but shutdown and reboot features will work normally. We're looking to bring back features of the management agent in the next releases. Check out the new Rust-based Xen Guest Agent included in our package! We're looking to bring back even more features to the agent.
• Older Windows versions are no longer supported. The driver requires at a minimum Windows 10 1607 or Windows Server 2016.

Help / Community Support

If you encounter installation/uninstallation errors, please try again with one of the following commands:

For installing:
msiexec.exe /i XenDrivers-x64.msi /l*vx install.log

For uninstalling:
msiexec.exe /x XenDrivers-x64.msi /l*vx uninstall.log

Please include this log along with the file C:\Windows\INF\setupapi.dev.log in your bug report. These files will help us troubleshoot any installation issues.

Additionally, please report any errors or BSOD you encounter during testing of this release. Your feedback is very appreciated.

• Discussion: https://xcp-ng.org/forum (preferred)
• Issue Tracker: https://github.com/xcp-ng/xcp/issues
• IRC: #xcp-ng and #xcp-ng-dev on irc.freenode.net

Full Changelog: https://github.com/xcp-ng/win-pv-drivers/compare/v9.0.9000.0...v9.0.9030

This is a driver bug that we fixed in XCP-ng Windows tools v9.0.9030 but hasn't been integrated by Citrix yet. You can try it out if you're not running a production system.

@anilx You can use the XenClean tool from our 9.0.9000 driver release: https://github.com/xcp-ng/win-pv-drivers/releases/tag/v9.0.9000.0

Just go into package\XenClean\x64 and run Invoke-XenClean.ps1 as admin, it will remove all existing drivers and automatically reboot.
Afterwards you can install the Citrix 9.4.0 tools.

Note: Please make sure to take a snapshot before running the tool.

You simply install the XenServer tools as usual, making sure to disable driver installing and auto-updating in the setup wizard:

@ickwilli Interesting. It points to a possible issue between QEMU's USB tablet emulation and Windows rather than an issue related to XO.

You should look into using Sysprep, with it you can create a generalized template and specify domain join credentials for joining each new VM.

It's a known issue with the new Citrix drivers: https://support.citrix.com/s/article/CTX235403-updates-to-xenserver-vm-tools-for-windows-for-xenserver-and-citrix-hypervisor?language=en_US

It shouldn't happen again in future updates.

If you use the Windows Update option, then you still need to install XenServer VM tools to provide the management agent.
If the 8.2 XCP-ng tools work for you, feel free to continue using it; however, if you want the newest fixes you should use XenServer VM Tools 9.4.0.
In any case, you should keep the "Manage Citrix PV drivers" option disabled and let the installer manage its drivers.

It looks like the GeoVision dongle's driver is failing somehow. Could you open the crash dump with Windbg, run the command !analyze -v and post its output?

@tatitica Not having stornvme\StartOverride is expected on systems that use NVMe controllers to boot (e.g. XCP-ng Windows templates). Can your imported VM boot on the correct Windows template once you have made sure stornvme\StartOverride is not present? Once it boots with the correct template and NVMe controller then let's see what the performances are.

@tatitica If you're using the Windows templates, then stornvme is correct. Do you have some unusual NVMe drivers (Intel drivers being one) installed?

@tatitica You can do that before powering down. There's XenBootFix, which isn't really made for this, but may also work.

@tatitica It's because of a change in Windows storage driver. Deleting HKLM\SYSTEM\CurrentControlSet\Services\stornvme\StartOverride before booting on the new template should make it work.

This could be done with XO Powershell too (eventually).

This is a driver bug that we fixed in XCP-ng Windows tools v9.0.9030 but hasn't been integrated by Citrix yet. You can try it out if you're not running a production system.

@he_jest I agree that sudden crashes probably mean hardware failure. Could you post the crash logs?

@evilDan Windows 11 never fails to surprise... Can you try with Server 2025?

There's a specific timing needed to attach the debugger, refer to this thread: https://xcp-ng.org/forum/topic/10284/vm-failing-to-reboot

@evilDan It's only mandatory for the installation. Once it's installed you can disable secure boot and enable debugging.

@evilDan I noticed that in your earlier freezes, the CPU went to 0% but in these recent freezes the CPU is pegged to 100% instead. Does your guest still behave the same way during the freeze (no RDP/XOA console but still responding to ping)? It's an advanced process, but you can try attaching a kernel debugger and see what's happening. https://docs.xcp-ng.org/troubleshooting/windows-pv-tools/#connecting-to-guests-using-serial-console