XenServer VM Tools 9.3.3 from Citrix causes bluescreen
-
@Toni
How did you disable it? With the registry-key? -
@KPS In Xen Orchestra (VM/Advanced) I turned the switch "Windows Update tools" to off.
-
@Toni
Did you reboot twice?https://xcp-ng.org/forum/topic/9207/disable-manage-citrix-pv-drivers-via-windows-update-after-vm-creation/9
--> I am having issues after the second reboot... -
@KPS Yes, I rebooted twice. With the "Windows Update Tool" switch in the off position everything is ok.
Normally I always have the switch in the on position.Summary:
Windows Update Tools off:
Installation of 9.3.3 via Citrix Installer -> OKWindows Update Tools on:
Upgrade from a previous version to 9.3.3 -> blue screen
Removal of the VM Tools and reinstallation via Citrix Installer -> blue screen after the second reboot
It seems like the error is caused by Windows Update. -
@Toni yep. I had to do this too. 2 of my 5 Windows VMs got blue screened by the Citrix drivers. There is something going on beyond a doubt! My VMs have been flawless for years now until yesterday 6/17/24.
-
@Mt_KEGan
What was/is your config? Did youvhave auto update enabled IN the VM and on hypervisor level?I am worried about that… did you open a support ticket?
-
@KPS I have the Citrix package installed and the XCP-ng "manage PV drivers via Windows update" toggled on. I remember being very careful on my initial setup and following the proper guidelines, but I wouldn't bet my life on it.
The more concerning thing for me is that I have 1 vm with the same configuration and it's fine, with another 2 blue screening. They are literally clones of each other. Maybe the others will fail soon too (I know it takes a couple reboots).
I have backups and remediation options to attempt as others have posted here. Ie: uninstall Citrix package, update, toggle off auto update etc. I feel confident I'll find the right combo.
Again, just strange that I've had none of these issues after all my years using this hypervisor. ... I love Vates and this community... So I'm not overly concerned yet.
Great work keeping an eye on this and thank you!!!!
-
Confirming issue.
All Windows 10 VMs on one host bluescreen upon update to 9.3.3 (even with toggling XOA option to update by Windows Update off). Used recommended install settings.
9.3.1 and 9.3.2 cause no issues.
Interestingly, all Windows 10 VMs on another host (different pool) have no issues with 9.3.3.
Seems hardware specific: Xeon host fine; Intel i7 host bluescreening.
-
@clip
Did you update manually, or did you have automatic updates active? -
@KPS Updated manually via download and 'automatically' via running the task in Windows Scheduler.
Upon manual update tried allowing the XenServer Management Agent to update both Agent and Drivers - as well as just Agent: both setup attempts bluescreened.
-
@clip I will add that I'm experiencing same results with Windows 11 (no TPM used) and a Ryzen hypervisor. Both working and blue screening VMs have uefi boot firmware and each have the same 9.3.3 XenServer Tools updated on 6/7/24. No MS update issues pre 6/17/24
-
@olivierlambert
Could you please give advice on how to „reliably“ disable XenTools update for a running VM? Is it sufficient to set the registry keys? Is it sufficient to disable the scheduler job?
How can I safely disable all the updates until this is solved? -
@Mt_KEGan Confirming that on my end UEFI is what distinguishes my bluescreening VMs from non-bluescreening (BIOS) ones.
Correspondingly, hardware (Intel i7) may not be the deciding factor (at least for my pools).
-
I heard some XenServer users have the same issue (which is logical). We can expect XenServer is working on the issue. Meanwhile, best disable the Windows Update option.
-
We have had similar issues with an older version of the Citrix Xenserver tools (9.3.1) and several Windows 2022 servers. Windows 2k19 worked fine so after after a double reboot.
-
@stormi disabling that option won't let my VM boot. Just more BOD. I will continue trying things of course. My main concern is access to proper agents as the one referenced in XCP-ng docs seemed to have been developed and also abandoned in the year 2019! The other options have literally failed. This issue would frankly keep me from having confidence in the stack if I was to run in production. How could Citrix or XenServerProject let this happen? Including Vates, we're talking about a pretty mature system/dev stack at this point.
-
@Mt_KEGan You could try Windows System repair or reboot the system in safe boot mode. It solved the BOD for our systems yesterday. Win2k19 should behave similar to Win2k22 and show up repair mode & advanced boot options after one or two boot failures.
HTH..... -
@HolgiB thank you. Since my instances are just for homelab, I'm waiting for a solution from Citrix/XS/Vates. I can afford to be stubborn and expect the parties that failed to supply a solution. But again thank you, if this was production I would do so. Or just use my backups and migrate data to a working instance. I'm just really curious how this happened and want to test if a solution is provided and when.
-
Workaround for a fresh installation of Windows Server 2022:
I tried a fresh Installation of Windows 2022 Server. Windows Update installs the PV Storage Driver (XSC000 XENVIF 0) and requests a reboot.
If you open the device manager you will see:
Details:
Don´t reboot!!! If you reboot you get a BSOD.
You have to manually install 9.3.3 before reboot. Use default settings and select "Install I/O Drivers Now". This will correct the error.
You can check this in the Windows Device Manager.Now it is possible to reboot.
-
@Toni said in XenServer VM Tools 9.3.3 from Citrix causes bluescreen:
Workaround for a fresh installation of Windows Server 2022:
I tried a fresh Installation of Windows 2022 Server. Windows Update installs the PV Storage Driver (XSC000 XENVIF 0) and requests a reboot.
If you open the device manager you will see:
Details:
Don´t reboot!!! If you reboot you get a BSOD.
You have to manually install 9.3.3 before reboot. Use default settings and select "Install I/O Drivers Now". This will correct the error.
You can check this in the Windows Device Manager.Now it is possible to reboot.
Excuse me for people who don't speak what looks to be German (though I can't speak or understand it - but can recognise from other places which used similar words in a multi-lingual manual), please translate what the windows say.
