Migrating from XCP-ng Windows guest tools to Citrix
-
Hello.,
Some backstory:
Most of out VMs us the 8.2.2. XCP-ng guest tools in Windows. These have been rock solid, probably because there are not updates for it.Some VMs for whatever reason, depending on the method, and what the Human Admin is used to, end up with the Citrix device drivers. We've noticed that when this happens, and they go to update, it really hoses the VM. Its constantly rebooting several times, and needs a user to log in, get the message that it needs a reboot, and then reboot, at least a couple times. Often its completely offline, and an Admin needs to get on the VPN, log in to XO or XCP-center, and go to the VM console, and reboot, and or put the IP information back in to a new virtual ethernet interface.
In fact this happened last night, despite the Scheduled Task in windows set to disable, and the "get from Windows Update" set to OFF, the Citrix Guest drivers needed an update, and the VM was offline for a while. (the problem being this particular VM is required to be up for the VPN to work. So an Admin had to come to office, and get into the VM console. and reboot several times.)
-
What might we be missing? why do the Citrix drivers behave so badly at updates?
-
I see that there are recent updates to the XCP-ng drivers, that are here, and or coming, depending on MS. Thus this brings up my next question, and reason for reaching out.
-
Im hesitant to migrate to the Citrix drivers because of this. (Or even the newer XCP-ng drivers, if they are going to update the same way) The last thing I need is an Human Admin babysitting 130 VMs every patch Tuesday, or more. Especially when the other Admin is a staunch Hyper-V fan
Any suggestions would be helpful.
thanks. -
-
Hi!
Question for @dinhngtu when he will be around
-
@bberndt Both the Citrix and XCP-ng Windows drivers require several reboots to upgrade because they are stateful and need to keep track of changes in the bus driver. They do have an "autoreboot" feature that will take care of the reboots by themselves based on a registry value. You can refer to our update guide for more details. Though I do hope to find a way to reduce excessive reboots eventually.
Often its completely offline, and an Admin needs to get on the VPN, log in to XO or XCP-center, and go to the VM console, and reboot, and or put the IP information back in to a new virtual ethernet interface.
Static IP loss was a bug and shouldn't happen any more with the new Citrix drivers.
-
@dinhngtu Couple more questions I thought of.
-
When the setting in XO "get drivers from Windows Update", they are only updated when something is published from Citrix, -and- my normal Windows update procedure takes place? Otherwise, probably the same as what ever i'd installed from a Citrix ISO, or installer, and -that- would have its own update schedule separate from Windows Update?
-
Presumbly when the XCP-ng guest tools/ device drivers are finally signed (any time frame on that?) they would NOT be getting updates like the previous 8.2.2? Or are they now going to update on some schedule?
thanks.
-
-
- So there are 2 parts to be aware of:
- The PV I/O drivers which are pulled from/updated by Windows Update;
- The management agent which you get from XenServer. This has its own update platform that you can enable at install time.
Confusingly the XenServer management agent also has an option to install PV I/O drivers by itself without going through Windows Update at all. (This is why they state "Customers using Windows Update for I/O driver updates should not select this option)
- I'm not sure what you mean by not getting updates? Even our unsigned drivers do get updates, bug fixes etc. and this will continue once a signed version comes out.
For the final signed version, at the moment we're still waiting for authorization from Microsoft (they work extremely slowly and we've run into all sorts of problems with their developer program)
-
@dinhngtu Excuse the dumb questions
When we use the XCP-ng tools, 8.2.2, There are no unexpected reboots, I -thought- because they weren't updating by themselves, and have had zero problems. We have a ISO with these, that we load on most of out VMs, and never have problems.
Others, for whatever reason,. (admin-user forgets the Get drivers from Windows Update setting, etc), reboot unexpectedly, and on occasion are down, like I mentioned earlier, until someone logs into console and set up new ethernet interface in the VM.Im thinking of what to do in our environments, but currently leaning on letting Windows Update do it, and setting the reboot setting to 3 or so. (not ideal, we have 1 particular host that is very i/o deficient) We'd be rebooting for Updates anyway, and a real machine would get driver updates then anyway. Can you confirm this might be a good way to do it, or some other suggestion. (that I'd need to use to form my own ideas, for us
)
Or hold out for the new XCP-ng drivers. Not sure how long I should do that though. -
@bberndt Autoreboot works, but one issue is that you don't have policy-based control over when it reboots (unlike Windows Update). If that's important for you I'd recommend applying the guest agent+driver updates during a dedicated patch window, and setting the VMs to autoreboot only during that window.
I'd very much appreciate having people use the new XCP-ng drivers (which are stable & reliable despite the current lack of Microsoft signatures, and come with a few nice extras) but I get that it's not always possible. Without the necessary Microsoft hardware accounts it's hard to give a concrete timeline, but I'd like to believe it'll happen soon.
-
Will have to think more on this. We do have a a domian policy for Windows Updates, that mostly works; it never seems to work as intended. In any case, if a Citrix driver is updated, with the registry setting set, it will reboot x number of times? (3 is suggested I believe)
Wish there was a bomb proof method. Just this morning I came to office to find half the building down, because MS decided to push a Citrix update last night, and the DHCP server was down, until I rebooted the VM from the console once more. ;( -
can you please confirm this is as expected.
Started a new Windows 2019 Eval edition (what I had handy, I assume 2022 is similar)
installed XCP-ng 8.2.2 open source guest tools.
ran windows update to get windows up to date.
Used the xenclean script to remove XCP-ng Tools tools.
Installed Citirx guest tools : Management agent, drivers version 9.4.0; told to NOT update drivers through agent (installed the older one, so as to see it update on its own)
set registry value Autoreboot to 3.
shut down
Set VM to use Windows Update in XO advanced settings.
Started VMRan windows update, it found latest drivers. and then -I- rebooted when told to.
Windows booted as expected. Shortly after -it- rebooted again once more
(the '3' is a Max reboot?)Control panel, software and features still say Xen Tools 9.4.0 (old), but the individual drivers:
XenServer PV Network Device: 9.1.7.65 (old, and current)
XenServer PV Storage Host Adapter 9.1.8.79 (old)
XenServer PV Bus 9.19.105 (old)
XenServer PV Bus (C000) 9.1.11.115 (current)
XenServer PV Network Class 9.1.2.101 (old) -
@bberndt Things sound OK to me. Are you looking for an update procedure?
With Windows Update, some drivers won't be installed automatically but will be available in Optional updates instead.
-
@dinhngtu We have a Domain Policy that is intended to update and reboot on a set schedule. I didn't see more drivers in Optional Updates. I don't see Optional Drivers at all, as a section in the Windows Updates panel, maybe because its done in the Policy. I -have- seen this in Windows 10/11.
-
@bberndt said in Migrating from XCP-ng Windows guest tools to Citrix:
XenServer PV Bus 9.19.105 (old)
Could you confirm which driver/device this was? It didn't look right.
Also, are you pulling updates through WSUS or something similar? Normally Windows should be able to update all the PV drivers on its own.Otherwise, in a fresh Windows installation the drivers gotten from Windows Update are as following:
- Bus 9.1.11.115
- Interface 9.1.12.94
- Network Class 9.1.13.107
- Network Device 9.1.7.65 (old, but normal; there's been no updates to this driver since)
- Storage 9.1.9.82
Rolling back to older drivers then running Windows Update got me the newest drivers as expected.
-
@dinhngtu said in Migrating from XCP-ng Windows guest tools to Citrix:
XenServer PV Bus 9.19.105
Notice the other XenServer PV Bus (C000), this one is the current version compared to Citrix release page. Another test server I set up, and installed the latest Citrix installer 9.4.1, and it has a Xenserver PV Bus (002), and one (C000). We are not using WSUS at all. This is on Server 2019.
-
@bberndt It's normal to have 2 PV Bus devices if you have the "driver via Windows Update" VM option enabled, but I don't expect them to have different driver versions. Is the C000 device the active one? (you can see with View - Devices by connection) Maybe the non-C000 device didn't get the update due to it not having an update-tagged device ID.
-
@dinhngtu
Tried to get everyone in the same screen shot here.the 'older' one has the subtree under it,.
-
@bberndt That's really odd. What does Update Driver say? I'll try to ask the XS driver maintainer about it.
-
@dinhngtu
says none found, and asks to go to windows update, which says im up to date. Further up in this thread, I tried to recall my exact procedure. I suppose I can't say the group policy isn't doing something, but it just setting the date and time (wednesdays) to run once a week, and the deadline is to reboot Saturdays, 3 days later. -
@bberndt Okay, I managed to reproduce your situation. I think it's because the "driver via Windows Update" option was enabled after installing the XS drivers, which caused the drivers to lock onto the non-C000 device and prevent updates from coming in.
Normally, XenClean should be able to fix the situation. But if you want to fix things manually, or if things still don't work (C000 is still not active), here's a procedure that should fix the problem:
- Take a snapshot/backup/etc.
- Keep a note of static IP addresses (if you have any; there's a chance those will be lost). You can also use our script here: https://github.com/xcp-ng/win-pv-drivers/blob/xcp-ng-9.1/XenDriverUtils/Copy-XenVifSettings.ps1
- Reboot in safe mode and disable the non-C000 device.
- Reboot back to normal mode; it'll ask you to reboot a few more times.
- The C000 device should now be active and you should be able to get driver updates again.
- (Optional) You can now enable and manually update the non-C000 device (Browse my computer - Let me pick).
-
Updated with working procedure. (tl;dr XenClean should fix this)
-
I did it that way so as to get the old Citrix driver first, and then let it update and watch it reboot. That was my logic anyway.
@dinhngtu said in Migrating from XCP-ng Windows guest tools to Citrix:
@bberndt Okay, I managed to reproduce your situation. I think it's because the "driver via Windows Update" option was enabled after installing the XS drivers, which caused the drivers to lock onto the non-C000 device and prevent updates from coming in.
Normally, XenClean should be able to fix the situation. But if you want to fix things manually, or if things still don't work (C000 is still not active), here's a procedure that should fix the problem:
- Take a snapshot/backup/etc.
- Keep a note of static IP addresses (if you have any; there's a chance those will be lost). You can also use our script here: https://github.com/xcp-ng/win-pv-drivers/blob/xcp-ng-9.1/XenDriverUtils/Copy-XenVifSettings.ps1
- Reboot in safe mode and disable the non-C000 device.
- Reboot back to normal mode; it'll ask you to reboot a few more times.
- The C000 device should now be active and you should be able to get driver updates again.
- (Optional) You can now enable and manually update the non-C000 device (Browse my computer - Let me pick).
