@dinhngtu

can you please confirm this is as expected.

Started a new Windows 2019 Eval edition (what I had handy, I assume 2022 is similar)

installed XCP-ng 8.2.2 open source guest tools.

ran windows update to get windows up to date.

Used the xenclean script to remove XCP-ng Tools tools.

Installed Citirx guest tools : Management agent, drivers version 9.4.0; told to NOT update drivers through agent (installed the older one, so as to see it update on its own)

set registry value Autoreboot to 3.
shut down
Set VM to use Windows Update in XO advanced settings.
Started VM

Ran windows update, it found latest drivers. and then -I- rebooted when told to.
Windows booted as expected. Shortly after -it- rebooted again once more
(the '3' is a Max reboot?)

Control panel, software and features still say Xen Tools 9.4.0 (old), but the individual drivers:
XenServer PV Network Device: 9.1.7.65 (old, and current)
XenServer PV Storage Host Adapter 9.1.8.79 (old)
XenServer PV Bus 9.19.105 (old)
XenServer PV Bus (C000) 9.1.11.115 (current)
XenServer PV Network Class 9.1.2.101 (old)

Will have to think more on this. We do have a a domian policy for Windows Updates, that mostly works; it never seems to work as intended. In any case, if a Citrix driver is updated, with the registry setting set, it will reboot x number of times? (3 is suggested I believe)
Wish there was a bomb proof method. Just this morning I came to office to find half the building down, because MS decided to push a Citrix update last night, and the DHCP server was down, until I rebooted the VM from the console once more. ;(

@dinhngtu Excuse the dumb questions
When we use the XCP-ng tools, 8.2.2, There are no unexpected reboots, I -thought- because they weren't updating by themselves, and have had zero problems. We have a ISO with these, that we load on most of out VMs, and never have problems.
Others, for whatever reason,. (admin-user forgets the Get drivers from Windows Update setting, etc), reboot unexpectedly, and on occasion are down, like I mentioned earlier, until someone logs into console and set up new ethernet interface in the VM.

Im thinking of what to do in our environments, but currently leaning on letting Windows Update do it, and setting the reboot setting to 3 or so. (not ideal, we have 1 particular host that is very i/o deficient) We'd be rebooting for Updates anyway, and a real machine would get driver updates then anyway. Can you confirm this might be a good way to do it, or some other suggestion. (that I'd need to use to form my own ideas, for us )
Or hold out for the new XCP-ng drivers. Not sure how long I should do that though.

@dinhngtu Couple more questions I thought of.

1. When the setting in XO "get drivers from Windows Update", they are only updated when something is published from Citrix, -and- my normal Windows update procedure takes place? Otherwise, probably the same as what ever i'd installed from a Citrix ISO, or installer, and -that- would have its own update schedule separate from Windows Update?

2. Presumbly when the XCP-ng guest tools/ device drivers are finally signed (any time frame on that?) they would NOT be getting updates like the previous 8.2.2? Or are they now going to update on some schedule?

thanks.

removed the failed server connection.

Restarted xo-server service.

Added new host again. Seems fine now.

In hind sight, I could have very well recycled a DHCP ip here as well, if that is a known issue...

found a similar machine that was testing some XCP-ng 8.3 stuff on it.
Re-installed 8.2
Now can't add to XO at all. All I can find for people with this error is adding a host thats already in a pool. In this case, there are no pools.

hrm.

server.enable
{
  "id": "a6cce2cd-f3c8-[redacted]-2bfce24a6d96"
}
{
  "poolId": "b5a4db02-f996-[redacted]-7ed3cad86788",
  "connectedServerId": "f1f0aa28-f820-[redacted]-7c15c1686461",
  "connectingServerId": "a6cce2cd-f3c8[redacted]-2bfce24a6d96",
  "message": "this pool is already connected",
  "name": "PoolAlreadyConnected",
  "stack": "PoolAlreadyConnected: this pool is already connected
    at XenServers.connectXenServer (file:///opt/xo/xo-builds/xen-orchestra-202505021422/packages/xo-server/src/xo-mixins/xen-servers.mjs:358:15)
    at Xo.enable (file:///opt/xo/xo-builds/xen-orchestra-202505021422/packages/xo-server/src/api/server.mjs:121:3)
    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202505021422/@vates/task/index.js:175:22)
    at Task.run (/opt/xo/xo-builds/xen-orchestra-202505021422/@vates/task/index.js:159:20)
    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202505021422/packages/xo-server/src/xo-mixins/api.mjs:469:18)"
}

A couple Windows 2016, and 2019, and a couple Rocky Linux 8 is what Im moving.

I'll see if I have an 8.2 host suited (not too different CPU) to be a middle man.

thanks.

I'll post the error below. Im using XO Xen Orchestra, commit 80e72
Master, commit 48603, from sources which I update form time to time.

Im trying to migrate sme VMs from XCP 7.6 to a different host that has 8.3 on it. Keep getting errors. I can migrate from a XCP 8.2 to the same destination I want, in the error. And from the 7.6 to 8.2.

is 7.6 and 8.3 too big a jump for XO? or am I a little far old and need an update? And XCP Center doesn't connect to XCP 8.3

xo_error.txt

Hello.,
Some backstory:
Most of out VMs us the 8.2.2. XCP-ng guest tools in Windows. These have been rock solid, probably because there are not updates for it.

Some VMs for whatever reason, depending on the method, and what the Human Admin is used to, end up with the Citrix device drivers. We've noticed that when this happens, and they go to update, it really hoses the VM. Its constantly rebooting several times, and needs a user to log in, get the message that it needs a reboot, and then reboot, at least a couple times. Often its completely offline, and an Admin needs to get on the VPN, log in to XO or XCP-center, and go to the VM console, and reboot, and or put the IP information back in to a new virtual ethernet interface.

In fact this happened last night, despite the Scheduled Task in windows set to disable, and the "get from Windows Update" set to OFF, the Citrix Guest drivers needed an update, and the VM was offline for a while. (the problem being this particular VM is required to be up for the VPN to work. So an Admin had to come to office, and get into the VM console. and reboot several times.)

1. What might we be missing? why do the Citrix drivers behave so badly at updates?

2. I see that there are recent updates to the XCP-ng drivers, that are here, and or coming, depending on MS. Thus this brings up my next question, and reason for reaching out.

3. Im hesitant to migrate to the Citrix drivers because of this. (Or even the newer XCP-ng drivers, if they are going to update the same way) The last thing I need is an Human Admin babysitting 130 VMs every patch Tuesday, or more. Especially when the other Admin is a staunch Hyper-V fan

Any suggestions would be helpful.
thanks.

.51 for Rocky showed up today. Running my first test. and from .22.

Seems successful!

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

This may sound stupid, but here goes...

Does Rocky need to update to .50 before it can update to .51?

I'd guess no, but I've been away from RHEL based stuff since the IBM rug pull. Only coming back because I may need it in the future as I jockey around trying to improve my paycheck.

https://bugs.rockylinux.org/view.php?id=9307
so, prolly soon-ish.

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@olivierlambert

I just updated an Alma 8 and it has the .51 kernel now.

Tried one of my Rocky's and still .50. I wonder where we can see the status of downstream fixes like that? anyone know? thanks.

@Greg_E
I made a free account.
Resolution
The issue has been resolved with the errata: RHBA-2025:4337. Hence, update the kernel to kernel-4.18.0-553.51.1.el8_10 to fix the issue.

id assume it arrives soon? I as well, haven't seen it yet.

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

Ok, that might explain the difference.

Would a LEAPP from Rocky 8 up to Alma 9 be possible and solve the issue?

I did a (not LEAPP, but a migration script from Alama) from Rocky 8 to Alma 8, and it died. None of my Rocky 9's have had a problem so far, and of course end up with a completely different kernel.

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

If I find an hour, I'll give Rocky 8 a try.

Could you use LEAPP to migrate that to Alma, maybe they are doing something differently which is why mine are working.

I can tell you, there is nothing special that I'm doing, my systems are as vanilla as they get.

@Greg_E
google AI says what I think is familiar:
Rocky Linux strives for 1:1 bug-for-bug compatibility with RHEL, while Alma Linux is more of an RHEL rebuild, making some adjustments and adding its own features

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

Is there something different about Alma? Friday I may have some time to fiddle and can try a Rocky 8 to see what happens.

Is this an 8.2 and 8.3 issue or just 8.2? I have 8.2 in production and could try there, 8.3 in my lab with a very fresh build.

Migrated to a XCP-ng 8.3 host. Xeon E5-2689 v4
No change.

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt that's why I mentioned uefi, wondering if legacy is part of the problem. I won't have time to fiddle with this for a while, broke a couple things today that I need to fix, and need to set up glpi for some testing.

Made a new Rocky Linux 8 install, on a lab host. UEFI boot mode, and mostly all defaults. on XCP-ng 8.2 on a E5 2620 v0 host.
Used the guest tool from the Rocky and or EPEL repository. (added the EPEL repo and then installed xe-guest-utilities)
Does NOT boot after updating to the latest kernel.

@Greg_E
I checked a few of mine, and they appear to all be BIOS mode. not UEFI.
I've had a couple hardware machines as well, that updated OK. I know at least one was UEFI.

@olivierlambert @kpark

Can you please let me know where this bug is filed (I see some numbers mentioned above), so I might keep an eye on it for my own curiosity. A PM is fine as well.

Thanks!

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt

Not to go off topic, but lack of v2 and lack of uefi were reasons I moved my lab to newer hardware.

@Greg_E you're not wrong, but I don't get to hold the checkbook.