@Kreeblah Not related to your i40e issue but when you are running the standard kernel, before the system shuts down due to a thermal shutdown do you see anything when running the command "xl dmesg" from ssh? I have seen xen thermal throttle cpu cores there and report when it happens. On some SFF systems (Minisforum MS-01) i have simply adjusted the boost clocks (which you can do from inside xcp-ng) to stop it from happening. Though you mention you are not going over 60C which should not trigger any throttling.

I believe this was a bug that was fixed in later versions:
https://support.citrix.com/support-home/kbsearch/article?articleNumber=CTX678047

@Pilow You can watch it via the cli using Journal using this command. Not as nice as having it in the UI but useful if you are waiting for a merge for finish before running maintenance on your remotes.

sudo journalctl -u xo-server -f -n 50

@dinhngtu Thats great news!

I had a job what was interrupted for some reason during its backup run, upon retrying a full backup was required (expected) however the UI shows contradicting information about the VMs backup job.

I think ideally it should be clarified that a Delta is not running but instead a full? (At the bottom)

@dinhngtu Well that is slightly concerning! Ill be sure to not remove Xen Tools on any of these VMs until we can get this resolved. We have a handful of production server 2025 VMs that im now slightly worried about! I should note that last week we did update them to version 9.4.2 of the tools, which tends to require 2 reboots to fully install and we didn't run into any issues.

@dinhngtu Very strange! I wonder why installing from an older ISO and then installing all the updates has no issues at all? Im relieved to know that our current 2025 VMs wont stop working after a reboot!

@McHenry No the pool master must always been patched and rebooted first. Do you have a pool metadata backup? Are your VMs on shared storage of some sort? In case you need to rebuild the pool.

@McHenry I am wondering if you are in a situation where you need to reboot. You have patches installed but have not rebooted, but have restarted tool stacks on the salves, which means some components have restarted and are running on their new versions? If you have support it may be best to reach out to Vates for guidance.

@McHenry Have you tried restarting the toolstack on the hosts since running pool-recover-slave?

I have only had to do this once before but remember it going fairly smoothly at the time. (As smooth as you can expect a host failure to be anyways)

@McHenry

Yes you'd run that command on the pool master.

I linked this earlier in the thread, it outlines the process you need to follow:
https://docs.xenserver.com/en-us/xenserver/8/dr/machine-failures.html

Let us know if this works!

@McHenry did you run

xe pool-recover-slave

after selecting a new master?

@McHenry No moving the pool master shouldn't result in any config loss.

@McHenry i think that us your best bet, you can also leave the new master as the master, and the rebuilt host as a salve. Did the other hosts all update ok? Kind of concerning that running updates resulted in an unbootable host.

@McHenry i think the command you need to run on a current slave is

xe pool-emergency-transition-to-master
followed by
xe-toolstack-restart

This will make that slave the new pool master. You should only do this though if the current pool master for sure dead.

This may also be useful:
https://docs.xenserver.com/en-us/xenserver/8/dr/machine-failures.html

I can confirm this happens when running a clean install from SW_DVD9_Win_Server_STD_CORE_2025_24H2.11_64Bit_English_DC_STD_MLF_X24-14362.ISO

xl dmesg shows:

(XEN) [1424074.898990] d25v1 VIRIDIAN GUEST_CRASH: 0x7b 0xffff970394606848 0xffffffffc0000034 0 0x1

Disabling Viridian has no effect.

Interestingly a Windows 11 install using SW_DVD9_Win_Pro_11_24H2.11_64BIT_English_Pro_Ent_EDU_N_MLF_X24-14245.ISO works totally fine.

@gduperrey Updated my usual test hosts, (Minisforum and Supermicro X11) as well as an two sets of 2 host AMD pools (one pool of HP DL320 Gen10s and another of Asus Epyc servers of some sort, and lastly a Dell R360 without issue.

@gduperrey Installed on about 50 servers across various pools and remote sites. No issues. Ran a couple backup jobs as well which completed without issue.

@olivierlambert On my test instance which is admittedly not very busy (only 2 hosts) this fixed the issue!

@marcoi Same issue here with the latest commit (c1c3b). Rolling back to 4e3c1 from 2025-08-06 which i was running previously resolves the issue. Not sure whats going on or when it started happening exactly as i usually only update once a month.

Edit: Updated to commit 4eeae and also do not experience the task spam. So im guessing this is something that happened during the commits today (Aug 25, 2025)

Edit 2: Tested on both my homelab install and a second test environment install and both seem to exhibit this issue on the latest commit.