@MasterOSkillio Typically, C-states need to be changed in the BIOS. In some cases, it can be very helpful. I wrote a few blogs including this topic entitled "A Tale of Two Servers" but cannot readily find them on-line at the moment. Alas, Citrix has purged a lot of still relevant older content over the years.

@RS One option would be this, assuming in this se you want to run the job at midnight on Dec. 25:
/bin/echo "/path/to/your/script.sh" | at midnight Dec 25

While cron doesn't offer a specific one-time execution, you could also do this in cron but would have to remove the entry afterwards:
0 0 25 12 * /path/to/your/script.sh

Also, take a look at this option: https://www.fastcron.com/guides/one-time-cronjobs/

@DKirk That all makes sense, thanks for clarifying. Looks like there are further comments below that seem to pinpoint where the issue may lay. The key point you make is only "after the last updates" is when this started happening!

@DKirk Very odd. Maybe a electrical power issue? Do you see this if you run xentop on each host and really important, do they happen at the same time on all your servers?
Any chance they are overheating and pausing briefly?

@olivierlambert Agreed. The Citrix forum used to be very active, but especially since Citrix was taken over, https://community.citrix.com has had way less activity, sadly.
It's still gratifying that a lot of the functionality still is common to both platforms, although as XCP-ng evolves, there will be continually less commonality.

@Chrome Cheers -- always glad to help out. I put in many thousands of posts on the old Citrix XenServer site, and am happy to share whatever knowledge I still have, as long as it's still relevant! In a few years, it probably won't be, so carpe diem!

@Chrome Fantastic! Please mark my post as helpful if you found it as such. Was traveling much of today, hence the late response.

BTW, it's always good to make a backup and/or archive of your LVM configuration anytime you change it, as the restore option is the cleanest way to deal with connectivity issues if there is some sort of corruption. It's saved my rear end before, I can assure you!

Yeah, if the SSD drive got wiped, there's no option to get those back unless you made a backup somewhere of all that before you installed XCP-ng onto it.

BTW, another very useful command for LVM is "vgchange -ay" which will attempt to renew VG information if a VG seems missing or the like.

@Chrome As M. Lambert says, you may be able to sue pbd-plug to re-attach the SR if you can sr-introduce the old SR back into the system.
If not, and if your LVM configuration has not been wiped out, here are some steps t try to recover it (it's an ugly process!):

1. Identify the LVM configuration:
   Check for Backups: Look for LVM metadata backups in /etc/lvm/archive/ or /etc/lvm/backup/.
   Use vgscan: This command will search for volume groups and their metadata.
   Use pvscan: This command will scan for physical volumes.
   Use lvs: This command will list logical volumes and their status.
   Use vgs: This command will list volume groups.
2. Restore from Backup (if available):
   Find the Backup: Locate the LVM metadata backup file (e.g., /etc/lvm/backup/<vg_name>).
   Boot into Rescue Mode: If you're unable to access the system, boot into a rescue environment.
   Restore Metadata: Use vgcfgrestore to restore the LVM configuration.
3. Recreate LVM Configuration (if no backup):
   Identify PVs: Use pvscan to list available physical volumes.
   Identify VGs: Use vgscan to identify volume groups if they are present.
   Recreate PVs: If necessary, use pvcreate to create physical volumes.
   Create VGs: If necessary, use vgcreate to create a new volume group.
   Create LVs: If necessary, use lvcreate to create logical volumes.
4. Mount and Verify:
   Mount the Logical Volumes: Mount the restored LVM volumes to their respective mount points.
   Verify Data: Check the integrity of the data on the restored LVM volumes.
5. Extend LVM (if adding capacity):
   Add a new disk: Ensure the new disk is recognized by the system.
   Create PV: Use pvcreate on the new disk.
   Add PV to VG: Use vgextend to add the PV to the volume group.
   Extend LV: Use lvextend to extend the size of an existing logical volume.
   Extend Filesystem: Use e2resize (for ext4) or resize2fs (for ext3) to extend the filesystem on the LV.

@Chrome Do then just a "xe vm-list" and see if you recogniize any VMs other than the dom0 instance of XCP-ng.
If there is nothing else showing up, you will need to try to find your other LVM storage.

@Chrome Try "xe vm-list params=all"
Do you only have a local storage or did you have any attached storage that's not showing up?

@Chrome OK, now try "xe sr-introduce" (check the syntax for the full command syntax you need), depending on what your connection type is:

xe sr-introduce uuid=<device uuid> shared=true type=lvmohba name-label=<name>

xe sr-introduce uuid=<device uuid> shared=true type=lvmoiscsi name-label=<name>

xe sr-introduce uuid=<device uuid> shared=true type=nfs name-label=<name>

If you are lucky and the interface still exists by which the SR was attached, that might do the trick.

No metadata backup? All that info should be contained within. Otherwise, it could be a long, painful process. Are there LVM drives or how were they initially created?
I assume "xe sr-list" shows nothing or?

@Andrew Right, backups should be shut off during the RPU process.

@manilx If the VMs can be shut down, yes, otherwise migrate the VMs. Luckily, you can migrate from a host with a lower hotfix level to one that has a higher level, but I do not believe the reverse is possible.

@ecoutinho Performing manual updates is an option. The master probably checked the hosts for the hotfix uniformity before the rolling pool upgrade started, but since it failed to complete, you now have a discrepancy in the patch level for those two hosts. I had that happen once because of a root space error, which was a pain to deal with, and though I cannot recall the specific fix, I think I had to migrate the VMs to the updated hosts and do a whole new install after redoing the partition table (it was that dreaded extra "Dell" partition at the time that caused the issue).

@AlbertK Running a SQL instance on a VM has been done for many years with success. This article may help shed some light on the subject:
https://www.varonis.com/blog/sql-server-best-practices-in-virtualized-environments

@syscon-chrisl Note that when changing the management interface, it's highly recommended to reduce the PMI down to just one NIC on all your hosts before you make the change. That said, it's always a scary thing to do and as others have stated, best avoided if at all possible. Making sure all hosts are at the same hotfix levelasand that their NICs are all in the same order and speeds are essential requirements.

@yeopil21 Run top, xentop, and iostat to see if dom0 and.or the storage device might be a bottleneck. The configuration of your storage device can also be a big factor,
and in some casaes, various performance tweaks are possible. The specific configuration would be helpful: connectivity (NFS, fibre channel, iSCSI), number and size and speeds of disks, RAID configuration, total number of VMs resident on the device, number of independent SRs, provisioning (thin or thick), network (if not fibre channel) speed and settings.
STorage optimization is a bit of an art and in many cases, can be the limiting factor, but as stated, so can the lack of dom0 resources. Also, what clock speed and number of CPUs are on your hosts?

@olivierlambert I guess migrating the VM to a different host would not kick in the changes, either, since the VM is technically not shut down in the process?

@McHenry Am wondering if defragmenting the drives might help, at least some, if nothing else perhaps slightly better I/O performance?