@Danp Only one pool has XOSTOR, the other one is just using NFS, but same behavior after upgrade.
@olivierlambert Thanks, will play around with xl command, didn't even know they exist, any docs available?

@Danp The xapi issue it self happened to two separate pools, one pool have two hosts, the other one have 3, non of the VMs had HA enabled. The pool patch was rolled via XO compiled from source(not XOA). One of the pool was already rebuild, I left the other one as is to investigate a bit, both are just for testing and validation. This happened in later April. It did appears that xcp-ng/xapi was trying to preform some type of database migration after the update but failing (I don't have the log readily available, but will look for it, it was something like XEN_INCOMPATIBLE error causing the toolstack to keep restarting)

Would you like me to start another thread on the DB corruption? I am not super crazy about recovering it and it might not be an XCP-ng issue, although the two pool have same behavior, they are running on different configuration, one has XOSTOR, but rather looking to see if it's possible or valuable to be able to preform lower level operation w/o xapi.
Many thanks for your time.

Greetings to the fine folks at XCP-ng community.
I recently encountered a scenario where potentially an update corrupted pool meta data (not super important, but will attach some details below for context), as an result xapi/xe command is no longer available to preform operations on VMs (VM power, console, migration etc.) while the VM is still running fine.
I am wonder if there is an established method to preform some basic VM operation on a even lower level without using 'xe' command or xapi toolstack. Perhaps directly interacting with QEMU or Xen Hypervisor? Or are there any other lower level recovery options available?

Many thanks in advance.

---

Some details about the pool meta data corruption.

XCP-ng 8.3.0
Platform 3.4.0
Version 24.11.0

After an xcp-ng update, xapi appears to stuck in some type of crash loop (xe command and XO will connect to the pool for a few second then unresponsive). XO will have connection reset error and xe command will hang or get a

'Connection refused (calling connect )'

error , but if i spam the same command, with the right timing, simple command like listing vms will work.
Can't make out anything meaningful from

/var/log/xensources.log

other than

xapi: message repeated 2 times: [ [error||0 |Registering SMAPIv1 plugins D:a8f0c896a50f|smint] SM.feature: unknown feature ATOMIC_PAUSE]

which doesn't help much, also had the pool meta data backup, but was not able to restore it due to version mismatch, will be playing around with it later perhaps start another thread if there is something interesting.