xo-server executable not found

[EddieA]

Had some time today, so went to update my home XCP-ng with the latest updates, on both the base and XOA. After rebooting I wasn't able to connect to the XOA appliance UI.

ssh'ing into XOA and poking around, I find this in the logs:

Jun 09 18:57:31 xoa systemd[1]: Started xo-server.service - XO Server.
Jun 09 18:57:31 xoa (o-server)[557]: xo-server.service: Failed to locate executable /usr/local/bin/xo-server: No such file or directory

I've looked through the update notes and don't see anything related. Did I miss something, or did my updates screw up somehow.

How do I recover this. I did find one write up on deleting the XOA appliance and reinstalling, but the delete fails with:

[17:15 xcp-ng ~]# xe vm-destroy uuid="3d80a504-3460-96ab-d438-2e6f2e9e0487"
You attempted an operation that was explicitly blocked (see the blocked_operations field of the given object).
ref: 3d80a504-3460-96ab-d438-2e6f2e9e0487 (XOA)
code: true
[17:15 xcp-ng ~]#

***** Update *****

Was able to delete and re-create the XOA appliance.

So now I'd just like to understand what happened.

Cheers.

[poddingue]

Glad you got it back up and running.
I'm not certain, but a missing /usr/local/bin/xo-server right after an XOA update usually looks like an update that got interrupted or only half-applied, so the service ends up pointing at a binary that never got laid down.
I think the gentler recovery before rebuilding would have been re-running the updater from the CLI; there's a writeup here: https://docs.xen-orchestra.com/xo5/updater#from-the-cli, which often repairs a broken update in place.
The vm-destroy refusal is expected by the way: XOA sets blocked_operations as a safety net so you can't accidentally destroy your own orchestrator, so that flag has to be cleared first. I could easily be wrong on the exact trigger, though, and if it ever happens again on a fresh update, @Team-XO-Backend would be the ones who could tell whether a specific update is implicated.

[EddieA]

@poddingue said:

usually looks like an update that got interrupted or only half-applied

Thinking back on it, I think that may be the issue. in that I was too quick off the mark rebooting after the base upgrades.

@poddingue said:

I think the gentler recovery before rebuilding would have been re-running the updater from the CLI

Kinda tried that, but:

[18:47 09] xoa@xoa:~$ xoa check
-bash: xoa: command not found
[18:47 09] xoa@xoa:~$ sudo xoa-updater --upgrade
[sudo] password for xoa:
sudo: xoa-updater: command not found
[18:48 09] xoa@xoa:~$

But regardless, I'm all good now.

Cheers.