Hosts have disappeared from XO Web UI

dsiminiuk

@Finallf I built a new minimalist Ubuntu 22 server and followed the same path and it does EXACTLY the same thing.

dsiminiuk

@olivierlambert
I tried to build from sources at the link you provided and there are instructions missing or things not obvious to me.
From /root I did git clone.
cd packages/xo-server
"Note: If you're installing xo-server as a global service, you may want to copy the file to /etc/xo-server/config.toml instead."
OK, I want to d othat so I cp sample.config.toml /etc/xo-server/ <- this directory does not exist
So I crated it and copied it there, but that is the only file there now.
Then I went back to /root/xen-orchestra/packages/xo-server
yarn start

yarn run v1.22.19
$ node dist/cli.mjs
2023-06-07T02:17:09.370Z xo:main INFO Configuration loaded.
2023-06-07T02:17:09.378Z xo:main INFO Web server listening on http://[::]:80
2023-06-07T02:17:09.379Z xo:main INFO User changed to root
2023-06-07T02:17:10.165Z xo:mixins:hooks WARN start failure {
  error: Error: spawn xenstore-read ENOENT
      at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
      at onErrorNT (node:internal/child_process:476:16)
      at processTicksAndRejections (node:internal/process/task_queues:82:21) {
    errno: -2,
    code: 'ENOENT',
    syscall: 'spawn xenstore-read',
    path: 'xenstore-read',
    spawnargs: [ 'vm' ],
    cmd: 'xenstore-read vm'
  }
}
2023-06-07T02:17:10.230Z xo:mixins:hooks WARN start failure {
  error: Error: Command failed with exit code 5: vgchange -an
    Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.
    Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)
      at makeError (file:///root/xen-orchestra/packages/xo-server/node_modules/execa/lib/error.js:59:11)
      at handlePromise (file:///root/xen-orchestra/packages/xo-server/node_modules/execa/index.js:124:26) {
    shortMessage: 'Command failed with exit code 5: vgchange -an',
    command: 'vgchange -an',
    escapedCommand: 'vgchange -an',
    exitCode: 5,
    signal: undefined,
    signalDescription: undefined,
    stdout: '',
    stderr: '  Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.\n' +
      `  Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)`,
    failed: true,
    timedOut: false,
    isCanceled: false,
    killed: false
  }
}
2023-06-07T02:17:10.427Z xo:main INFO Setting up / → /etc/xo/xo-web/dist/

When I went to the Web address I got
"Cannot GET /"

dsiminiuk

I've managed to get the build from sources to work and it does EXACTLY the same thing.
That excludes the XenServerInstallerUpdater as being the problem.

julien-f

@dsiminiuk If you cannot pinpoint the problematic commit, you can deploy an official XOA, then open a support tunnel and we'll be able to check it out directly.

dsiminiuk

@julien-f I tried that but got "An error occurred", but I saw at my host's address (after accepting the SSL certificate alert a directed) that it could be deployed from right there so I tried that and got another error.

If you can give me the master commits from newest to oldest over the last two weeks I can try those one at a time. I have no idea how to find them.

olivierlambert

As @julien-f told you, see https://xcp-ng.org/forum/topic/6940/xo-task-watcher-issue-cr-broken/55?_=1686148304910

dsiminiuk

@olivierlambert I will try that on the new VM I built last night since it is built from sources per your doco. I could not use that for the xo deployed by ronivay's script.

Additional observation: The behavior I see when there is one host online and another offline (web page freezes when clicking the Hosts icon from the Pool view) does NOT happen when both hosts are online (perhaps that there are no hosts offline). This is a clue as to the source of the issue.

Am I really the only user seeing this?

dsiminiuk

@olivierlambert I've done rebuilds going back to f77675a8a3590668413ec5bda02030c34b43408a and still the problem persists. I'm suspecting something wrong with the pool master now.

dsiminiuk

There is definitely something wrong with my 2nd host. It says it is not a member of a pool and the management interface is gone and does not persist even after an Emergency Network reset.

Time to rebuild that machine and I'll report back about XO.

dsiminiuk

@olivierlambert @Finallf @julien-f After a rebuild of my 2nd host everything is fine now. It must have become corrupted during the last rolling pool update.
Please accept my apologies for the fire drill.

Ooopsie! Right up until I powered off the 2nd host and now no hosts are showing and the page freezes again.

Oh well. At least I had a chance to swap out spinning hard disks for SSDs during the rebuild of host 2.

dsiminiuk

@olivierlambert I moved my pool master to the 2nd host, then rebuilt the 1st, so now both are booting on SSD. Both are updated to latest patches, and still I cannot deploy the XOA appliance over the web, I get the same error as above.

"message": "INVALID_VALUE",

I'm going to stop complaining. I can manage my VMs. Hopefully the issue will appear elsewhere and it can be exposed.

In the meantime I am writing a script on the xo VM with nut-client and xo-cli to shutdown VMs and hosts on power fail, since enabling the epel-release repo is now blocked on XCP-NG.

dsiminiuk

@olivierlambert as of today the build done on commit 3688e762b (or perhaps the previous, can't be sure) has resolved this issue. I can now see all hosts, online or when offline, and clicking on the hosts from the Pool view takes me to the Hosts list and no longer hangs the page.

julien-f

@dsiminiuk I don't it's related to XO itself, there has been no changes regarding this.

dsiminiuk

@julien-f @julien-f All I have done is let ronivay's script do the builds daily on schedule if there is a new commit on the master branch. His script has not changed. There have been no new patches to the hosts. I don't have an explanation.