Hosts have disappeared from XO Web UI

Gheppy

I use this steps to build XOCE, starting from clean debian with only ssh

#Connect to ssh and change to root account, or use sudo in front of each command

Change Ip on LAN
nano /etc/network/interfaces

iface eth0 inet static
address 192.168.2.10
netmask 255.255.255.0
gateway 192.168.2.1
dns-nameservers 8.8.8.8 8.8.4.4

systemctl restart networking

#Update and some utility
apt update && apt upgrade -y && apt autoremove -y
apt --yes install git curl apt-transport-https gnupg mc wget net-tools

#Install node
curl -fsSL https://deb.nodesource.com/setup_18.x | bash - &&
apt install -y nodejs
node -v

#Install yarn
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
apt update
apt install -y yarn
yarn --version

#Aditional packages
apt install -y build-essential redis-server libpng-dev git python3-minimal libvhdi-utils lvm2 cifs-utils nfs-common
npm install -g n

#Install XOCE
cd /opt/

#Clone git and build XOCE
git clone -b master https://github.com/vatesfr/xen-orchestra
cd xen-orchestra
yarn
yarn build
yarn cache clean
cp /opt/xen-orchestra/packages/xo-server/sample.config.toml /opt/xen-orchestra/packages/xo-server/.xo-server.toml

#Activate all plug-ins
cd /opt
mkdir -p /usr/local/lib/node_modules/

cat > /opt/plugins.sh <<EOF
#!/bin/bash

dest='/usr/local/lib/node_modules'
cale='/opt'
ignoreplugins=("xo-server-test")

echo "Cleaning plugin-urile..."
find \$dest/xo-server-* -xtype l -delete

for plugin in "\${ignoreplugins[@]}"; do
  if [ -L \$dest/\$plugin ]; then
    echo "Removing link for \$plugin"
    rm \$dest/\$plugin
  fi
done

echo "Cleaning plugin-urile..."
for source in \$(ls -d \$cale/xen-orchestra/packages/xo-server-*); do
  plugin=\$(basename \$source)
  if [[ "\${ignoreplugins[@]}" =~ \$plugin ]]; then
    echo "Ignoring \$plugin plugin"
  elif [ ! -L \$dest\$plugin ];  then
    echo "Delete old link \$plugin from \$dest/\$plugin"
    rm \$dest/\$plugin
    echo "Make link for \$plugin on \$dest/\$plugin"
    ln -s "\$source" "\$dest"
  fi
done

  
EOF

chmod 755 /opt/plugins.sh
/opt/plugins.sh

#XOCE in service mode

cat > /lib/systemd/system/xo-server.service <<EOF
#Systemd service for XO-Server.

[Unit]
Description= XO Server
After=network-online.target

[Service]
WorkingDirectory=/opt/xen-orchestra/packages/xo-server/
ExecStart=/usr/bin/node ./dist/cli.mjs

Restart=always
SyslogIdentifier=xo-server

[Install]
WantedBy=multi-user.target

EOF

systemctl daemon-reload
systemctl enable xo-server

dsiminiuk

Something else Hosts related...
When I click on the Pools list, I see my pool. I click on it and it shows me the 3 icons. The 2x server icon, 3x Storage SRs and 10x VMs.

if I click on the Storage Icon it takes me to the storage list http://xo/#/home?p=1&s=&t=SR
if I click on the VM icon it takes me to the VM list http://xo/#/home?s=$pool:ac9585b6-39ac-.....c082b
But if I click on the Server icon I expect it to take me to the hosts list (even if it is only the one I see) and the the page freezes and never comes back.
Refreshing the page gives me the "circling the drain" logo forever.
However, I can click logout on the left menu and log back in and it's fine.

I will try a rebuild from scratch.

dsiminiuk

@Finallf I built a new minimalist Ubuntu 22 server and followed the same path and it does EXACTLY the same thing.

dsiminiuk

@olivierlambert
I tried to build from sources at the link you provided and there are instructions missing or things not obvious to me.
From /root I did git clone.
cd packages/xo-server
"Note: If you're installing xo-server as a global service, you may want to copy the file to /etc/xo-server/config.toml instead."
OK, I want to d othat so I cp sample.config.toml /etc/xo-server/ <- this directory does not exist
So I crated it and copied it there, but that is the only file there now.
Then I went back to /root/xen-orchestra/packages/xo-server
yarn start

yarn run v1.22.19
$ node dist/cli.mjs
2023-06-07T02:17:09.370Z xo:main INFO Configuration loaded.
2023-06-07T02:17:09.378Z xo:main INFO Web server listening on http://[::]:80
2023-06-07T02:17:09.379Z xo:main INFO User changed to root
2023-06-07T02:17:10.165Z xo:mixins:hooks WARN start failure {
  error: Error: spawn xenstore-read ENOENT
      at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
      at onErrorNT (node:internal/child_process:476:16)
      at processTicksAndRejections (node:internal/process/task_queues:82:21) {
    errno: -2,
    code: 'ENOENT',
    syscall: 'spawn xenstore-read',
    path: 'xenstore-read',
    spawnargs: [ 'vm' ],
    cmd: 'xenstore-read vm'
  }
}
2023-06-07T02:17:10.230Z xo:mixins:hooks WARN start failure {
  error: Error: Command failed with exit code 5: vgchange -an
    Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.
    Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)
      at makeError (file:///root/xen-orchestra/packages/xo-server/node_modules/execa/lib/error.js:59:11)
      at handlePromise (file:///root/xen-orchestra/packages/xo-server/node_modules/execa/index.js:124:26) {
    shortMessage: 'Command failed with exit code 5: vgchange -an',
    command: 'vgchange -an',
    escapedCommand: 'vgchange -an',
    exitCode: 5,
    signal: undefined,
    signalDescription: undefined,
    stdout: '',
    stderr: '  Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.\n' +
      `  Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)`,
    failed: true,
    timedOut: false,
    isCanceled: false,
    killed: false
  }
}
2023-06-07T02:17:10.427Z xo:main INFO Setting up / → /etc/xo/xo-web/dist/

When I went to the Web address I got
"Cannot GET /"

dsiminiuk

I've managed to get the build from sources to work and it does EXACTLY the same thing.
That excludes the XenServerInstallerUpdater as being the problem.

julien-f

@dsiminiuk If you cannot pinpoint the problematic commit, you can deploy an official XOA, then open a support tunnel and we'll be able to check it out directly.

dsiminiuk

@julien-f I tried that but got "An error occurred", but I saw at my host's address (after accepting the SSL certificate alert a directed) that it could be deployed from right there so I tried that and got another error.

If you can give me the master commits from newest to oldest over the last two weeks I can try those one at a time. I have no idea how to find them.

olivierlambert

As @julien-f told you, see https://xcp-ng.org/forum/topic/6940/xo-task-watcher-issue-cr-broken/55?_=1686148304910

dsiminiuk

@olivierlambert I will try that on the new VM I built last night since it is built from sources per your doco. I could not use that for the xo deployed by ronivay's script.

Additional observation: The behavior I see when there is one host online and another offline (web page freezes when clicking the Hosts icon from the Pool view) does NOT happen when both hosts are online (perhaps that there are no hosts offline). This is a clue as to the source of the issue.

Am I really the only user seeing this?

dsiminiuk

@olivierlambert I've done rebuilds going back to f77675a8a3590668413ec5bda02030c34b43408a and still the problem persists. I'm suspecting something wrong with the pool master now.

dsiminiuk

There is definitely something wrong with my 2nd host. It says it is not a member of a pool and the management interface is gone and does not persist even after an Emergency Network reset.

Time to rebuild that machine and I'll report back about XO.

dsiminiuk

@olivierlambert @Finallf @julien-f After a rebuild of my 2nd host everything is fine now. It must have become corrupted during the last rolling pool update.
Please accept my apologies for the fire drill.

Ooopsie! Right up until I powered off the 2nd host and now no hosts are showing and the page freezes again.

Oh well. At least I had a chance to swap out spinning hard disks for SSDs during the rebuild of host 2.

dsiminiuk

@olivierlambert I moved my pool master to the 2nd host, then rebuilt the 1st, so now both are booting on SSD. Both are updated to latest patches, and still I cannot deploy the XOA appliance over the web, I get the same error as above.

"message": "INVALID_VALUE",

I'm going to stop complaining. I can manage my VMs. Hopefully the issue will appear elsewhere and it can be exposed.

In the meantime I am writing a script on the xo VM with nut-client and xo-cli to shutdown VMs and hosts on power fail, since enabling the epel-release repo is now blocked on XCP-NG.

dsiminiuk

@olivierlambert as of today the build done on commit 3688e762b (or perhaps the previous, can't be sure) has resolved this issue. I can now see all hosts, online or when offline, and clicking on the hosts from the Pool view takes me to the Hosts list and no longer hangs the page.

julien-f

@dsiminiuk I don't it's related to XO itself, there has been no changes regarding this.

dsiminiuk

@julien-f @julien-f All I have done is let ronivay's script do the builds daily on schedule if there is a new commit on the master branch. His script has not changed. There have been no new patches to the hosts. I don't have an explanation.