Hosts have disappeared from XO Web UI

dsiminiuk

I am having a new issue.

XCP-ng 8.2.1 on two hosts, host #2 is powered off most of the time.

Issue: No Hosts listed in XO.

VMs are listed, Pool is listed, Stats showing, and I reconnected pool master, still no hosts listed.

Building from source using the script from ronivay/XenOrchestraInstallerUpdater

This is happening on commit 41bb16ca3 (xo-server 5.116.3, xo-web 5.119.1) and commit 4f7f1 (xo-server 5.116.0, xo-web 5.119.1)

I reverted to commit fdeaa67 (tried just because it was older) and they are back now.

Thanks
Danny

dsiminiuk

Walking through the commits, it was working at 36edfdc64

But... Updating Xen Orchestra from '36edfdc64' to '9686af970'

is when it stopped working.

olivierlambert

Any idea @julien-f ? (in case there's something obvious there)

julien-f

No I have no idea what could be happening.

@dsiminiuk You can follow the process described in this comment to pinpoint the problematic commit. It would help us a lot fix the problem.

olivierlambert

He said it appeared on 9686af970

julien-f

@olivierlambert Nope, what I understand is that it appeared somewhere between 36edfdc64 and 9686af970.

Also, none of these commits are on the master branch.

olivierlambert

Okay I assumed he was talking about XO commits

AtaxyaNetwork

@olivierlambert I was thinking he's talking about a commit number from the third party installer ronivay/XenOrchestraInstallerUpdater, but i couldn't find the commit either.

@dsiminiuk Can you tell us the commit number displayed in the page "about" ?

julien-f

@dsiminiuk also, which version of Node are you using?

dsiminiuk

Updating to master branch...

Script says: "Updating Xen Orchestra from '36edfdc64' to 'f28721b84'"

node -v
v18.13.0

yarn -v
1.22.19

Xen Orchestra, commit f2872
xo-server 5.116.3
xo-web 5.119.1

Screenshot from 2023-06-06 10-19-28.png

julien-f

@dsiminiuk Unfortunately that does not help me a lot, I don't see what change could have trigger this behaviour.

Is there anything in xo-server's output?

Also, doing the bisect search to pinpoint the precise commit (as explained here) would help a lot.

Finallf

@dsiminiuk
I am having this same problem.
In my case it was because I used a not so clean debian system.
I solved it as follows:
I installed debia with minimal configuration, In Software selection, select only "SSH server" and "standard system utilities"
already as root in debian I gave an apt update

Config static IP:
nano /etc/network/interfaces

	# The loopback network interface
	auto lo
	iface lo inet loopback

	# The primary network interface
	auto eth0
	iface eth0 inet static
		address 192.168.1.101
		netmask 255.255.255.0
		gateway 192.168.1.1

#Address, netmask and gateway change according to your network

#Config SSH:
nano /etc/ssh/sshd_config

	Modify the line:
		#PermitRootLogin prohibit-password
	To:
		PermitRootLogin yes
	Restart ssh:
		systemctl restart sshd
	It is now possible to connect to Debian via SSH

#Installing XCP-ng Tools

mount /dev/cdrom /mnt
cd /mnt
apt install ./xe-guest-utilities_7.20.2-1_amd64.deb

#Install Xen Orchestra with script:
apt instal git
git clone https://github.com/ronivay/XenOrchestraInstallerUpdater.git
cd XenOrchestraInstallerUpdater/
cp sample.xo-install.cfg xo-install.cfg

Edit xo-install.cfg depending on your needs.

./xo-install.sh

#First Configs:
When installation is complete, browse the WebUI.
User: admin@admin.net
password: admin

Go to Settings > Users
Create a new User with Admin powers
Log out of the WebUI and log in with the User created
Go to Settings > Users
Select the User admin@admin.net and click Delete User

Go to Settings > Servers
Add your XCP-ng server and click Connect
In Status if a red exclamation appears, click on it and accept the self-signed certificate

olivierlambert

Hi,

I'd like to remember that we can't support 3rd party scripted installs, just following our doc isn't that much complicated: https://xen-orchestra.com/docs/installation.html#from-the-sources

dsiminiuk

@Finallf gave me a clue, about accounts.

I switched back to master f28721b84.

When I login with admin@admin.net I can see the one Online host, but not the Offline host. Removing the filter "power_state:running" did not reveal it.
I powered up the 2nd host and now I can see both online.
Logged out.

Logged in with my personal account which is also an Admin.
Could not see any Hosts listed.
In account settings I cleared the filter I had previously saved by the name "None" (so that the pesky power_state:running didn't have to be cleared every time), and now I can see both hosts online.

When I power off host#2, I can not see it listed on the Hosts page using either account.

Not seeing ANY hosts appears to have been caused by the filter being corrupted or no longer valid for whatever reason on the personal Admin account.

However, I should now be able to see the online AND offline host, and it it is not listed using either Admin account.

Better, but still something wrong.

Gheppy

I use this steps to build XOCE, starting from clean debian with only ssh

#Connect to ssh and change to root account, or use sudo in front of each command

Change Ip on LAN
nano /etc/network/interfaces

iface eth0 inet static
address 192.168.2.10
netmask 255.255.255.0
gateway 192.168.2.1
dns-nameservers 8.8.8.8 8.8.4.4

systemctl restart networking

#Update and some utility
apt update && apt upgrade -y && apt autoremove -y
apt --yes install git curl apt-transport-https gnupg mc wget net-tools

#Install node
curl -fsSL https://deb.nodesource.com/setup_18.x | bash - &&
apt install -y nodejs
node -v

#Install yarn
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
apt update
apt install -y yarn
yarn --version

#Aditional packages
apt install -y build-essential redis-server libpng-dev git python3-minimal libvhdi-utils lvm2 cifs-utils nfs-common
npm install -g n

#Install XOCE
cd /opt/

#Clone git and build XOCE
git clone -b master https://github.com/vatesfr/xen-orchestra
cd xen-orchestra
yarn
yarn build
yarn cache clean
cp /opt/xen-orchestra/packages/xo-server/sample.config.toml /opt/xen-orchestra/packages/xo-server/.xo-server.toml

#Activate all plug-ins
cd /opt
mkdir -p /usr/local/lib/node_modules/

cat > /opt/plugins.sh <<EOF
#!/bin/bash

dest='/usr/local/lib/node_modules'
cale='/opt'
ignoreplugins=("xo-server-test")

echo "Cleaning plugin-urile..."
find \$dest/xo-server-* -xtype l -delete

for plugin in "\${ignoreplugins[@]}"; do
  if [ -L \$dest/\$plugin ]; then
    echo "Removing link for \$plugin"
    rm \$dest/\$plugin
  fi
done

echo "Cleaning plugin-urile..."
for source in \$(ls -d \$cale/xen-orchestra/packages/xo-server-*); do
  plugin=\$(basename \$source)
  if [[ "\${ignoreplugins[@]}" =~ \$plugin ]]; then
    echo "Ignoring \$plugin plugin"
  elif [ ! -L \$dest\$plugin ];  then
    echo "Delete old link \$plugin from \$dest/\$plugin"
    rm \$dest/\$plugin
    echo "Make link for \$plugin on \$dest/\$plugin"
    ln -s "\$source" "\$dest"
  fi
done

  
EOF

chmod 755 /opt/plugins.sh
/opt/plugins.sh

#XOCE in service mode

cat > /lib/systemd/system/xo-server.service <<EOF
#Systemd service for XO-Server.

[Unit]
Description= XO Server
After=network-online.target

[Service]
WorkingDirectory=/opt/xen-orchestra/packages/xo-server/
ExecStart=/usr/bin/node ./dist/cli.mjs

Restart=always
SyslogIdentifier=xo-server

[Install]
WantedBy=multi-user.target

EOF

systemctl daemon-reload
systemctl enable xo-server

dsiminiuk

Something else Hosts related...
When I click on the Pools list, I see my pool. I click on it and it shows me the 3 icons. The 2x server icon, 3x Storage SRs and 10x VMs.

if I click on the Storage Icon it takes me to the storage list http://xo/#/home?p=1&s=&t=SR
if I click on the VM icon it takes me to the VM list http://xo/#/home?s=$pool:ac9585b6-39ac-.....c082b
But if I click on the Server icon I expect it to take me to the hosts list (even if it is only the one I see) and the the page freezes and never comes back.
Refreshing the page gives me the "circling the drain" logo forever.
However, I can click logout on the left menu and log back in and it's fine.

I will try a rebuild from scratch.

dsiminiuk

@Finallf I built a new minimalist Ubuntu 22 server and followed the same path and it does EXACTLY the same thing.

dsiminiuk

@olivierlambert
I tried to build from sources at the link you provided and there are instructions missing or things not obvious to me.
From /root I did git clone.
cd packages/xo-server
"Note: If you're installing xo-server as a global service, you may want to copy the file to /etc/xo-server/config.toml instead."
OK, I want to d othat so I cp sample.config.toml /etc/xo-server/ <- this directory does not exist
So I crated it and copied it there, but that is the only file there now.
Then I went back to /root/xen-orchestra/packages/xo-server
yarn start

yarn run v1.22.19
$ node dist/cli.mjs
2023-06-07T02:17:09.370Z xo:main INFO Configuration loaded.
2023-06-07T02:17:09.378Z xo:main INFO Web server listening on http://[::]:80
2023-06-07T02:17:09.379Z xo:main INFO User changed to root
2023-06-07T02:17:10.165Z xo:mixins:hooks WARN start failure {
  error: Error: spawn xenstore-read ENOENT
      at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
      at onErrorNT (node:internal/child_process:476:16)
      at processTicksAndRejections (node:internal/process/task_queues:82:21) {
    errno: -2,
    code: 'ENOENT',
    syscall: 'spawn xenstore-read',
    path: 'xenstore-read',
    spawnargs: [ 'vm' ],
    cmd: 'xenstore-read vm'
  }
}
2023-06-07T02:17:10.230Z xo:mixins:hooks WARN start failure {
  error: Error: Command failed with exit code 5: vgchange -an
    Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.
    Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)
      at makeError (file:///root/xen-orchestra/packages/xo-server/node_modules/execa/lib/error.js:59:11)
      at handlePromise (file:///root/xen-orchestra/packages/xo-server/node_modules/execa/index.js:124:26) {
    shortMessage: 'Command failed with exit code 5: vgchange -an',
    command: 'vgchange -an',
    escapedCommand: 'vgchange -an',
    exitCode: 5,
    signal: undefined,
    signalDescription: undefined,
    stdout: '',
    stderr: '  Logical volume ubuntu-vg/ubuntu-lv contains a filesystem in use.\n' +
      `  Can't deactivate volume group "ubuntu-vg" with 1 open logical volume(s)`,
    failed: true,
    timedOut: false,
    isCanceled: false,
    killed: false
  }
}
2023-06-07T02:17:10.427Z xo:main INFO Setting up / → /etc/xo/xo-web/dist/

When I went to the Web address I got
"Cannot GET /"

dsiminiuk

I've managed to get the build from sources to work and it does EXACTLY the same thing.
That excludes the XenServerInstallerUpdater as being the problem.

julien-f

@dsiminiuk If you cannot pinpoint the problematic commit, you can deploy an official XOA, then open a support tunnel and we'll be able to check it out directly.