Xen-Orchestra broken after rollback

mavoff

Hi,

A recent update made XO from sources incompatible with the ARM platform, so I tried to rollback to a previous version which I knew it was working.

To move back I simply did
# git reset --hard c5d2726faa0d373ee58371d05395fec2affaa7a5
then

# yarn
# yarn build

Commands completed without errors.
However when I access the web interface I get an error:

Cannot GET /

Using the forever service and using systemctl the service starts fine and journalctl isn't showing nothing out of the ordinary.
However if I stop the service and do yarn start at the xo-server folder I get the following:

# yarn start
yarn run v1.22.19
$ node dist/cli.mjs
2022-09-22T10:50:18.609Z xo:main INFO Configuration loaded.
2022-09-22T10:50:18.615Z xo:main INFO Web server listening on http://[::]:80
2022-09-22T10:50:18.623Z xo:main INFO Web server listening on https://[::]:443
2022-09-22T10:50:18.800Z xo:mixins:hooks WARN start failure {
  error: Error: spawn xenstore-read ENOENT
      at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
      at onErrorNT (node:internal/child_process:477:16)
      at processTicksAndRejections (node:internal/process/task_queues:83:21) {
    errno: -2,
    code: 'ENOENT',
    syscall: 'spawn xenstore-read',
    path: 'xenstore-read',
    spawnargs: [ 'vm' ],
    cmd: 'xenstore-read vm'
  }
}
2022-09-22T10:50:18.804Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}
2022-09-22T10:50:18.823Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}
2022-09-22T10:50:18.824Z xo:mixins:hooks WARN start failure {
  error: Error: Command failed with exit code 5: vgchange -an
    Logical volume ocivolume/oled contains a filesystem in use.
    Can't deactivate volume group "ocivolume" with 2 open logical volume(s)
      at makeError (file:///opt/xen-orchestra/packages/xo-server/node_modules/execa/lib/error.js:59:11)
      at handlePromise (file:///opt/xen-orchestra/packages/xo-server/node_modules/execa/index.js:119:26) {
    shortMessage: 'Command failed with exit code 5: vgchange -an',
    command: 'vgchange -an',
    escapedCommand: 'vgchange -an',
    exitCode: 5,
    signal: undefined,
    signalDescription: undefined,
    stdout: '',
    stderr: '  Logical volume ocivolume/oled contains a filesystem in use.\n' +
      `  Can't deactivate volume group "ocivolume" with 2 open logical volume(s)`,
    failed: true,
    timedOut: false,
    isCanceled: false,
    killed: false
  }
}
2022-09-22T10:50:18.874Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}
2022-09-22T10:50:18.977Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}
2022-09-22T10:50:19.130Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}
2022-09-22T10:50:19.332Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}
2022-09-22T10:50:19.585Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}

And that goes forever.
port 6379 is redis so checking redis:

# systemctl status redis
● redis.service - Redis persistent key-value database
   Loaded: loaded (/usr/lib/systemd/system/redis.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/redis.service.d
           └─limit.conf
   Active: active (running) since Wed 2022-09-21 19:28:28 GMT; 15h ago
 Main PID: 1573 (redis-server)
    Tasks: 4 (limit: 8892)
   Memory: 938.2M
   CGroup: /system.slice/redis.service
           └─1573 /usr/bin/redis-server 127.0.0.1:6379

Sep 21 19:28:28 xo-admin systemd[1]: Starting Redis persistent key-value database...
Sep 21 19:28:28 xo-admin systemd[1]: Started Redis persistent key-value database.

Any hint on how can I put this back to work? thank you.

Danp

Have you tried rm -rf node_modules and then rebuilding with yarn; yarn build?

mavoff

@Danp Hi
Thank you for your suggestion. So changed back to that commit, moved node_modules from/opt/xen-orchestra/ to /opt and rebuilt with yarn; yarn build but the error is the same. When I open the URL I get Cannot GET /

Danp

You could review this thread to see if you have the same issue with a missing config file.

Other options that come to mind --

Restore from snapshot
Restore from backup
Rebuild from scratch

mavoff

@Danp thank you for your suggestion.
I had already seen that thread, we have the /etc/xo-server/config.toml file in place.
We didn't have this snapshotted, blame on us.

Rebuilding from source is the current best candidate option, but I just would like to figure out why is it that "cannot get /" as it seems the webserver is up and kicking and something is failing. Do you have an idea where I could check these logs? Thank you.

Danp

@maverick said in Xen-Orchestra broken after rollback:

c5d2726faa0d373ee58371d05395fec2affaa7a5

Why did you restore to this particular commit? Looking at it on GH, it is related to redis, as are many of the others around it.

Perhaps you should fall back to an earlier commit, such as d87db05b2bee125305bf84d537246ee16342a198.

mavoff

@Danp thank you for the suggestion. Actually I didn't choose that commit for any reason in particular, I just saw the log back to a date which I sorta reminded having done an update and XO was working and that one seemed back enough but with a huge level of uncertainty was just ball parking.

The commit you suggested worked!!! holy guacamole! that commit is gold!

mavoff

So I'm very very sorry for digging this up; I had this XO installation halted for a while wanted to come back to it.

In the meanwhile the resolution for the error presented on the initial post:

2022-09-22T10:50:18.823Z xo:xo-server ERROR unhandled error event {
  error: Error: connect ECONNREFUSED ::1:6379
      at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1161:16)
      at TCPConnectWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
    errno: -111,
    code: 'ECONNREFUSED',
    syscall: 'connect',
    address: '::1',
    port: 6379
  }
}

Solution:

Edit /etc/redis.conf;
Comment the bind directive;
Save, restart redis, done.