Unable to reconnect XCP server after XOA update

erwand

Hi all ,

I've got an old version of XOA and update to the latest master branch using Git. Before everything work fine.

Bu now i'm unable to reconnect my server in log i've got this message

watchEvents Error: HTTP connection has timed out
août 16 11:23:27 xoa xo-server[5237]:     at ClientRequest.<anonymous> (/opt/xo/xo-builds/xen-orchestra-202308161116/node_modules/http-request-plus/index.js:61:25)
août 16 11:23:27 xoa xo-server[5237]:     at ClientRequest.emit (node:events:514:28)
août 16 11:23:27 xoa xo-server[5237]:     at ClientRequest.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202308161116/@xen-orchestra/log/configure.js:52:17)
août 16 11:23:27 xoa xo-server[5237]:     at TLSSocket.emitRequestTimeout (node:_http_client:840:9)
août 16 11:23:27 xoa xo-server[5237]:     at Object.onceWrapper (node:events:628:28)
août 16 11:23:27 xoa xo-server[5237]:     at TLSSocket.emit (node:events:526:35)
août 16 11:23:27 xoa xo-server[5237]:     at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202308161116/@xen-orchestra/log/configure.js:52:17)
août 16 11:23:27 xoa xo-server[5237]:     at TLSSocket.Socket._onTimeout (node:net:589:8)
août 16 11:23:27 xoa xo-server[5237]:     at listOnTimeout (node:internal/timers:573:17)
août 16 11:23:27 xoa xo-server[5237]:     at processTimers (node:internal/timers:514:7) {
août 16 11:23:27 xoa xo-server[5237]:   url: 'https://195.xxx.yy.zz/jsonrpc',
août 16 11:23:27 xoa xo-server[5237]:   originalUrl: 'https://195.xxx.yy.zz/jsonrpc',
août 16 11:23:27 xoa xo-server[5237]:   call: {
août 16 11:23:27 xoa xo-server[5237]:     method: 'event.from',
août 16 11:23:27 xoa xo-server[5237]:     params: [ [Array], '00000000000002030971,00000000000001764125', 60.1 ]
août 16 11:23:27 xoa xo-server[5237]:   }

I've try opening the adress in a browser the adresse 'https://195.xxx.yy.zz/jsonrpc', and i've got 404

I've check the network between XOA and XCP server and no restriction, no firewal, ...

Have you got an idea ?

FYI; XCP version (8.2 LTS)

erwand

@erwand

I'dont know if it's related but on the XCP server i've got this error message when i try to connect

Aug 16 16:47:30 xcp-ng8 xapi: [debug||5054 HTTPS aaa.bbb.cc.zzz->:::80|session.login_with_password D:4854db6a1911|xapi_session] Success: local auth, user root from HTTP request from Internet with User-Agent: unknown
Aug 16 16:47:30 xcp-ng8 xapi: [ info||5054 HTTPS aaa.bbb.cc.zzz->:::80|session.login_with_password D:4854db6a1911|xapi_session] Session.create trackid=142f95adb60374a681fd4b96d756e616 pool=false uname=root originator= is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Aug 16 16:47:30 xcp-ng8 xapi: [debug||5055 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:5763c1938d1e created by task D:4854db6a1911
Aug 16 16:47:30 xcp-ng8 xapi: [debug||5056 :::80||api_effect] pool.add_to_other_config
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 HTTPS aaa.bbb.cc.zzz->:::80|pool.add_to_other_config D:9e2b8e1bc66f|sql] Duplicate key in set or map: table pool; field other_config; ref OpaqueRef:ccd0e30e-c0e1-4428-be2d-1d0e4534546c; key xo:clientInfo:yt8uh0ytrxj
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] pool.add_to_other_config D:9e2b8e1bc66f failed with exception Db_exn.Duplicate_key("pool", "other_config", "OpaqueRef:ccd0e30e-c0e1-4428-be2d-1d0e4534546c", "xo:clientInfo:yt8uh0ytrxj")
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] Raised Db_exn.Duplicate_key("pool", "other_config", "OpaqueRef:ccd0e30e-c0e1-4428-be2d-1d0e4534546c", "xo:clientInfo:yt8uh0ytrxj")
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 1/8 xapi Raised at file ocaml/database/db_cache_impl.ml, line 316
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 2/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/rbac.ml, line 205
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 92
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 113
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 7/8 xapi Called from file map.ml, line 135
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace] 8/8 xapi Called from file src/sexp_conv.ml, line 156
Aug 16 16:47:30 xcp-ng8 xapi: [error||5056 :::80||backtrace]
Aug 16 16:47:30 xcp-ng8 xapi: [debug||5056 :::80||api_effect] pool.remove_from_other_config
Aug 16 16:47:30 xcp-ng8 xapi: [debug||5056 :::80||api_effect] pool.add_to_other_config

Danp

Check the IP address under Setttings > Servers to make sure it points to the current pool master.

Also would be good to make sure that you don't have another device on the network using the same IP address.

erwand

@Danp Thnks for the replys

It's a Pool master and pointing on the good IP address.

I've check and no conflict

Andrew

@erwand Is this XO source, not XOA? If It's XO source, did you update node (18.17.1) and other required packages?

erwand

@Andrew node version is v20.5.1

Andrew

@erwand Please downgrade to 18. It's unknown if 20 will work. 18.17 is the current supported version for XO source.

erwand

@Andrew Same things happen wit node 18.17.1

erwand

In fact, the problem seems to come from the way I launch the orchestra server.

If I run it manually with the "Yarn star" command, everything works fine.

However, if I use the "Forever-service" method in this case, the XCP servers connect then disconnect before displaying an authentication error message.

I've tried running xoa-server as root and also sudo with another user and the problem is identical.

But if i just user "forever start dist/cli.mjs" as root it works

Does anyone have any ideas?

Danp

How old was your XO install? Have you reviewed this thread?

erwand

@Danp a couple of days

I've make some new tests if it can help:

When i try to start orchestra using this commande in rc.local of my ubuntu 20.04.6

 cd /usr/src/xen-orchestra/packages/xo-server && forever start dist/cli.mjs

it's not working, i can connect to the web interface bu unable to connect to XCP

but if i lauch the same command on the CLI just after the start, it works

Danp

Sounds like a rights issue.