XOA won't start HTTPS listen on configured port

olivierlambert

The "well known installed script" is a 3rd party script that we have no review on it, so use it with caution But it's not your problem here.

Indeed, the output shows that XO only listens on port 80.

Let me check the doc.

olivierlambert

Ping @julien-f

julien-f

@bill-gertz We should see in xo-server's logs which config files it loads, it's weird there is not trace of it.

Please run xo-server manually and tell me what you see:

env DEBUG='*' xo-server

bill.gertz

@julien-f @olivierlambert
@julien-f said in XOA won't start HTTPS listen on configured port:

env DEBUG='*' xo-server

I took your request to have a slight typo, is this what you where looking for?

root@bfxoa001:/etc/xo/xo-server# systemctl stop xo-server.service 
root@bfxoa001:/etc/xo/xo-server# env DEBUG='*'; ./bin/xo-server
LANG=en_US.UTF-8
USER=root
PWD=/etc/xo/xo-server
HOME=/root
MAIL=/var/mail/root
SHELL=/bin/bash
TERM=xterm-256color
SHLVL=1
LANGUAGE=en_US:en
LOGNAME=root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LE_WORKING_DIR=/root/.acme.sh
_=/usr/bin/env
OLDPWD=/etc/xo/xo-server/src
DEBUG=*
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/config.toml +0ms
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/config.toml.orig +2ms
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/.xo-server.toml +0ms
2019-11-25T17:23:01.242Z xo:main INFO Configuration loaded.
2019-11-25T17:23:01.260Z xo:main INFO Web server listening on https://0.0.0.0:80
2019-11-25T17:23:01.364Z xo:xo-mixins:hooks DEBUG start start…
2019-11-25T17:23:01.773Z xo:xo-mixins:hooks DEBUG start finished
2019-11-25T17:23:01.774Z xo:xo-mixins:hooks DEBUG clean start…
2019-11-25T17:23:02.112Z xo:xo-mixins:hooks DEBUG clean finished
Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
2019-11-25T17:23:02.223Z xo:main INFO Setting up / → /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-web/dist
2019-11-25T17:23:02.227Z xo:plugin INFO register auth-github
2019-11-25T17:23:02.241Z xo:plugin INFO register auth-google
2019-11-25T17:23:02.244Z xo:plugin INFO register auth-ldap
2019-11-25T17:23:02.325Z xo:plugin INFO register auth-saml
2019-11-25T17:23:02.429Z xo:plugin INFO register backup-reports
2019-11-25T17:23:02.433Z xo:plugin INFO register load-balancer
2019-11-25T17:23:02.441Z xo:plugin INFO register perf-alert
2019-11-25T17:23:02.444Z xo:plugin INFO register sdn-controller
2019-11-25T17:23:02.455Z xo:plugin INFO register test
2019-11-25T17:23:02.456Z xo:plugin INFO register test-plugin
2019-11-25T17:23:02.457Z xo:plugin INFO register transport-email
2019-11-25T17:23:02.497Z xo:plugin INFO register transport-icinga2
2019-11-25T17:23:02.499Z xo:plugin INFO register transport-nagios
2019-11-25T17:23:02.500Z xo:plugin INFO register transport-slack
2019-11-25T17:23:02.608Z xo:plugin INFO register transport-xmpp
2019-11-25T17:23:02.658Z xo:plugin INFO register usage-report
2019-11-25T17:23:02.768Z xo:plugin INFO register web-hooks
2019-11-25T17:23:02.771Z xo:plugin INFO failed register test
2019-11-25T17:23:02.771Z xo:plugin INFO Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test' { error: { Error: Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Xo.registerPlugin (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:259:17)
    at Xo.registerPluginWrapper (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:315:24)
    at Promise.all.name (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:340:37)
    at arrayMap (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/_arrayMap.js:16:21)
    at map (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/map.js:50:10)
    at Xo.registerPluginsInPath (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:338:15)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7) code: 'MODULE_NOT_FOUND' } }
2019-11-25T17:23:02.826Z xo:xo-server:sdn-controller DEBUG No cert-dir provided, using default self-signed certificates
2019-11-25T17:23:02.861Z xo:plugin INFO successfully register auth-github
2019-11-25T17:23:02.861Z xo:plugin INFO successfully register auth-google
2019-11-25T17:23:02.861Z xo:plugin INFO successfully register auth-ldap
2019-11-25T17:23:02.861Z xo:plugin INFO successfully register auth-saml
2019-11-25T17:23:02.862Z xo:plugin INFO successfully register test-plugin
2019-11-25T17:23:02.862Z xo:plugin INFO successfully register transport-email
2019-11-25T17:23:02.862Z xo:plugin INFO successfully register transport-icinga2
2019-11-25T17:23:02.863Z xo:plugin INFO successfully register transport-nagios
2019-11-25T17:23:02.863Z xo:plugin INFO successfully register transport-slack
2019-11-25T17:23:02.863Z xo:plugin INFO successfully register transport-xmpp
2019-11-25T17:23:02.864Z xo:plugin INFO successfully register usage-report
2019-11-25T17:23:02.864Z xo:plugin INFO successfully register web-hooks
2019-11-25T17:23:02.864Z xo:plugin INFO successfully register backup-reports
2019-11-25T17:23:02.866Z xo:plugin INFO successfully register load-balancer
2019-11-25T17:23:02.867Z xo:plugin INFO successfully register perf-alert
2019-11-25T17:23:02.943Z xo:plugin INFO successfully register sdn-controller
2019-11-25T17:23:02.972Z xo:main INFO + WebSocket connection (10.69.0.222)

Danp

@bill-gertz Take out the semi-colon and rerun to get a more detailed output.

bill.gertz

@Danp @julien-f @olivierlambert

Roger that, run as requested:

root@bfxoa001:/etc/xo/xo-server# systemctl stop xo-server.service 
root@bfxoa001:/etc/xo/xo-server# env DEBUG='*' ./bin/xo-server
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/config.toml +0ms
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/config.toml.orig +2ms
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/.xo-server.toml +0ms
2019-11-25T17:37:34.487Z xo:main INFO Configuration loaded.
2019-11-25T17:37:34.505Z xo:main INFO Web server listening on https://0.0.0.0:80
2019-11-25T17:37:34.609Z xo:xo-mixins:hooks DEBUG start start…
  xen-api root@10.69.0.200: session.login_with_password(...) [56ms] ==> string +0ms
  xen-api root@10.69.0.200: pool.get_all_records(...) [7ms] ==> object +9ms
  xen-api root@10.69.0.200: system.listMethods(...) [7ms] ==> array +9ms
  xen-api root@10.69.0.200: connected +1ms
  xen-api root@10.69.0.200: event.inject(...) [6ms] ==> string +8ms
  xen-api root@10.69.0.200: subject.get_all_records(...) [87ms] ==> object +88ms
  xen-api root@10.69.0.200: pool_patch.get_all_records(...) [84ms] ==> object +0ms
  xen-api root@10.69.0.200: task.get_all_records(...) [123ms] ==> object +39ms
  xen-api root@10.69.0.200: DR_task.get_all_records(...) [123ms] ==> object +6ms
  xen-api root@10.69.0.200: host_patch.get_all_records(...) [125ms] ==> object +4ms
  xen-api root@10.69.0.200: Bond.get_all_records(...) [119ms] ==> object +0ms
  xen-api root@10.69.0.200: VMSS.get_all_records(...) [129ms] ==> object +0ms
  xen-api root@10.69.0.200: host_cpu.get_all_records(...) [124ms] ==> object +1ms
  xen-api root@10.69.0.200: pool.get_all_records(...) [143ms] ==> object +8ms
  xen-api root@10.69.0.200: VLAN.get_all_records(...) [127ms] ==> object +0ms
  xen-api root@10.69.0.200: PIF_metrics.get_all_records(...) [136ms] ==> object +7ms
  xen-api root@10.69.0.200: pool_update.get_all_records(...) [148ms] ==> object +0ms
  xen-api root@10.69.0.200: VM_metrics.get_all_records(...) [148ms] ==> object +1ms
  xen-api root@10.69.0.200: VM_guest_metrics.get_all_records(...) [161ms] ==> object +14ms
  xen-api root@10.69.0.200: VMPP.get_all_records(...) [161ms] =!> XapiError: MESSAGE_REMOVED() +1ms
  xen-api root@10.69.0.200: host.get_all_records(...) [158ms] ==> object +0ms
  xen-api root@10.69.0.200: VM.get_all_records(...) [177ms] ==> object +13ms
  xen-api root@10.69.0.200: host_crashdump.get_all_records(...) [188ms] ==> object +17ms
  xen-api root@10.69.0.200: network.get_all_records(...) [185ms] ==> object +0ms
  xen-api root@10.69.0.200: VIF.get_all_records(...) [184ms] ==> object +0ms
  xen-api root@10.69.0.200: VM_appliance.get_all_records(...) [205ms] ==> object +15ms
  xen-api root@10.69.0.200: crashdump.get_all_records(...) [190ms] ==> object +0ms
  xen-api root@10.69.0.200: VBD_metrics.get_all_records(...) [191ms] =!> XapiError: MESSAGE_REMOVED() +1ms
  xen-api root@10.69.0.200: PGPU.get_all_records(...) [185ms] ==> object +0ms
  xen-api root@10.69.0.200: SM.get_all_records(...) [195ms] ==> object +0ms
  xen-api root@10.69.0.200: VBD.get_all_records(...) [193ms] ==> object +0ms
  xen-api root@10.69.0.200: PIF.get_all_records(...) [218ms] ==> object +20ms
  xen-api root@10.69.0.200: secret.get_all_records(...) [208ms] ==> object +0ms
  xen-api root@10.69.0.200: host_metrics.get_all_records(...) [223ms] ==> object +1ms
  xen-api root@10.69.0.200: SR.get_all_records(...) [216ms] ==> object +0ms
  xen-api root@10.69.0.200: VIF_metrics.get_all_records(...) [220ms] =!> XapiError: MESSAGE_REMOVED() +0ms
  xen-api root@10.69.0.200: tunnel.get_all_records(...) [208ms] ==> object +0ms
  xen-api root@10.69.0.200: Cluster_host.get_all_records(...) [195ms] ==> object +0ms
  xen-api root@10.69.0.200: Feature.get_all_records(...) [200ms] ==> object +0ms
  xen-api root@10.69.0.200: VDI.get_all_records(...) [216ms] ==> object +1ms
  xen-api root@10.69.0.200: VGPU.get_all_records(...) [205ms] ==> object +0ms
  xen-api root@10.69.0.200: console.get_all_records(...) [212ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_site.get_all_records(...) [204ms] ==> object +0ms
  xen-api root@10.69.0.200: blob.get_all_records(...) [244ms] ==> object +34ms
  xen-api root@10.69.0.200: GPU_group.get_all_records(...) [240ms] ==> object +0ms
  xen-api root@10.69.0.200: USB_group.get_all_records(...) [233ms] ==> object +0ms
  xen-api root@10.69.0.200: network_sriov.get_all_records(...) [242ms] ==> object +0ms
  xen-api root@10.69.0.200: VGPU_type.get_all_records(...) [239ms] ==> object +1ms
  xen-api root@10.69.0.200: Cluster.get_all_records(...) [232ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_server.get_all_records(...) [238ms] ==> object +0ms
  xen-api root@10.69.0.200: PCI.get_all_records(...) [244ms] ==> object +1ms
  xen-api root@10.69.0.200: PBD.get_all_records(...) [250ms] ==> object +0ms
  xen-api root@10.69.0.200: PUSB.get_all_records(...) [235ms] ==> object +0ms
  xen-api root@10.69.0.200: SDN_controller.get_all_records(...) [237ms] ==> object +1ms
  xen-api root@10.69.0.200: role.get_all_records(...) [272ms] ==> object +0ms
2019-11-25T17:37:35.097Z xo:xo-mixins:hooks DEBUG start finished
  xen-api root@10.69.0.200: VUSB.get_all_records(...) [253ms] ==> object +18ms
  xen-api root@10.69.0.200: PVS_proxy.get_all_records(...) [258ms] ==> object +1ms
  xen-api root@10.69.0.200: PVS_cache_storage.get_all_records(...) [257ms] ==> object +0ms
2019-11-25T17:37:35.099Z xo:xo-mixins:hooks DEBUG clean start…
  xen-api root@10.69.0.200: message.get_all_records(...) [275ms] ==> object +8ms
2019-11-25T17:37:35.341Z xo:xo-mixins:hooks DEBUG clean finished
  express:application set "x-powered-by" to true +0ms
  express:application set "etag" to 'weak' +2ms
  express:application set "etag fn" to [Function: generateETag] +1ms
  express:application set "env" to 'production' +0ms
  express:application set "query parser" to 'extended' +0ms
  express:application set "query parser fn" to [Function: parseExtendedQueryString] +0ms
  express:application set "subdomain offset" to 2 +1ms
  express:application set "trust proxy" to false +0ms
  express:application set "trust proxy fn" to [Function: trustNone] +0ms
  express:application booting in production mode +0ms
  express:application set "view" to [Function: View] +0ms
  express:application set "views" to '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/views' +1ms
  express:application set "jsonp callback name" to 'callback' +0ms
  express:application set "view cache" to true +0ms
  express:router use '/' query +2ms
  express:router:layer new '/' +0ms
  express:router use '/' expressInit +1ms
  express:router:layer new '/' +0ms
  express:router use '/' helmet +1ms
  express:router:layer new '/' +0ms
  express:router use '/' compression +0ms
  express:router:layer new '/' +0ms
  express:router use '/' cookieParser +1ms
  express:router:layer new '/' +0ms
Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
  express:router use '/' session +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +0ms
  express:router use '/' urlencodedParser +1ms
  express:router:layer new '/' +0ms
  express:router use '/' initialize +0ms
  express:router:layer new '/' +1ms
  express:router use '/' bound _handleHttpRequest +0ms
  express:router:layer new '/' +1ms
  express:router:route new '/signin' +103ms
  express:router:layer new '/signin' +1ms
  express:router:route get '/signin' +0ms
  express:router:layer new '/' +1ms
  express:router:route new '/signout' +0ms
  express:router:layer new '/signout' +1ms
  express:router:route get '/signout' +0ms
  express:router:layer new '/' +0ms
  express:router:route new '/signin-otp' +1ms
  express:router:layer new '/signin-otp' +0ms
  express:router:route get '/signin-otp' +0ms
  express:router:layer new '/' +1ms
  express:router:route new '/signin-otp' +0ms
  express:router:layer new '/signin-otp' +0ms
  express:router:route post '/signin-otp' +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +1ms
  express:router use '/' <anonymous> +1ms
  express:router:layer new '/' +1ms
2019-11-25T17:37:35.468Z xo:main INFO Setting up / → /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-web/dist
  express:router use '/' serveStatic +1ms
  express:router:layer new '/' +0ms
2019-11-25T17:37:35.473Z xo:plugin INFO register auth-github
2019-11-25T17:37:35.485Z xo:plugin INFO register auth-google
2019-11-25T17:37:35.489Z xo:plugin INFO register auth-ldap
2019-11-25T17:37:35.564Z xo:plugin INFO register auth-saml
2019-11-25T17:37:35.664Z xo:plugin INFO register backup-reports
2019-11-25T17:37:35.668Z xo:plugin INFO register load-balancer
2019-11-25T17:37:35.676Z xo:plugin INFO register perf-alert
2019-11-25T17:37:35.680Z xo:plugin INFO register sdn-controller
2019-11-25T17:37:35.691Z xo:plugin INFO register test
2019-11-25T17:37:35.692Z xo:plugin INFO register test-plugin
2019-11-25T17:37:35.693Z xo:plugin INFO register transport-email
2019-11-25T17:37:35.734Z xo:plugin INFO register transport-icinga2
2019-11-25T17:37:35.735Z xo:plugin INFO register transport-nagios
2019-11-25T17:37:35.737Z xo:plugin INFO register transport-slack
2019-11-25T17:37:35.847Z xo:plugin INFO register transport-xmpp
2019-11-25T17:37:35.897Z xo:plugin INFO register usage-report
2019-11-25T17:37:36.010Z xo:plugin INFO register web-hooks
2019-11-25T17:37:36.012Z xo:plugin INFO failed register test
2019-11-25T17:37:36.012Z xo:plugin INFO Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test' { error: { Error: Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Xo.registerPlugin (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:259:17)
    at Xo.registerPluginWrapper (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:315:24)
    at Promise.all.name (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:340:37)
    at arrayMap (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/_arrayMap.js:16:21)
    at map (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/map.js:50:10)
    at Xo.registerPluginsInPath (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:338:15)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7) code: 'MODULE_NOT_FOUND' } }
2019-11-25T17:37:36.066Z xo:xo-server:sdn-controller DEBUG No cert-dir provided, using default self-signed certificates
2019-11-25T17:37:36.090Z xo:plugin INFO successfully register auth-github
2019-11-25T17:37:36.090Z xo:plugin INFO successfully register auth-google
2019-11-25T17:37:36.090Z xo:plugin INFO successfully register auth-ldap
2019-11-25T17:37:36.090Z xo:plugin INFO successfully register auth-saml
2019-11-25T17:37:36.091Z xo:plugin INFO successfully register test-plugin
2019-11-25T17:37:36.091Z xo:plugin INFO successfully register transport-email
2019-11-25T17:37:36.091Z xo:plugin INFO successfully register transport-icinga2
2019-11-25T17:37:36.091Z xo:plugin INFO successfully register transport-nagios
2019-11-25T17:37:36.092Z xo:plugin INFO successfully register transport-slack
2019-11-25T17:37:36.092Z xo:plugin INFO successfully register transport-xmpp
2019-11-25T17:37:36.092Z xo:plugin INFO successfully register usage-report
2019-11-25T17:37:36.092Z xo:plugin INFO successfully register web-hooks
2019-11-25T17:37:36.093Z xo:plugin INFO successfully register backup-reports
2019-11-25T17:37:36.095Z xo:plugin INFO successfully register load-balancer
2019-11-25T17:37:36.095Z xo:plugin INFO successfully register perf-alert
  xen-api root@10.69.0.200: event.from(...) [677ms] ==> object +1s
2019-11-25T17:37:36.159Z xo:plugin INFO successfully register sdn-controller
2019-11-25T17:37:38.977Z xo:main INFO + WebSocket connection (10.69.0.222)
2019-11-25T17:37:38.992Z xo:api DEBUG (unknown user) | session.signIn(...) [11ms] ==> object
2019-11-25T17:37:39.014Z xo:api DEBUG admin.bill | xo.getAllObjects(...) [9ms] ==> object
  express:router dispatching GET /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +4s
  express:router query  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +1ms
  express:router expressInit  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +1ms
  express:router helmet  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +1ms
  express:router compression  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +1ms
  express:router cookieParser  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +1ms
  express:router session  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +1ms
  express-session fetching C7vNjZR_gJbYflyBRght4xmngAK6sTwc +1ms
  express-session no session found +1ms
  express:router <anonymous>  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +2ms
  express:router urlencodedParser  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +0ms
  body-parser:urlencoded skip empty body +1ms
  express:router initialize  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +0ms
  express:router bound _handleHttpRequest  : /api/yeJvXYqpoEKE5iIJ7l0-LlsmdyOgKKM-eQ3pGxRDoZU +0ms
  compression gzip compression +9ms
  xen-api root@10.69.0.200: event.from(...) [4s] ==> object +4s
  xen-api root@10.69.0.200: host.call_plugin(...) [350ms] ==> string +292ms
2019-11-25T17:37:40.164Z xo:api DEBUG admin.bill | pool.listMissingPatches(...) [360ms] ==> array
  xen-api root@10.69.0.200: event.from(...) [139ms] ==> object +48ms
  xen-api root@10.69.0.200: event.from(...) [455ms] ==> object +657ms
  xen-api root@10.69.0.200: host.call_plugin(...) [1s] ==> string +1s
2019-11-25T17:37:42.033Z xo:api DEBUG admin.bill | pool.listMissingPatches(...) [1s] ==> array
  xen-api root@10.69.0.200: event.from(...) [1s] ==> object +48ms

It looks like it's loading the older config config.toml.orig and a "dot hidden" file .xo-server.toml. Let me move these pigs out of the directory and restart.

bill.gertz

@Danp @julien-f @olivierlambert

Winner, winner, chicken dinner. The debug trick was the key to see WTF it was loading. Moved those two pigs out of the directory to an a hidden directory off of the parent and it starts as it's supposed to. Here is the same debug output with those two files out of the way:

root@bfxoa001:/etc/xo/xo-server# systemctl stop xo-server.service 
root@bfxoa001:/etc/xo/xo-server# env DEBUG='*' ./bin/xo-server
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/config.toml +0ms
2019-11-25T17:53:45.979Z xo:main INFO Configuration loaded.
2019-11-25T17:53:45.998Z xo:main INFO Web server listening on https://0.0.0.0:443
2019-11-25T17:53:46.105Z xo:xo-mixins:hooks DEBUG start start…
  xen-api root@10.69.0.200: session.login_with_password(...) [58ms] ==> string +0ms
  xen-api root@10.69.0.200: pool.get_all_records(...) [7ms] ==> object +8ms
  xen-api root@10.69.0.200: system.listMethods(...) [6ms] ==> array +8ms
  xen-api root@10.69.0.200: connected +1ms
  xen-api root@10.69.0.200: event.inject(...) [6ms] ==> string +8ms
  xen-api root@10.69.0.200: subject.get_all_records(...) [81ms] ==> object +82ms
  xen-api root@10.69.0.200: pool_patch.get_all_records(...) [90ms] ==> object +12ms
  xen-api root@10.69.0.200: VM_appliance.get_all_records(...) [86ms] ==> object +2ms
  xen-api root@10.69.0.200: pool_update.get_all_records(...) [144ms] ==> object +53ms
  xen-api root@10.69.0.200: VIF_metrics.get_all_records(...) [148ms] =!> XapiError: MESSAGE_REMOVED() +15ms
  xen-api root@10.69.0.200: DR_task.get_all_records(...) [158ms] ==> object +4ms
  xen-api root@10.69.0.200: VLAN.get_all_records(...) [150ms] ==> object +1ms
  xen-api root@10.69.0.200: tunnel.get_all_records(...) [141ms] ==> object +1ms
  xen-api root@10.69.0.200: pool.get_all_records(...) [167ms] ==> object +0ms
  xen-api root@10.69.0.200: PIF_metrics.get_all_records(...) [152ms] ==> object +0ms
  xen-api root@10.69.0.200: VM.get_all_records(...) [179ms] ==> object +15ms
  xen-api root@10.69.0.200: role.get_all_records(...) [220ms] ==> object +37ms
  xen-api root@10.69.0.200: USB_group.get_all_records(...) [244ms] ==> object +62ms
  xen-api root@10.69.0.200: host_metrics.get_all_records(...) [271ms] ==> object +0ms
  xen-api root@10.69.0.200: PCI.get_all_records(...) [254ms] ==> object +0ms
  xen-api root@10.69.0.200: task.get_all_records(...) [281ms] ==> object +0ms
  xen-api root@10.69.0.200: VBD.get_all_records(...) [262ms] ==> object +1ms
  xen-api root@10.69.0.200: VMSS.get_all_records(...) [276ms] ==> object +0ms
  xen-api root@10.69.0.200: PBD.get_all_records(...) [261ms] ==> object +0ms
  xen-api root@10.69.0.200: SM.get_all_records(...) [265ms] ==> object +0ms
  xen-api root@10.69.0.200: Cluster.get_all_records(...) [244ms] ==> object +1ms
  xen-api root@10.69.0.200: VM_metrics.get_all_records(...) [280ms] ==> object +0ms
  xen-api root@10.69.0.200: host.get_all_records(...) [288ms] ==> object +13ms
  xen-api root@10.69.0.200: console.get_all_records(...) [273ms] ==> object +0ms
  xen-api root@10.69.0.200: VBD_metrics.get_all_records(...) [276ms] =!> XapiError: MESSAGE_REMOVED() +0ms
  xen-api root@10.69.0.200: host_crashdump.get_all_records(...) [288ms] ==> object +1ms
  xen-api root@10.69.0.200: VM_guest_metrics.get_all_records(...) [293ms] ==> object +0ms
  xen-api root@10.69.0.200: SDN_controller.get_all_records(...) [262ms] ==> object +0ms
  xen-api root@10.69.0.200: VMPP.get_all_records(...) [292ms] =!> XapiError: MESSAGE_REMOVED() +0ms
  xen-api root@10.69.0.200: secret.get_all_records(...) [273ms] ==> object +1ms
  xen-api root@10.69.0.200: blob.get_all_records(...) [275ms] ==> object +0ms
  xen-api root@10.69.0.200: host_patch.get_all_records(...) [288ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_cache_storage.get_all_records(...) [264ms] ==> object +1ms
  xen-api root@10.69.0.200: VUSB.get_all_records(...) [261ms] ==> object +0ms
  xen-api root@10.69.0.200: network_sriov.get_all_records(...) [272ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_server.get_all_records(...) [267ms] ==> object +0ms
  xen-api root@10.69.0.200: SR.get_all_records(...) [282ms] ==> object +1ms
  xen-api root@10.69.0.200: VGPU.get_all_records(...) [270ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_proxy.get_all_records(...) [267ms] ==> object +0ms
  xen-api root@10.69.0.200: VGPU_type.get_all_records(...) [269ms] ==> object +0ms
  xen-api root@10.69.0.200: crashdump.get_all_records(...) [279ms] ==> object +1ms
  xen-api root@10.69.0.200: Feature.get_all_records(...) [267ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_site.get_all_records(...) [270ms] ==> object +0ms
  xen-api root@10.69.0.200: Cluster_host.get_all_records(...) [261ms] ==> object +0ms
  xen-api root@10.69.0.200: GPU_group.get_all_records(...) [273ms] ==> object +1ms
  xen-api root@10.69.0.200: network.get_all_records(...) [290ms] ==> object +0ms
  xen-api root@10.69.0.200: PUSB.get_all_records(...) [266ms] ==> object +0ms
  xen-api root@10.69.0.200: Bond.get_all_records(...) [286ms] ==> object +0ms
  xen-api root@10.69.0.200: host_cpu.get_all_records(...) [292ms] ==> object +1ms
  xen-api root@10.69.0.200: PGPU.get_all_records(...) [275ms] ==> object +0ms
  xen-api root@10.69.0.200: PIF.get_all_records(...) [289ms] ==> object +0ms
  xen-api root@10.69.0.200: VIF.get_all_records(...) [290ms] ==> object +0ms
  xen-api root@10.69.0.200: VDI.get_all_records(...) [285ms] ==> object +1ms
2019-11-25T17:53:46.616Z xo:xo-mixins:hooks DEBUG start finished
  xen-api root@10.69.0.200: message.get_all_records(...) [290ms] ==> object +10ms
2019-11-25T17:53:46.700Z xo:xo-mixins:hooks DEBUG clean start…
2019-11-25T17:53:46.878Z xo:xo-mixins:hooks DEBUG clean finished
  express:application set "x-powered-by" to true +0ms
  express:application set "etag" to 'weak' +2ms
  express:application set "etag fn" to [Function: generateETag] +0ms
  express:application set "env" to 'production' +1ms
  express:application set "query parser" to 'extended' +0ms
  express:application set "query parser fn" to [Function: parseExtendedQueryString] +0ms
  express:application set "subdomain offset" to 2 +0ms
  express:application set "trust proxy" to false +1ms
  express:application set "trust proxy fn" to [Function: trustNone] +0ms
  express:application booting in production mode +1ms
  express:application set "view" to [Function: View] +0ms
  express:application set "views" to '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/views' +0ms
  express:application set "jsonp callback name" to 'callback' +0ms
  express:application set "view cache" to true +1ms
  express:router use '/' query +2ms
  express:router:layer new '/' +0ms
  express:router use '/' expressInit +1ms
  express:router:layer new '/' +1ms
  express:router use '/' helmet +0ms
  express:router:layer new '/' +0ms
  express:router use '/' compression +1ms
  express:router:layer new '/' +0ms
  express:router use '/' cookieParser +1ms
  express:router:layer new '/' +0ms
Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
  express:router use '/' session +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +1ms
  express:router:layer new '/' +0ms
  express:router use '/' urlencodedParser +1ms
  express:router:layer new '/' +0ms
  express:router use '/' initialize +1ms
  express:router:layer new '/' +0ms
  express:router use '/' bound _handleHttpRequest +1ms
  express:router:layer new '/' +0ms
  express:router:route new '/signin' +114ms
  express:router:layer new '/signin' +1ms
  express:router:route get '/signin' +0ms
  express:router:layer new '/' +1ms
  express:router:route new '/signout' +0ms
  express:router:layer new '/signout' +0ms
  express:router:route get '/signout' +1ms
  express:router:layer new '/' +0ms
  express:router:route new '/signin-otp' +0ms
  express:router:layer new '/signin-otp' +0ms
  express:router:route get '/signin-otp' +1ms
  express:router:layer new '/' +0ms
  express:router:route new '/signin-otp' +0ms
  express:router:layer new '/signin-otp' +0ms
  express:router:route post '/signin-otp' +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +1ms
2019-11-25T17:53:47.018Z xo:main INFO Setting up / → /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-web/dist
  express:router use '/' serveStatic +2ms
  express:router:layer new '/' +0ms
2019-11-25T17:53:47.022Z xo:plugin INFO register auth-github
2019-11-25T17:53:47.044Z xo:plugin INFO register auth-google
2019-11-25T17:53:47.049Z xo:plugin INFO register auth-ldap
2019-11-25T17:53:47.157Z xo:plugin INFO register auth-saml
2019-11-25T17:53:47.250Z xo:plugin INFO register backup-reports
2019-11-25T17:53:47.254Z xo:plugin INFO register load-balancer
2019-11-25T17:53:47.263Z xo:plugin INFO register perf-alert
2019-11-25T17:53:47.266Z xo:plugin INFO register sdn-controller
2019-11-25T17:53:47.277Z xo:plugin INFO register test
2019-11-25T17:53:47.278Z xo:plugin INFO register test-plugin
2019-11-25T17:53:47.279Z xo:plugin INFO register transport-email
2019-11-25T17:53:47.319Z xo:plugin INFO register transport-icinga2
2019-11-25T17:53:47.320Z xo:plugin INFO register transport-nagios
2019-11-25T17:53:47.322Z xo:plugin INFO register transport-slack
2019-11-25T17:53:47.441Z xo:plugin INFO register transport-xmpp
2019-11-25T17:53:47.492Z xo:plugin INFO register usage-report
2019-11-25T17:53:47.600Z xo:plugin INFO register web-hooks
2019-11-25T17:53:47.603Z xo:plugin INFO failed register test
2019-11-25T17:53:47.603Z xo:plugin INFO Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test' { error: { Error: Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Xo.registerPlugin (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:259:17)
    at Xo.registerPluginWrapper (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:315:24)
    at Promise.all.name (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:340:37)
    at arrayMap (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/_arrayMap.js:16:21)
    at map (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/map.js:50:10)
    at Xo.registerPluginsInPath (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:338:15)
    at <anonymous> code: 'MODULE_NOT_FOUND' } }
  xen-api root@10.69.0.200: event.from(...) [712ms] ==> object +994ms
2019-11-25T17:53:47.666Z xo:xo-server:sdn-controller DEBUG No cert-dir provided, using default self-signed certificates
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-github
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-google
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-ldap
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-saml
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register test-plugin
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register transport-email
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-icinga2
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-nagios
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-slack
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-xmpp
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register usage-report
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register web-hooks
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register backup-reports
2019-11-25T17:53:47.684Z xo:plugin INFO successfully register load-balancer
2019-11-25T17:53:47.685Z xo:plugin INFO successfully register perf-alert
2019-11-25T17:53:47.742Z xo:plugin INFO successfully register sdn-controller

Looks like some unexpected files lying about plus some unexpected behavior:

Is the .xo-server.toml supposed to be there?
Could we add a warning to doc set that xo-server seems to load any file named *.toml.*?

That said I think it's fixed, if there is something I'm not seeing as a consequence of moving the .xo-server.toml file out, please, please let me know. Here is the contents of the .xo-server.toml file:

# Example XO-Server configuration.
#
# This file is automatically looking for at the following places:
# - `$HOME/.config/xo-server/config.toml`
# - `/etc/xo-server/config.toml`
#
# The first entries have priority.
#
# Note: paths are relative to the configuration file.

#=====================================================================

# HTTP proxy configuration used by xo-server to fetch resources on the Internet.
#
# See: https://github.com/TooTallNate/node-proxy-agent#maps-proxy-protocols-to-httpagent-implementations
# httpProxy = 'http://jsmith:qwerty@proxy.lan:3128'

#=====================================================================

# It may be necessary to run XO-Server as a privileged user (e.g. `root`) for
# instance to allow the HTTP server to listen on a
# [privileged ports](http://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html).
#
# To avoid security issues, XO-Server can drop its privileges by changing the
# user and the group is running with.
#
# Note: XO-Server will change them just after reading the configuration.

# User to run XO-Server as.
#
# Note: The user can be specified using either its name or its numeric
# identifier.
#
# Default: undefined
#user = 'nobody'

# Group to run XO-Server as.
#
# Note: The group can be specified using either its name or its numeric
# identifier.
#
# Default: undefined
# group = 'nogroup'

#=====================================================================

# Directory containing the database of XO.
# Currently used for logs.
#
# Default: '/var/lib/xo-server/data'
#datadir = '/var/lib/xo-server/data'

#=====================================================================

# Configuration of the embedded HTTP server.
[http]
# If set to true, all HTTP traffic will be redirected to the first HTTPs
# configuration.
# redirectToHttps = true

# Settings applied to cookies created by xo-server's embedded HTTP server.
#
# See https://www.npmjs.com/package/cookie#options-1
[http.cookies]
#sameSite = true
#secure = true

# Basic HTTP.
[[http.listen]]
# Address on which the server is listening on.
#
# Sets it to 'localhost' for IP to listen only on the local host.
#
# Default: all IPv6 addresses if available, otherwise all IPv4 addresses.
# hostname = 'localhost'

# Port on which the server is listening on.
#
# Default: undefined
port = 80

# Instead of `host` and `port` a path to a UNIX socket may be specified
# (overrides `host` and `port`).
#
# Default: undefined
# socket = './http.sock'

# # Basic HTTPS.
# #
# # You can find the list of possible options there
# # https://nodejs.org/docs/latest/api/tls.html#tls.createServer
# #
# # The only difference is the presence of the certificate and the key.
# [[http.listen]]
# #hostname = '127.0.0.1'
# port = 443
#
# # File containing the certificate (PEM format).
# #
# # If a chain of certificates authorities is needed, you may bundle them
# # directly in the certificate.
# #
# # Note: the order of certificates does matter, your certificate should come
# # first followed by the certificate of the above
# # certificate authority up to the root.
# #
# # Default: undefined
# cert = './certificate.pem'
#
# # File containing the private key (PEM format).
# #
# # If the key is encrypted, the passphrase will be asked at
# # server startup.
# #
# # Default: undefined
# key = './key.pem'

# List of files/directories which will be served.
[http.mounts]
#'/any/url' = '/path/to/directory'

# List of proxied URLs (HTTP & WebSockets).
[http.proxies]
#'/any/url' = 'http://localhost:54722'

#=====================================================================

# Connection to the Redis server.
[redis]
# Unix sockets can be used
#
# Default: undefined
#socket = '/var/run/redis/redis.sock'

# Syntax: redis://[db[:password]@]hostname[:port][/db-number]
#
# Default: redis://localhost:6379/0
#uri = 'redis://redis.company.lan/42'

# List of aliased commands.
#
# See http://redis.io/topics/security#disabling-of-specific-commands
#renameCommands:
#  del = '3dda29ad-3015-44f9-b13b-fa570de92489'
#  srem = '3fd758c9-5610-4e9d-a058-dbf4cb6d8bf0'

#=====================================================================

# Configuration for remotes
[remoteOptions]
# Directory used to mount remotes
#
# Default: '/run/xo-server/mounts'
#mountsDir = '/run/xo-server/mounts'

# Use sudo for mount with non-root user
#
# Default: false
#useSudo = false

Thanks to Damp, olivierlambert and to julien-f for the quick help.

bill.gertz

@olivierlambert @julien-f @Danp

Looks like the .xo-server.toml is installed by the the well known installer script, from xo-install.sh script:

.
.
.
echo -e "${INFO} Activating modified configuration file"
                mv $INSTALLDIR/xo-builds/xen-orchestra-$TIME/packages/xo-server/sample.config.toml $INSTALLDIR/xo-builds/xen-orchestra-$TIME/packages/xo-server/.xo-server.toml
.
.
.

Looks like we either need to document the 'load any toml' behavior or get the other script to declare uncle and let your script drop down the default config.

olivierlambert

That's why we prefer to let people follow our official documentation to install XO: there's less potential issues than with 3rd party scripts we have no control over it.

For the doc itself, we'll improve it (you can do a PR if you like, it's all in markdown! )

bill.gertz

@olivierlambert

Thanks again for the help, and will submit a PR but under which repo for the documentation? Looks like your using GitBook and I'm clueless on submitting PRs on GitBook. Is there some sort of way to submit changes to your doc set from within GitBook?

I ran into the problem anyway due to the load any toml behavior. xo-server startup loaded my backup of the original config (config.toml.orig). That was not really expected.

julien-f

This is already correctly documented: https://github.com/vatesfr/xen-orchestra/blob/377552103eabbf99d12101136fc8b37eb8f2dcf4/packages/xo-server/sample.config.toml#L3-L5

Avoid using .xo-server.toml as they are relative to the current working directory (similar to other command tools like Git).

bill.gertz

@julien-f @olivierlambert

Thanks for getting back to me, not trying to pick a fight. Loading config.toml.orig is not clear based on what the comments say:

# This file is automatically looking for at the following places:
# - `$HOME/.config/xo-server/config.toml`
# - `/etc/xo-server/config.toml`

xo-server is loading config.toml.orig that just doesn't seem to square with the comments. If it's documented elsewhere in the file and I just missed it please help me find it Just the same shouldn't this behavior be documented in https://xen-orchestra.com/docs/configuration.html? I can't be the only person to make .orig style backup copies before making changes?

julien-f

@bill-gertz Indeed I missed this, xo-server loads all files named config.* with a known extension.

I have to take a look at why it loads a .orig.

bill.gertz

@julien-f

Along with .xo-server.toml as you pointed out earlier and I've experienced :-/. Thanks again for looking into this.

julien-f

@bill-gertz AFAICT, this is a log issue, the file .orig is not taken into account for the configuration, I'll fix this.

Regarding .xo-server.toml, I will not change anything, it's used by some devs in some specific setups, but it's not documented as it does not make sense for user to do this.

bill.gertz

@julien-f

Again not trying to pick a fight but your own documentation says to use .xo-server.toml here when compiling from the sources.

Shouldn't this be changed from:

Now you have to create a config file for xo-server:

$ cd packages/xo-server
$ cp sample.config.toml .xo-server.toml

To something like:

Now you have to create a config file for xo-server:

$ cd packages/xo-server
$ cp sample.config.toml config.toml

This is what happened when the 3rd party script ran. I can submit PRs for the GitBook docs, but I don't see anyway of doing that from GitBooks. Is there a repo for this GitBook doc in which I can submit a PR? I'll be happy to submit a PR to the 3rd party script repo as well.

julien-f

@bill-gertz said in XOA won't start HTTPS listen on configured port:

Again not trying to pick a fight

Don't worry about this

@bill-gertz said in XOA won't start HTTPS listen on configured port:

your own documentation says to use .xo-server.toml here when compiling from the sources

Hmm, indeed, I will change this to avoid any issues in the future.

@bill-gertz said in XOA won't start HTTPS listen on configured port:

Is there a repo for this GitBook doc in which I can submit a PR?

https://github.com/vatesfr/xen-orchestra/tree/master/docs

bill.gertz

@julien-f

Thanks, I'll lay my cards more directly on the table next time. Glad you're not taking it personally. I'll use that repo for PRs. Thanks again.

julien-f

@bill-gertz I've made the change, the documentation will update soon: https://github.com/vatesfr/xen-orchestra/commit/7b47e4024492d38bca6e928748490e0a4880264c

0

julien-f committed to vatesfr/xen-orchestra

fix(docs/from sources): dont use `.xo-server.toml`

olivierlambert

Thanks for the feedback @bill-gertz !

Thanks for the doc update @julien-f