XOA won't start HTTPS listen on configured port

bill.gertz

Winner, winner, chicken dinner. The debug trick was the key to see WTF it was loading. Moved those two pigs out of the directory to an a hidden directory off of the parent and it starts as it's supposed to. Here is the same debug output with those two files out of the way:

root@bfxoa001:/etc/xo/xo-server# systemctl stop xo-server.service 
root@bfxoa001:/etc/xo/xo-server# env DEBUG='*' ./bin/xo-server
  app-conf /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/config.toml +0ms
2019-11-25T17:53:45.979Z xo:main INFO Configuration loaded.
2019-11-25T17:53:45.998Z xo:main INFO Web server listening on https://0.0.0.0:443
2019-11-25T17:53:46.105Z xo:xo-mixins:hooks DEBUG start start…
  xen-api root@10.69.0.200: session.login_with_password(...) [58ms] ==> string +0ms
  xen-api root@10.69.0.200: pool.get_all_records(...) [7ms] ==> object +8ms
  xen-api root@10.69.0.200: system.listMethods(...) [6ms] ==> array +8ms
  xen-api root@10.69.0.200: connected +1ms
  xen-api root@10.69.0.200: event.inject(...) [6ms] ==> string +8ms
  xen-api root@10.69.0.200: subject.get_all_records(...) [81ms] ==> object +82ms
  xen-api root@10.69.0.200: pool_patch.get_all_records(...) [90ms] ==> object +12ms
  xen-api root@10.69.0.200: VM_appliance.get_all_records(...) [86ms] ==> object +2ms
  xen-api root@10.69.0.200: pool_update.get_all_records(...) [144ms] ==> object +53ms
  xen-api root@10.69.0.200: VIF_metrics.get_all_records(...) [148ms] =!> XapiError: MESSAGE_REMOVED() +15ms
  xen-api root@10.69.0.200: DR_task.get_all_records(...) [158ms] ==> object +4ms
  xen-api root@10.69.0.200: VLAN.get_all_records(...) [150ms] ==> object +1ms
  xen-api root@10.69.0.200: tunnel.get_all_records(...) [141ms] ==> object +1ms
  xen-api root@10.69.0.200: pool.get_all_records(...) [167ms] ==> object +0ms
  xen-api root@10.69.0.200: PIF_metrics.get_all_records(...) [152ms] ==> object +0ms
  xen-api root@10.69.0.200: VM.get_all_records(...) [179ms] ==> object +15ms
  xen-api root@10.69.0.200: role.get_all_records(...) [220ms] ==> object +37ms
  xen-api root@10.69.0.200: USB_group.get_all_records(...) [244ms] ==> object +62ms
  xen-api root@10.69.0.200: host_metrics.get_all_records(...) [271ms] ==> object +0ms
  xen-api root@10.69.0.200: PCI.get_all_records(...) [254ms] ==> object +0ms
  xen-api root@10.69.0.200: task.get_all_records(...) [281ms] ==> object +0ms
  xen-api root@10.69.0.200: VBD.get_all_records(...) [262ms] ==> object +1ms
  xen-api root@10.69.0.200: VMSS.get_all_records(...) [276ms] ==> object +0ms
  xen-api root@10.69.0.200: PBD.get_all_records(...) [261ms] ==> object +0ms
  xen-api root@10.69.0.200: SM.get_all_records(...) [265ms] ==> object +0ms
  xen-api root@10.69.0.200: Cluster.get_all_records(...) [244ms] ==> object +1ms
  xen-api root@10.69.0.200: VM_metrics.get_all_records(...) [280ms] ==> object +0ms
  xen-api root@10.69.0.200: host.get_all_records(...) [288ms] ==> object +13ms
  xen-api root@10.69.0.200: console.get_all_records(...) [273ms] ==> object +0ms
  xen-api root@10.69.0.200: VBD_metrics.get_all_records(...) [276ms] =!> XapiError: MESSAGE_REMOVED() +0ms
  xen-api root@10.69.0.200: host_crashdump.get_all_records(...) [288ms] ==> object +1ms
  xen-api root@10.69.0.200: VM_guest_metrics.get_all_records(...) [293ms] ==> object +0ms
  xen-api root@10.69.0.200: SDN_controller.get_all_records(...) [262ms] ==> object +0ms
  xen-api root@10.69.0.200: VMPP.get_all_records(...) [292ms] =!> XapiError: MESSAGE_REMOVED() +0ms
  xen-api root@10.69.0.200: secret.get_all_records(...) [273ms] ==> object +1ms
  xen-api root@10.69.0.200: blob.get_all_records(...) [275ms] ==> object +0ms
  xen-api root@10.69.0.200: host_patch.get_all_records(...) [288ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_cache_storage.get_all_records(...) [264ms] ==> object +1ms
  xen-api root@10.69.0.200: VUSB.get_all_records(...) [261ms] ==> object +0ms
  xen-api root@10.69.0.200: network_sriov.get_all_records(...) [272ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_server.get_all_records(...) [267ms] ==> object +0ms
  xen-api root@10.69.0.200: SR.get_all_records(...) [282ms] ==> object +1ms
  xen-api root@10.69.0.200: VGPU.get_all_records(...) [270ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_proxy.get_all_records(...) [267ms] ==> object +0ms
  xen-api root@10.69.0.200: VGPU_type.get_all_records(...) [269ms] ==> object +0ms
  xen-api root@10.69.0.200: crashdump.get_all_records(...) [279ms] ==> object +1ms
  xen-api root@10.69.0.200: Feature.get_all_records(...) [267ms] ==> object +0ms
  xen-api root@10.69.0.200: PVS_site.get_all_records(...) [270ms] ==> object +0ms
  xen-api root@10.69.0.200: Cluster_host.get_all_records(...) [261ms] ==> object +0ms
  xen-api root@10.69.0.200: GPU_group.get_all_records(...) [273ms] ==> object +1ms
  xen-api root@10.69.0.200: network.get_all_records(...) [290ms] ==> object +0ms
  xen-api root@10.69.0.200: PUSB.get_all_records(...) [266ms] ==> object +0ms
  xen-api root@10.69.0.200: Bond.get_all_records(...) [286ms] ==> object +0ms
  xen-api root@10.69.0.200: host_cpu.get_all_records(...) [292ms] ==> object +1ms
  xen-api root@10.69.0.200: PGPU.get_all_records(...) [275ms] ==> object +0ms
  xen-api root@10.69.0.200: PIF.get_all_records(...) [289ms] ==> object +0ms
  xen-api root@10.69.0.200: VIF.get_all_records(...) [290ms] ==> object +0ms
  xen-api root@10.69.0.200: VDI.get_all_records(...) [285ms] ==> object +1ms
2019-11-25T17:53:46.616Z xo:xo-mixins:hooks DEBUG start finished
  xen-api root@10.69.0.200: message.get_all_records(...) [290ms] ==> object +10ms
2019-11-25T17:53:46.700Z xo:xo-mixins:hooks DEBUG clean start…
2019-11-25T17:53:46.878Z xo:xo-mixins:hooks DEBUG clean finished
  express:application set "x-powered-by" to true +0ms
  express:application set "etag" to 'weak' +2ms
  express:application set "etag fn" to [Function: generateETag] +0ms
  express:application set "env" to 'production' +1ms
  express:application set "query parser" to 'extended' +0ms
  express:application set "query parser fn" to [Function: parseExtendedQueryString] +0ms
  express:application set "subdomain offset" to 2 +0ms
  express:application set "trust proxy" to false +1ms
  express:application set "trust proxy fn" to [Function: trustNone] +0ms
  express:application booting in production mode +1ms
  express:application set "view" to [Function: View] +0ms
  express:application set "views" to '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/views' +0ms
  express:application set "jsonp callback name" to 'callback' +0ms
  express:application set "view cache" to true +1ms
  express:router use '/' query +2ms
  express:router:layer new '/' +0ms
  express:router use '/' expressInit +1ms
  express:router:layer new '/' +1ms
  express:router use '/' helmet +0ms
  express:router:layer new '/' +0ms
  express:router use '/' compression +1ms
  express:router:layer new '/' +0ms
  express:router use '/' cookieParser +1ms
  express:router:layer new '/' +0ms
Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
  express:router use '/' session +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +1ms
  express:router:layer new '/' +0ms
  express:router use '/' urlencodedParser +1ms
  express:router:layer new '/' +0ms
  express:router use '/' initialize +1ms
  express:router:layer new '/' +0ms
  express:router use '/' bound _handleHttpRequest +1ms
  express:router:layer new '/' +0ms
  express:router:route new '/signin' +114ms
  express:router:layer new '/signin' +1ms
  express:router:route get '/signin' +0ms
  express:router:layer new '/' +1ms
  express:router:route new '/signout' +0ms
  express:router:layer new '/signout' +0ms
  express:router:route get '/signout' +1ms
  express:router:layer new '/' +0ms
  express:router:route new '/signin-otp' +0ms
  express:router:layer new '/signin-otp' +0ms
  express:router:route get '/signin-otp' +1ms
  express:router:layer new '/' +0ms
  express:router:route new '/signin-otp' +0ms
  express:router:layer new '/signin-otp' +0ms
  express:router:route post '/signin-otp' +1ms
  express:router:layer new '/' +0ms
  express:router use '/' <anonymous> +0ms
  express:router:layer new '/' +1ms
2019-11-25T17:53:47.018Z xo:main INFO Setting up / → /etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-web/dist
  express:router use '/' serveStatic +2ms
  express:router:layer new '/' +0ms
2019-11-25T17:53:47.022Z xo:plugin INFO register auth-github
2019-11-25T17:53:47.044Z xo:plugin INFO register auth-google
2019-11-25T17:53:47.049Z xo:plugin INFO register auth-ldap
2019-11-25T17:53:47.157Z xo:plugin INFO register auth-saml
2019-11-25T17:53:47.250Z xo:plugin INFO register backup-reports
2019-11-25T17:53:47.254Z xo:plugin INFO register load-balancer
2019-11-25T17:53:47.263Z xo:plugin INFO register perf-alert
2019-11-25T17:53:47.266Z xo:plugin INFO register sdn-controller
2019-11-25T17:53:47.277Z xo:plugin INFO register test
2019-11-25T17:53:47.278Z xo:plugin INFO register test-plugin
2019-11-25T17:53:47.279Z xo:plugin INFO register transport-email
2019-11-25T17:53:47.319Z xo:plugin INFO register transport-icinga2
2019-11-25T17:53:47.320Z xo:plugin INFO register transport-nagios
2019-11-25T17:53:47.322Z xo:plugin INFO register transport-slack
2019-11-25T17:53:47.441Z xo:plugin INFO register transport-xmpp
2019-11-25T17:53:47.492Z xo:plugin INFO register usage-report
2019-11-25T17:53:47.600Z xo:plugin INFO register web-hooks
2019-11-25T17:53:47.603Z xo:plugin INFO failed register test
2019-11-25T17:53:47.603Z xo:plugin INFO Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test' { error: { Error: Cannot find module '/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/dist/../node_modules//xo-server-test'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Xo.registerPlugin (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:259:17)
    at Xo.registerPluginWrapper (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:315:24)
    at Promise.all.name (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:340:37)
    at arrayMap (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/_arrayMap.js:16:21)
    at map (/etc/xo/xo-builds/xen-orchestra-201925111145/node_modules/lodash/map.js:50:10)
    at Xo.registerPluginsInPath (/etc/xo/xo-builds/xen-orchestra-201925111145/packages/xo-server/src/index.js:338:15)
    at <anonymous> code: 'MODULE_NOT_FOUND' } }
  xen-api root@10.69.0.200: event.from(...) [712ms] ==> object +994ms
2019-11-25T17:53:47.666Z xo:xo-server:sdn-controller DEBUG No cert-dir provided, using default self-signed certificates
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-github
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-google
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-ldap
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register auth-saml
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register test-plugin
2019-11-25T17:53:47.682Z xo:plugin INFO successfully register transport-email
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-icinga2
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-nagios
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-slack
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register transport-xmpp
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register usage-report
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register web-hooks
2019-11-25T17:53:47.683Z xo:plugin INFO successfully register backup-reports
2019-11-25T17:53:47.684Z xo:plugin INFO successfully register load-balancer
2019-11-25T17:53:47.685Z xo:plugin INFO successfully register perf-alert
2019-11-25T17:53:47.742Z xo:plugin INFO successfully register sdn-controller

Looks like some unexpected files lying about plus some unexpected behavior:

Is the .xo-server.toml supposed to be there?
Could we add a warning to doc set that xo-server seems to load any file named *.toml.*?

That said I think it's fixed, if there is something I'm not seeing as a consequence of moving the .xo-server.toml file out, please, please let me know. Here is the contents of the .xo-server.toml file:

# Example XO-Server configuration.
#
# This file is automatically looking for at the following places:
# - `$HOME/.config/xo-server/config.toml`
# - `/etc/xo-server/config.toml`
#
# The first entries have priority.
#
# Note: paths are relative to the configuration file.

#=====================================================================

# HTTP proxy configuration used by xo-server to fetch resources on the Internet.
#
# See: https://github.com/TooTallNate/node-proxy-agent#maps-proxy-protocols-to-httpagent-implementations
# httpProxy = 'http://jsmith:qwerty@proxy.lan:3128'

#=====================================================================

# It may be necessary to run XO-Server as a privileged user (e.g. `root`) for
# instance to allow the HTTP server to listen on a
# [privileged ports](http://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html).
#
# To avoid security issues, XO-Server can drop its privileges by changing the
# user and the group is running with.
#
# Note: XO-Server will change them just after reading the configuration.

# User to run XO-Server as.
#
# Note: The user can be specified using either its name or its numeric
# identifier.
#
# Default: undefined
#user = 'nobody'

# Group to run XO-Server as.
#
# Note: The group can be specified using either its name or its numeric
# identifier.
#
# Default: undefined
# group = 'nogroup'

#=====================================================================

# Directory containing the database of XO.
# Currently used for logs.
#
# Default: '/var/lib/xo-server/data'
#datadir = '/var/lib/xo-server/data'

#=====================================================================

# Configuration of the embedded HTTP server.
[http]
# If set to true, all HTTP traffic will be redirected to the first HTTPs
# configuration.
# redirectToHttps = true

# Settings applied to cookies created by xo-server's embedded HTTP server.
#
# See https://www.npmjs.com/package/cookie#options-1
[http.cookies]
#sameSite = true
#secure = true

# Basic HTTP.
[[http.listen]]
# Address on which the server is listening on.
#
# Sets it to 'localhost' for IP to listen only on the local host.
#
# Default: all IPv6 addresses if available, otherwise all IPv4 addresses.
# hostname = 'localhost'

# Port on which the server is listening on.
#
# Default: undefined
port = 80

# Instead of `host` and `port` a path to a UNIX socket may be specified
# (overrides `host` and `port`).
#
# Default: undefined
# socket = './http.sock'

# # Basic HTTPS.
# #
# # You can find the list of possible options there
# # https://nodejs.org/docs/latest/api/tls.html#tls.createServer
# #
# # The only difference is the presence of the certificate and the key.
# [[http.listen]]
# #hostname = '127.0.0.1'
# port = 443
#
# # File containing the certificate (PEM format).
# #
# # If a chain of certificates authorities is needed, you may bundle them
# # directly in the certificate.
# #
# # Note: the order of certificates does matter, your certificate should come
# # first followed by the certificate of the above
# # certificate authority up to the root.
# #
# # Default: undefined
# cert = './certificate.pem'
#
# # File containing the private key (PEM format).
# #
# # If the key is encrypted, the passphrase will be asked at
# # server startup.
# #
# # Default: undefined
# key = './key.pem'

# List of files/directories which will be served.
[http.mounts]
#'/any/url' = '/path/to/directory'

# List of proxied URLs (HTTP & WebSockets).
[http.proxies]
#'/any/url' = 'http://localhost:54722'

#=====================================================================

# Connection to the Redis server.
[redis]
# Unix sockets can be used
#
# Default: undefined
#socket = '/var/run/redis/redis.sock'

# Syntax: redis://[db[:password]@]hostname[:port][/db-number]
#
# Default: redis://localhost:6379/0
#uri = 'redis://redis.company.lan/42'

# List of aliased commands.
#
# See http://redis.io/topics/security#disabling-of-specific-commands
#renameCommands:
#  del = '3dda29ad-3015-44f9-b13b-fa570de92489'
#  srem = '3fd758c9-5610-4e9d-a058-dbf4cb6d8bf0'

#=====================================================================

# Configuration for remotes
[remoteOptions]
# Directory used to mount remotes
#
# Default: '/run/xo-server/mounts'
#mountsDir = '/run/xo-server/mounts'

# Use sudo for mount with non-root user
#
# Default: false
#useSudo = false

Thanks to Damp, olivierlambert and to julien-f for the quick help.

bill.gertz

@olivierlambert @julien-f @Danp

Looks like the .xo-server.toml is installed by the the well known installer script, from xo-install.sh script:

.
.
.
echo -e "${INFO} Activating modified configuration file"
                mv $INSTALLDIR/xo-builds/xen-orchestra-$TIME/packages/xo-server/sample.config.toml $INSTALLDIR/xo-builds/xen-orchestra-$TIME/packages/xo-server/.xo-server.toml
.
.
.

Looks like we either need to document the 'load any toml' behavior or get the other script to declare uncle and let your script drop down the default config.

olivierlambert

That's why we prefer to let people follow our official documentation to install XO: there's less potential issues than with 3rd party scripts we have no control over it.

For the doc itself, we'll improve it (you can do a PR if you like, it's all in markdown! )

bill.gertz

@olivierlambert

Thanks again for the help, and will submit a PR but under which repo for the documentation? Looks like your using GitBook and I'm clueless on submitting PRs on GitBook. Is there some sort of way to submit changes to your doc set from within GitBook?

I ran into the problem anyway due to the load any toml behavior. xo-server startup loaded my backup of the original config (config.toml.orig). That was not really expected.

julien-f

This is already correctly documented: https://github.com/vatesfr/xen-orchestra/blob/377552103eabbf99d12101136fc8b37eb8f2dcf4/packages/xo-server/sample.config.toml#L3-L5

Avoid using .xo-server.toml as they are relative to the current working directory (similar to other command tools like Git).

bill.gertz

@julien-f @olivierlambert

Thanks for getting back to me, not trying to pick a fight. Loading config.toml.orig is not clear based on what the comments say:

# This file is automatically looking for at the following places:
# - `$HOME/.config/xo-server/config.toml`
# - `/etc/xo-server/config.toml`

xo-server is loading config.toml.orig that just doesn't seem to square with the comments. If it's documented elsewhere in the file and I just missed it please help me find it Just the same shouldn't this behavior be documented in https://xen-orchestra.com/docs/configuration.html? I can't be the only person to make .orig style backup copies before making changes?

julien-f

@bill-gertz Indeed I missed this, xo-server loads all files named config.* with a known extension.

I have to take a look at why it loads a .orig.

bill.gertz

@julien-f

Along with .xo-server.toml as you pointed out earlier and I've experienced :-/. Thanks again for looking into this.

julien-f

@bill-gertz AFAICT, this is a log issue, the file .orig is not taken into account for the configuration, I'll fix this.

Regarding .xo-server.toml, I will not change anything, it's used by some devs in some specific setups, but it's not documented as it does not make sense for user to do this.

bill.gertz

@julien-f

Again not trying to pick a fight but your own documentation says to use .xo-server.toml here when compiling from the sources.

Shouldn't this be changed from:

Now you have to create a config file for xo-server:

$ cd packages/xo-server
$ cp sample.config.toml .xo-server.toml

To something like:

Now you have to create a config file for xo-server:

$ cd packages/xo-server
$ cp sample.config.toml config.toml

This is what happened when the 3rd party script ran. I can submit PRs for the GitBook docs, but I don't see anyway of doing that from GitBooks. Is there a repo for this GitBook doc in which I can submit a PR? I'll be happy to submit a PR to the 3rd party script repo as well.

julien-f

@bill-gertz said in XOA won't start HTTPS listen on configured port:

Again not trying to pick a fight

Don't worry about this

@bill-gertz said in XOA won't start HTTPS listen on configured port:

your own documentation says to use .xo-server.toml here when compiling from the sources

Hmm, indeed, I will change this to avoid any issues in the future.

@bill-gertz said in XOA won't start HTTPS listen on configured port:

Is there a repo for this GitBook doc in which I can submit a PR?

https://github.com/vatesfr/xen-orchestra/tree/master/docs

bill.gertz

@julien-f

Thanks, I'll lay my cards more directly on the table next time. Glad you're not taking it personally. I'll use that repo for PRs. Thanks again.

julien-f

@bill-gertz I've made the change, the documentation will update soon: https://github.com/vatesfr/xen-orchestra/commit/7b47e4024492d38bca6e928748490e0a4880264c

0

julien-f committed to vatesfr/xen-orchestra

fix(docs/from sources): dont use `.xo-server.toml`

olivierlambert

Thanks for the feedback @bill-gertz !

Thanks for the doc update @julien-f

bill.gertz

@olivierlambert @julien-f @Danp

Thanks to everyone for the clear, helpful support. Impressed.

bill.gertz

@julien-f @olivierlambert

Submitted PR to change attribute documented as 'host' to 'hostname'.
https://github.com/vatesfr/xen-orchestra/pull/4681

billgertz opened this pull request in vatesfr/xen-orchestra

closed fix(docs/configuration): `host` → `hostname` #4681

ronivay

Hi,

@julien-f from sources documentation still instructs to set path relative to packages/xo-server like it used to be:

[http.mounts]
'/' = '../xo-web/dist/'

sample.config.toml says that paths are relative to configuration file so this would end up looking xo-web/dist path from $HOME/.config/ since configuration is at $HOME/.config/xo-server/config.toml

Noticed this while fixing my "well known installer script"

julien-f

@ronivay Thanks for the report, I'll change this by make it part of the default config, it won't be necessary for the user to configure it themself.

julien-f

In fact it's already the case, I just need to remove this from the documentation

https://github.com/vatesfr/xen-orchestra/blob/14f0cbaec6244d66266fe2ee47b0bfb5716c2f61/packages/xo-server/config.toml#L99-L100

julien-f

Done: https://github.com/vatesfr/xen-orchestra/commit/14f0cbaec6244d66266fe2ee47b0bfb5716c2f61

0

julien-f committed to vatesfr/xen-orchestra

fix(docs/from sources): no need to configure `/` mount point