Xen Orchestra from source with Let's Encrypt certificates

gskger

Using CA issued certificates on Xen Orchestra from source (xo) improves security and browser compatibility. Creating Let's Encrypt (LE) certificates for Xen Orchestra is simple using pfsense with the acme plugin. It is a two-step process controlled by the acme plugin on pfsense:

request/renew LE certificates
copy certificates to xo and restart the xo-server

The acme plugin offers several methods for step 1. I am using the Webroot FTP method because I do not want to expose services to the internet and my domain provider (Strato AG) does not offer DNS and/or API based LE support.
Step 2 is the same for all methods in this example. With key-based authentication a ssh user (e.g. acme) will copy LE certificates from pfsense to xo and restart the xo-server

For this post I skip step 1 since it highly depends on your situation, but if acme has been setup correctly you end up with domain specific LE certificates in the /conf/acme/ folder on pfsense. xo.myplaylab.net is my Xen Orchestra from source VM and xo.myplaylab.net.key and xo.myplaylab.net.crt are the LE certificates from step 1.
Step 2 needs some preparation before the acme plugin can automate this step.

Prepare Xen Orchestra from source to receive certificates from pfsense

create user acme on xo.myplaylab.net
make acme member of sudo
limit acme sudo rights to restart xo-server only
create certificate folder on xo (e.g. /etc/acme/ssl)
change folder owner to acme
optional: make xo-servers non-privileged user member of acme if not run as 'root'.

Prepare pfsense to send certificates to xo via ssh

setup key-based ssh connection on pfsense to ssh into xo
copy ssh-keys to xo for user acme@xo.myplaylab.net using ssh-copy-id
ssh into xo with user acme@xo.myplaylab.net and complete process
check that you can scp the certificates from pfsense to the certification folder on xo
check that you can ssh into xo from pfsense and restart the xo-server
these manual tasks will later be executed by the acme plugin.

Prepare Xen Orchestra to use the LE certificates

make sure that the certificates have been copied to /etc/acme/ssl
edit /opt/xen-orchestra/packages/xo-server/.xo-server.toml (maybe not the right file )
uncomment [[http.listen]] and port = 443 in the Basic HTTPS section
uncomment and change cert to cert = '/etc/acme/ssl/xo.myplaylab.net.crt'
uncomment and change key to key = '/etc/acme/ssl/xo.myplaylab.net.key'
uncomment redirectToHttps = true
restart xo-server and check the status

The final step is to add the shell commands to the xo.myplaylab.net acme configuration. Edit the configuration on pfsense and add the shell commands to the action list.

scp /conf/acme/xo.myplaylab.net.crt /conf/acme/xo.myplaylab.net.key acme@xo.myplaylab.net:/etc/acme/ssl/
ssh acme@xo.myplaylab.net "sudo /bin/systemctl restart xo-server"

Did a complete rewrite to be more clear on the interaction between pfsense and xo.

olivierlambert

Ping @julien-f about XO, if we can have this documented and if it's the best way

Regarding XCP-ng, I'm aware about some related work from XAPI developers. Pinging @psafont

fohdeesha

I think when I last asked @julien-f about this he said it work work (just as you described), but the issue is xo-server will not reload certs without restarting the process. So the next time your let's encrypt instance updates those certs, xo-server will have no idea and you'll need to schedule a restart of that service after certs are updated

gskger

@fohdeesha just realized that I forgot to mention two steps (key- based ssh authentification from pfsense to xo for user acme@xo.myplaylab.net and making acmemember of sudo on xowith limited sudo rights). The acme plugin configuration allows adding shell commands which are executed after the LE certification renewal (e.g. ssh acme@xo.myplaylab.net "sudo /bin/systemctl restart xo-server" to force xo-server to reload the certififcates).

Wow, this now reads more complicated than it realy is

gskger

Did a complete rewrite with better structure and focus on the interaction between pfsense and Xen Orchestra. Skipped the part of setting up the acme plugin for Webroot FTP as well, because it is very specific to my setup.

gskger

@olivierlambert Regarding XCP-ng - just generated a Let's Encrypt certificate for one of my xcp-ng test hosts xcp01.myplaylab.net (XCP-ng 8.2 beta fully patched), replaced the /etc/xensource/xapi-ssl.pem with the LE certificatexcp01.myplaylab.net.all.pem while keeping the name xapi-ssl.pem and did a xe-toolstack-restart. That seems to work, since I get a valid LE certificate and secure connection when accessing https://xcp01.myplaylab.net. Could it be that simple ? Do not have a pool and xo available right now, so just wondering...

I was just too curious - the xcp-ng test host can be added to Xen Orchestra with the LE certificate - no need to accept unauthorized certificates anymore

olivierlambert

Now you can even replace the cert via XO web UI (see https://xen-orchestra.com/blog/xen-orchestra-5-52/#xcpngxenserverhostscertificatesmanager)

stevewest15

Hi,

I'm testing using certbot to install SSL cert for XOCE using Lets Encrypt instructions but need to know what is the path for XOCE webroot?

Thx,

SW

fohdeesha

@stevewest15 those instructions will not work as XOA is running on node and does not have a physical "web root" folder like you are thinking of. Also @gskger please be cautious about scheduling xo-server restarts, doing so interrupts and breaks any task xo-server is running at that time, like backups etc. I believe this is one of the main reasons we haven't implemented let's encrypt integration into XOA, it's not as simple as just firing off an xo-server restart everytime LE certs are updated - this would break a lot of important backups for the majority of our customers. We would need to add some type of sensing to see xo-server's current status, and schedule the restart for when there's no longer any tasks running

gskger

@fohdeesha good to know, thank you. Maybe setting some xo flag through cli telling xo-server to restart when convinient might be a way to automate the process of cert updates?

kevdog

@gskger Why don't you install acme.sh or something similar on XO host and deploy from there. No need to use pfSense.

Andrew

@fohdeesha Does a HUP signal reload the config? Can it also check for updated certs?

It's not a restart it's just a config change check.

gskger

@kevdog Internal server can not be reached from the internet (no port forwarding so no HTTP challenge) and my hosting provider does not have an API for DNS challenges. Thats why I use pfSense "on the edge" at the moment. I admit that a cheap VPS runing acme.sh could do the trick, but my automation works and I am lazy .

olivierlambert

The real solution will be to get XOA as a certification authority and then manage all XCP-ng's hosts certs

gskger

Having XO from source or XOA act as a certification authority for the XCP-ng hosts is for sure a good approach. Would be great if that could include the VMs running on the XCP-ng hosts, which is my main goal (apart from being able to HTTPS into XO from source of course).