Encrypt Server Passwords

olivierlambert

Hi,

As a reminder, xo-server needs to connect on start to all the pool masters. It's not a simple client to XAPI, it's a real server that needs to stay connected anytime to a XAPI (eg for backups), and even reconnect if connection is lost.

How would you store something with an encrypted password when there's nobody to unlock it? Also how would you ask a user to type a password to check against an encrypted record, since there's no user 24/7 when xo-server starts or restarts or reconnect to a pool master?

Sure, you can have and store a passphrase that will unlock the record, but the passphrase will be also stored directly in the DB, so I'm not sure it will make it more secure (instead of getting the password, you need the passphrase and the password, but since it's stored at the same place, what's the point?).

Finally, XO, via its default "implementation", XOA, is meant to be a protected appliance, with nothing else running in the OS, to reduce the attack surface. Don't get your XOA compromised is very important.

If you have a solution, I'm happy to know it

jeffmetal

@olivierlambert Just having a bit of an investigate and not sure if this is actually possible or if it is but there is a requirement for a TPM 2.0.

Assuming you only support Linux and there is a modern systemd you can provide keys on process start https://systemd.io/CREDENTIALS/ the SetCredentialEncrypted option looks like what could be used.

Honestly never tested any of this and I'm running XO on xcp-ng 8.2 that doesn't support vTPM's yet.

jeffmetal

So you can do

#Check you have a TPM 2.0 otherwise this isn't going to work

systemd-creds has-tpm2

#Create plaintext.txt that contains your strong password.

echo "i]BM|yWq=7+-Be}n{9k=%26$O95V7"E$$G,+n&:!" > plaintext.txt

#generate a credentials file that is encrypted will be called ciphertext.cred the name we are using will be XO-PassPhrase

systemd-creds --name=XO-PassPhrase encrypt plaintext.txt ciphertext.cred

#Add this to your service file in the [servce] section

LoadCredential=XO-PassPhrase:/path/where/you/stored/ciphertext.cred

#Start the service and the encrypted passphrase is stored in $CREDENTIALS_DIRECTORY/XO-PassPhrase you can unencrypt it in the process using

systemd-creds decrypt $CREDENTIALS_DIRECTORY/XO-PassPhrase

The downsides are this is complex to setup, Needs modern Linux so works con Centos 9 but not 8, if the process is running you can grab the passphrase as root running this. (I was using creds-test as a service name to test this)

systemd-creds decrypt /run/credentials/creds-test.service/XO-PassPhrase

olivierlambert

This doesn't sound viable on a virtual appliance we distribute around the world without any control on the destination hardware nor even XCP-ng version.

Maybe when 8.3 will be out, after some years. But reducing the capacity (by a lot) to easily deploy an appliance is really not a great thing for us. Treat XOA as an appliance, it shouldn't be exposed outside and treated as any important appliance.

bettysweetss

Storing passwords in plaintext isn’t the best move, for sure. It’s a little concerning to see that in Redis. A while back, I had to set up a secure environment for some critical systems, and I got worried about how passwords were being stored too. I ended up using a pin code generator to create unique codes for access. It made me feel a lot better knowing those codes were simple to remember but still secure. It would be awesome if XO could look into adding at least basic encryption for these passwords. Even a simple passphrase could make a big difference.

olivierlambert

Sure but how you would do it then in our case? If there was a simple solution, it would be the case already.

Forza

@olivierlambert said in Encrypt Server Passwords:

Sure but how you would do it then in our case? If there was a simple solution, it would be the case already.

I think XOA could solve it by storing sensitive passwords or keys in an encrypted form in the XOA itself. An administrator can unlock access to these passwords/keys from the XOA web interface after boot of XOA. This could be problematic if XOA somehow is rebooted without anyone there to unlock things. If this feature is optional, organisations can decide for themselves what to use. Perhaps even use MFA or Fido keys, etc.

In any case, the passwords are only ever store in plain text in memory, and never on-disk with such a solution.

olivierlambert

This would hurt a lot the availability of the solution. If you have a xo-server restart (and indeed, also a XOA reboot), you'll need the key again to unlock the password to connect to the host and then if you missed it, you'll miss backups, load balancing and everything else.

So how do you get it without sacrificing the usability?

Forza

I think it could "survive" a xo-server restart, but not a full XOA restart, depending on how this is implemented.

But yes, this is why it needs to be an option/optional feature for those that feel the security implications are more important than the effort to staff someone to login.

john.c

@olivierlambert @Forza @jeffmetal @bettysweetss During deployment of XOA it could allow for the capacity to enable such a feature. As well as a configuration option in XOA for afterwards, if not enabled during deployment. Then as part of the same section there would be the option for the user to choose for where the encryption key for the credentials can be stored:-

TPM 2.0
KMIP protocol server
Tang server (https://github.com/latchset/tang)
AWS Secrets Manager
HashiCorp Vault (or any open source forks)
Google Secrets Manager
Azure Key Vault
IBM Cloud Secrets Manager
CyberArk Conjur

As well other options, but any way being able to have the encryption key go to one of these, then accessed from there when required. Also through an option the encrypted SR could also gain the option to have the capacity to get the keys going to the users choice of one of the list above.

The KMIP protocol server can come in two forms hardware and software. The software form comes as a piece of software which is installed on a device of your choosing or as a virtual appliance. The hardware form is a physical appliance, which you setup, connect to your network, configure and administer.

An example of an open source fork of HashiCorp Vault is the OpenBao (https://openbao.org/). It's a place where secrets can be sent using compatible software.

Some of the above will work in an airgap network environment, as they don't require an Internet connection to work. There's also the option to use an online third party service which is provided by companies which may have higher security than you, on your network; or if the other options of having it on your network isn't viable.

olivierlambert

I'm not saying it's not possible, I'm saying this will be at the huge expense of usability.

john.c

@olivierlambert said in Encrypt Server Passwords:

I'm not saying it's not possible, I'm saying this will be at the huge expense of usability.

One such software implementation of a KMIP protocol server, which is in software form is an open source fork of HashiCorp Vault. This open source fork is called OpenBao (https://openbao.org/).