self signed certificate in certificate chain

techjeff

I have a pool with 2 xcp-ng 8.2 hosts and on the same management network I have a debian 10 based VM within which I have built xo from sources using https://github.com/ronivay/XenOrchestraInstallerUpdater.

I had previously followed the guide TLS Certificate on XCP-NG as well as the guide for additional configuration to specify on XenOrchestra side, but I was getting Xapi#getResource /rrd_updates tasks lingering if I connected to my host when Unauthorized Certificates was disabled but not when connecting with that enabled.

Please correct me if I'm wrong, but I suspect the lingering task is caused by xo making an HTTPS request to the pool master for rrd updates but the request fails due to a cert error and the task isn't being cancelled and instead timing out after 24 hours as Olivier has noted in past threads.

To try to resolve this, today I created a snapshot at 1:11pm PDT and booted up a fast-clone of that snapshot so I am not left without a paddle if I were to break my xo during the update. I then used the build tool's update feature to pull from master and rebuild to see if it would make a difference.

Now, when I try to connect my xo to my pool master with Unauthorized Certificates disabled, I get this error: self signed certificate in certificate chain and xo is not able to connect to the pool. If I connect with Unauthorized Certificates enabled, I am able to get stats without the lingering rrd updates tasks, but I still want to be able to connet with Unauthorized Certificates disabled.

I reviewed the xo logs and saw Hostname/IP does not match certificate's altnames: IP: 10.0.60.12 is not in the cert's list:. My master's management IP is 10.0.10.12 and the 10.0.60.12 is the address on my storage network where my storage server is located. Full disclosure, I originally mis-read the IP address as the management address, so I decided to create a new certificate following the official 8.2 instructions from Citrix with 4096 bit key, sha256 signature algorithm, 398 day lifetime, and I was sure to add both fqdns and both IP addresses as Subject Alternate Names to be sure I covered all bases.

In retrospect, the first instance of that Hostname/IP... error was over an hour before I performed the update today, so it's not new after the update. I have the original certificate which I can restore if requested.

Did I miss a step when configuring xcp-ng hosts and my xo to use the certs from my internal, self-signed CA? I have used update-ca-certificates on my xo vm to add my root ca certificate (actually, this was done and tested on the vm from which my xo vm was fast-cloned), but do I also need add my root ca certificate to each xcp-ng host as well or is that not required since the full cert chain is included in /etc/xensource/xapi-ssl.pem?

Thank you in advance for the assistance for any guidance/assistance.

stormi

Hi.

Quoting @julien-f

XO uses IP addresses to fetch stats on non-master hosts, which obviously break the name check with the certificate

The setting xapiOptions.reverseHostIpAddresses can help with that

Hope that helps.

HeMaN

@techjeff ~~also add ip 10.0.60.12 to the alt name of the cert?~~

never mind, my mistake,I did not see the part where you said you added all to a new cert.

olivierlambert

Stats, consoles and other parts of a pool are served locally on a node within a pool. Eg when we ask to display the stats of a host, XAPI will tell us to ask the slave (with an HTTP redirect code).

So it means you need to get all the IPs or names in the pools to be in the cert, not just the master.

techjeff

@olivierlambert thank you for that information!

I have been creating a unique certificate for each host, and that explains why the requests have been failing / hanging. Based on what you're saying, I ought be making one certificate that covers all hosts in a pool and applying that same certificate and key to each host, yes?

For example, a certificate with CN="Example Domain XCP-NG Hosts" and SANs xcp-ng-1.example.com, 10.0.10.11, xcp-ng-2.example.com, 10.0.10.12, etc.

I'll give that a try and follow up.

olivierlambert

I think that would be needed to be able to connect to each host of the pool, yes Please keep us posted!

techjeff

@olivierlambert Today I created a new key and certificate as per the official documentation and I added SANs for the hostname and IP address of each xcp-ng hosts in my pool. I used XO's Replace existing certificate button, I saw the correct fingerprint apear on both hosts, I restarted the toolstack on both hosts, and I verified via cli on one of the hosts that the fingerprint of /ext/xensource/xap-ssl.pem matches that which is displayed in xo.

Nonetheless, I get the same results in the two instances of xo: the clone of the snapshot I took before updating (on commit feat: release 5.63.0 (#5925)) connects just fine with Unauthorized Cerificates disabled, and the original instance that was updated will only connect when Unauthorized Certificates is enabled, otherwise it gives the error self signed certificate in certificate chain.

I then ran the update tool again and it did indeed move to what is as of this writing the latest commit at https://github.com/vatesfr/xen-orchestra: [info] Updating xen-orchestra from 'abcabb736' to '9ceba1d6e' however, I still get the same behavior.

I'm not sure what, but something happened after 5.63.0 that is causing xo to reject the certificate even though I'm using the same certificate chain as before.

olivierlambert

Pinging @julien-f

techjeff

@olivierlambert @julien-f Today I was reviewing Web UI Certificates for XCP-NG Host I found that when I was configuring XO to use my custom CA, I did a copy/pasta job into /etc/systemd/system/xo-server.service.d/ca.conf and didn't actually update it to point to my actual CA certificate.. so that file was actually configured with the default: Environment=NODE_EXTRA_CA_CERTS=/usr/local/share/ca-certificates/my-cert.crt which obviously will not work. It must have been late into the evening when I added this file.

In any case, once I fixed this to correctly point to my CA certificate, ran systemctl daemon-reload, and systemctl restart xo-server.service, I am now properly able to connect with "Unauthorized Certificates" disabled.

For the record, I did ultimately create one certificate with a general Common Name and then added a Subject Alt Name for every IP Address and Hostname used by all 3 of my xcp-ng hosts, e.g.

DNS=xcp-ng-1.mgmt.intangible.home.arpa
IP=10.0.10.11
DNS=xcp-ng-1.san.intangible.home.arpa
IP=10.0.60.11
DNS=xcp-ng-2.mgmt.intangible.home.arpa
IP=10.0.10.12
DNS=xcp-ng-2.san.intangible.home.arpa
IP=10.0.60.12

etc..

Since we were not certain whether each host needed the same certificate containing all of the SANs for all hosts or if creating individual certificates for each host would be sufficient, I plan on creating test certificates that are unique to each host, deploying these certificates, and then reporting back whether installing unique certificates per host works as I originally suspected it would.

Also for the record, I'm currently working on Xen Orchestra, commit afeb2, xo-server 5.107.5, and xo-web 5.109.0.

techjeff

@olivierlambert @julien-f

As a final confirmation, since my last message, I generated one certificate for each of my 3 hosts. Each certificate only contained the DNS and IP SANs for that specific host. I then deployed each of the 3 certificates to their respective hosts using xe host-server-certificate-install without issue.

Like I mentioned in my previous post, I am not not getting self signed certificate in certificate chain because I have properly configured xo-server to trust my CA cert.

However, I am now back to getting the endless Xapi#getResource /rrd_updates (on xcp-ng-1) 0% tasks every minute that last for ~24 hours (unless I run xe-toolstack-restart to clear the list.)

I then redeployed my "Pool Certificate" (contains all DNS/IP SANs for all hosts) to each host from the master, executed xe-toolstack-restart and now all is working without issue. I must admit, this is much easier to maintain than trying to maintain 1:1 cert:host.

In conclusion, as you mentioned in your previous reply @olivierlambert, it does appear that a person must sign 1 certificate per pool and that certificate must be configured with Subject Alternate Names for each DNS name and IP used by all hosts in the pool.

Thanks again for working with me on this!

olivierlambert

Thanks a lot for your detailed and valued feedback!

If our doc lack details about this input, we should probably improve it. What do you think @stormi ?

stormi

Yes, this should definitely be added. @techjeff, since the subject is fresh in your mind, would you mind contributing an improvement to the guide?

techjeff

@stormi Sure, I would be happy to do that. In which repo are these guides stored?

stormi

@techjeff There's a link at the bottom of each page which will open a text editor directly on github. You can also manually create a PR on the repository the link leads to if you prefer.

techjeff

@stormi @olivierlambert Please see my submitted PR and please provide feedback.

techjeff

@olivierlambert as pointed out by @psafont on my PR #216,

I believe there is no such technical requirement, when following a redirect the
new request should be done against a different IP/host and the TLS connection renegotiated with that host, meaning none of the hosts' certs should have identifying information from the other one.

I think I need to dive deeper and hope to find an a log message related to the lingering Xapi#getResource /rrd_updates tasks.

HeMaN

@techjeff said in self signed certificate in certificate chain:

@stormi @olivierlambert Please see my submitted PR and please provide feedback.

I followed the guides on the site to install my custom certificates (self signed) to my new xcp-ng servers (homelab) and installed the CA cert in XO (from sources).

Came across this post when having issues with deploying self signed certificates and subsequent rrd update errors, and saw there is additional info mentioned in the PR for the Guide section on the website.

Is it correct the PR is not published yet to the website, or am I looking into the wrong places?

techjeff

@HeMaN You're correct that no changes have been published yet.

We were under the impression that we had found an undocumented requirement, but I was reminded that giving each host the same certificate is not the best practice. xcp-ng should be able to handle each host having its own certificate as long as their respective certificate authorities are trusted.

In any case, I need to do some more testing to narrow down the exact cause of the issues that I was seeing. I have been working a systemd service and timer with a few supporting scripts that automatically renew certificates by making ACME requests to my local, private CA, specifically adding support for additional SANs (previous iterations just used the system's FQDN).

Specifically, I want to test each server with SANs that correspond to each of it's IP addresses and FQDNs, deploy them using xe host-server-certificate-install, then perform packet captures as needed to determine why the Xapi#getResource /rrd_updates (on xcp-ng-1) 0% task is getting stuck.

So far, life has gotten a bit in the way, so I haven't dedicated the time to testing this, but I hope to get back to this soon.

exr90

FWIW I just got bit by this same thing, spending half a day yesterday kicking the wall. I think I got about 2/3 of the way just by trial and error on the cert's.

Would love to see this doc'd with the proper method, using an in-house local CA (not a public CA).