@HeMaN You're correct that no changes have been published yet.

We were under the impression that we had found an undocumented requirement, but I was reminded that giving each host the same certificate is not the best practice. xcp-ng should be able to handle each host having its own certificate as long as their respective certificate authorities are trusted.

In any case, I need to do some more testing to narrow down the exact cause of the issues that I was seeing. I have been working a systemd service and timer with a few supporting scripts that automatically renew certificates by making ACME requests to my local, private CA, specifically adding support for additional SANs (previous iterations just used the system's FQDN).

Specifically, I want to test each server with SANs that correspond to each of it's IP addresses and FQDNs, deploy them using xe host-server-certificate-install, then perform packet captures as needed to determine why the Xapi#getResource /rrd_updates (on xcp-ng-1) 0% task is getting stuck.

So far, life has gotten a bit in the way, so I haven't dedicated the time to testing this, but I hope to get back to this soon.

@florent the CR job was completed with health checks. The issue appears to be fixed in the fix_cr_healthcheck branch.

@Andrew @Danp - I discovered my boneheaded oversight

I have been developing on a one-shot systemd service with a timer for automatically renewing my short-lived XO certificate daily between a prescribed window of time using my local Step CA instance's ACME provisioner. It has been a fun project and I now have mostly automatic renewal of short-live certificates using what is essentially a private letsencrypt server that uses the CA that I generated.

I just looked at the logs and realized that I completely overlooked the fact that the the step-renewer.service is being run starting at midnight + random number of seconds between 0 and 5 minutes. This service will get a new certificate and then it restarts xo-server.service and that is what has been interrupting my backups.

Facepalm

With mixed feelings, I can confidently report that this issue is entirely self-induced. Thanks for the assistance narrowing down the cause!

@stormi @olivierlambert Please see my submitted PR and please provide feedback.

@irtaza9 happy to help!

@olivierlambert Thank you and your team again for your commitment to this fantastic FOSS tool and for allowing me to build it myself!

I very much appreciate the personal touch of my issues being triaged by the CEO and Co-Founder. It's refreshing to see an executive officer stay in touch with their customer base.

@olivierlambert as pointed out by @psafont on my PR #216,

I believe there is no such technical requirement, when following a redirect the
new request should be done against a different IP/host and the TLS connection renegotiated with that host, meaning none of the hosts' certs should have identifying information from the other one.

I think I need to dive deeper and hope to find an a log message related to the lingering Xapi#getResource /rrd_updates tasks.

@olivierlambert @julien-f

As a final confirmation, since my last message, I generated one certificate for each of my 3 hosts. Each certificate only contained the DNS and IP SANs for that specific host. I then deployed each of the 3 certificates to their respective hosts using xe host-server-certificate-install without issue.

Like I mentioned in my previous post, I am not not getting self signed certificate in certificate chain because I have properly configured xo-server to trust my CA cert.

However, I am now back to getting the endless Xapi#getResource /rrd_updates (on xcp-ng-1) 0% tasks every minute that last for ~24 hours (unless I run xe-toolstack-restart to clear the list.)

I then redeployed my "Pool Certificate" (contains all DNS/IP SANs for all hosts) to each host from the master, executed xe-toolstack-restart and now all is working without issue. I must admit, this is much easier to maintain than trying to maintain 1:1 cert:host.

In conclusion, as you mentioned in your previous reply @olivierlambert, it does appear that a person must sign 1 certificate per pool and that certificate must be configured with Subject Alternate Names for each DNS name and IP used by all hosts in the pool.

Thanks again for working with me on this!

@olivierlambert said in Commands in Xen Orchestra Jobs no longer working:

The best solution is to rely on XOA, you know

I agree with @olivierlambert! Personally, I'm using XO from sources in my home lab environment -- nothing is "production" and I'm mostly having fun trying to give back to the open source community.