Adding new host to pool fails - Stunnel SSL certiticate verification failure

Bryanvh

Posting this here because in the hopes someone has an answer and that this helps anyone else encountering the issue.

I have a pool of a few hosts which I recently upgraded to XCP-NG 8.3 from 8.2. And, now, I am attempting to add a new host to this pool to increase my resource capacity. However, after adding the new server in Xen Orchestra, I go to my primary pool to begin the process of adding the new server but that fails with an error "Internal_Error(Stunnel.Stunnel [some text that runs off the screen] routines::certificate verify failed"))"

The full error is as follows:

"Stunnel.Stunnel_verify_error("0A000086:SSL routines::certificate verify failed")"

And the complete readout of the event is as follows:

{
  "id": "0mpn7bwnk",
  "properties": {
    "method": "pool.mergeInto",
    "params": {
      "sources": [
        "65c279b5-5a9d-db33-92f1-3f057fbafda6"
      ],
      "target": "f735841b-af37-0547-5d1e-8cb11bc51f0d",
      "force": true
    },
    "name": "API call: pool.mergeInto",
    "userId": "905ebdb9-6698-4902-8e60-9a028d1aa441",
    "type": "api.call"
  },
  "start": 1779834203408,
  "status": "failure",
  "updatedAt": 1779834206165,
  "end": 1779834206165,
  "result": {
    "code": "INTERNAL_ERROR",
    "params": [
      "Stunnel.Stunnel_verify_error(\"0A000086:SSL routines::certificate verify failed\")"
    ],
    "call": {
      "duration": 2713,
      "method": "pool.join_force",
      "params": [
        "* session id *",
        "192.168.1.11",
        "root",
        "* obfuscated *"
      ]
    },
    "message": "INTERNAL_ERROR(Stunnel.Stunnel_verify_error(\"0A000086:SSL routines::certificate verify failed\"))",
    "name": "XapiError",
    "stack": "XapiError: INTERNAL_ERROR(Stunnel.Stunnel_verify_error(\"0A000086:SSL routines::certificate verify failed\"))\n    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)\n    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at processImmediate (node:internal/timers:454:9)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

Obviously, it's unhappy about the certs. But I can't figure out why. For additional context, I have never messed with the certs on these servers previously. Based on some other forum posts, I went and checked the cert at /etc/stunnel/xapi-stunnel-ca-bundle.pem on the pool master as well as this new host. Seeing that it exists but unsure of whether it was still integral, I even ran xe host-refresh-server-certificate host=hostname on both just in case. Despite that, this error persists. Does anyone have any insight into the error or a possible fix from what they may have encountered themselves previously?

olivierlambert

Ping @Team-OS-Platform-Release

semarie

Just my 2 cents, but with SSL involved time is important: could you check the date is accurate on the two hosts ?

having the output of the following commands might help too:

stat /etc/stunnel/xapi-stunnel-ca-bundle.pem
openssl x509 -in /etc/stunnel/xapi-stunnel-ca-bundle.pem -noout -text

Bryanvh

@semarie

Maybe this points at an issue. It looks like the cert file is empty? And this is after I ran that command to refresh the cert. I get this same output for both the pool master and the host I am trying to add.

Then the openssl x509 command says it's unable to load the cert or read it. I assume that's because it's empty?

As for the time and date, yes the pool master and this server are in sync. At first, I had forgotten to set the new host to use the NTP pool during setup and Xen Orchestra helpfully yelled at me about that. Haha

semarie

Yes, if the file is empty, it is expected to the openssl x509 command to fail.
Does is it the same on the master ?

Bryanvh

@semarie
Yes. This screenshot is from the pool master. But, both it and the new host had the same output.

For clarity's sake, I have never applied an SSL cert to these hosts. This seems to be whatever built-in certs the system is using and signing.

Is there a way to fix these certs? Was the xe host-refresh-server-certificate host=hostname command not the correct command to fix this?

semarie

Sorry, but it is outside my competence zone. I prefer to not tell you to try something that I don't know the exact consequences of.

Does someone else could reply ?

LucienLassalle

@semarie I'll try to investigate to help you.

Is it possible to run:

stat /etc/xensource/xapi-pool-tls.pem
openssl x509 -in /etc/xensource/xapi-pool-tls.pem -noout -text
stat /etc/xensource/xapi-ssl.pem
openssl x509 -in /etc/xensource/xapi-ssl.pem -noout -text

(This file must exist; if not, I'd like the output of cat /etc/stunnel/xapi.conf.)
And I'd like the same output for /etc/xensource/xapi-ssl.pem.

If the certificate for /etc/xensource/xapi-pool.tls.pem has expired or it's empty, you can run:
xe host-refresh-server-certificate host=$(hostname)
If the certificate for /etc/xensource/xapi-ssl.pem has expired or it's empty, you can run:
xe host-emergency-reset-server-certificate

After running one of the two commands above, I recommend to do: xe-toolstack-restart
(This should indeed restart the stunnel@xapi.service)

I hope this helps.

Bryanvh

@LucienLassalle

Here's the output from the pool master.
The xapi-pool-tls cert at least isn't empty.

And it still appears to be valid

The xapi-ssl cert also looks correct and un-expired

LucienLassalle

@Bryanvh Thank you for your feedback,
Your previous certificates look correct. I have not been able to reproduce the issue on my side, but I will try to diagnose it based on the code.

[MASTER]
I have a few preliminary commands. The first one is to retrieve the MASTER_UUID:
cat /etc/xensource-inventory | grep INSTALLATION_UUID | cut -d'=' -f2 | tr -d "'"

Then we can compare fingerprints between the master certificate and the one stored for the pool:
openssl x509 -in /etc/xensource/xapi-pool-tls.pem -noout -fingerprint -sha256
openssl x509 -in /etc/stunnel/certs-pool/{MASTER_UUID}.pem -noout -fingerprint -sha256
(please replace {MASTER_UUID} with the value retrieved above)

Normally, both fingerprints should match.
Also check that the CA bundle exists and is not empty:
ls -l /etc/stunnel/xapi-pool-ca-bundle.pem

If you previously ran:
xe host-refresh-server-certificate
you should probably run:
xe pool-certificate-sync

[JOINER]
Based on the code, the first phase has already been completed. You should therefore have files under /etc/stunnel/certs-pool/, including the master certificate:
openssl x509 -in /etc/stunnel/certs-pool/{MASTER_UUID}.pem -noout -fingerprint -sha256

[Additional checks]
Are all hosts synchronized to the same NTP server? date & timedatectl
Are all hosts fully updated to XCP-ng 8.3 and rebooted after updates?
Do you see the same error when joining the pool using XCP-ng (via Console or CLI) instead of Xen Orchestra?
Is there any more detailed error in /var/log/xensource.log ?
How many hosts are in your pool?
Is stunnel running correctly on all hosts? systemctl status stunnel@xapi

Do certificate chains validate correctly?
openssl verify -CAfile /etc/stunnel/xapi-pool-ca-bundle.pem /etc/stunnel/certs-pool/{MASTER_UUID}.pem

Respectfully,