Adding new host to pool fails - Stunnel SSL certiticate verification failure

Bryanvh

Posting this here because in the hopes someone has an answer and that this helps anyone else encountering the issue.

I have a pool of a few hosts which I recently upgraded to XCP-NG 8.3 from 8.2. And, now, I am attempting to add a new host to this pool to increase my resource capacity. However, after adding the new server in Xen Orchestra, I go to my primary pool to begin the process of adding the new server but that fails with an error "Internal_Error(Stunnel.Stunnel [some text that runs off the screen] routines::certificate verify failed"))"

The full error is as follows:

"Stunnel.Stunnel_verify_error("0A000086:SSL routines::certificate verify failed")"

And the complete readout of the event is as follows:

{
  "id": "0mpn7bwnk",
  "properties": {
    "method": "pool.mergeInto",
    "params": {
      "sources": [
        "65c279b5-5a9d-db33-92f1-3f057fbafda6"
      ],
      "target": "f735841b-af37-0547-5d1e-8cb11bc51f0d",
      "force": true
    },
    "name": "API call: pool.mergeInto",
    "userId": "905ebdb9-6698-4902-8e60-9a028d1aa441",
    "type": "api.call"
  },
  "start": 1779834203408,
  "status": "failure",
  "updatedAt": 1779834206165,
  "end": 1779834206165,
  "result": {
    "code": "INTERNAL_ERROR",
    "params": [
      "Stunnel.Stunnel_verify_error(\"0A000086:SSL routines::certificate verify failed\")"
    ],
    "call": {
      "duration": 2713,
      "method": "pool.join_force",
      "params": [
        "* session id *",
        "192.168.1.11",
        "root",
        "* obfuscated *"
      ]
    },
    "message": "INTERNAL_ERROR(Stunnel.Stunnel_verify_error(\"0A000086:SSL routines::certificate verify failed\"))",
    "name": "XapiError",
    "stack": "XapiError: INTERNAL_ERROR(Stunnel.Stunnel_verify_error(\"0A000086:SSL routines::certificate verify failed\"))\n    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)\n    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at processImmediate (node:internal/timers:454:9)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

Obviously, it's unhappy about the certs. But I can't figure out why. For additional context, I have never messed with the certs on these servers previously. Based on some other forum posts, I went and checked the cert at /etc/stunnel/xapi-stunnel-ca-bundle.pem on the pool master as well as this new host. Seeing that it exists but unsure of whether it was still integral, I even ran xe host-refresh-server-certificate host=hostname on both just in case. Despite that, this error persists. Does anyone have any insight into the error or a possible fix from what they may have encountered themselves previously?

olivierlambert

Ping @Team-OS-Platform-Release

semarie

Just my 2 cents, but with SSL involved time is important: could you check the date is accurate on the two hosts ?

having the output of the following commands might help too:

stat /etc/stunnel/xapi-stunnel-ca-bundle.pem
openssl x509 -in /etc/stunnel/xapi-stunnel-ca-bundle.pem -noout -text

Bryanvh

@semarie

Maybe this points at an issue. It looks like the cert file is empty? And this is after I ran that command to refresh the cert. I get this same output for both the pool master and the host I am trying to add.

Then the openssl x509 command says it's unable to load the cert or read it. I assume that's because it's empty?

As for the time and date, yes the pool master and this server are in sync. At first, I had forgotten to set the new host to use the NTP pool during setup and Xen Orchestra helpfully yelled at me about that. Haha

semarie

Yes, if the file is empty, it is expected to the openssl x509 command to fail.
Does is it the same on the master ?

Bryanvh

@semarie
Yes. This screenshot is from the pool master. But, both it and the new host had the same output.

For clarity's sake, I have never applied an SSL cert to these hosts. This seems to be whatever built-in certs the system is using and signing.

Is there a way to fix these certs? Was the xe host-refresh-server-certificate host=hostname command not the correct command to fix this?

semarie

Sorry, but it is outside my competence zone. I prefer to not tell you to try something that I don't know the exact consequences of.

Does someone else could reply ?

LucienLassalle

@semarie I'll try to investigate to help you.

Is it possible to run:

stat /etc/xensource/xapi-pool-tls.pem
openssl x509 -in /etc/xensource/xapi-pool-tls.pem -noout -text
stat /etc/xensource/xapi-ssl.pem
openssl x509 -in /etc/xensource/xapi-ssl.pem -noout -text

(This file must exist; if not, I'd like the output of cat /etc/stunnel/xapi.conf.)
And I'd like the same output for /etc/xensource/xapi-ssl.pem.

If the certificate for /etc/xensource/xapi-pool.tls.pem has expired or it's empty, you can run:
xe host-refresh-server-certificate host=$(hostname)
If the certificate for /etc/xensource/xapi-ssl.pem has expired or it's empty, you can run:
xe host-emergency-reset-server-certificate

After running one of the two commands above, I recommend to do: xe-toolstack-restart
(This should indeed restart the stunnel@xapi.service)

I hope this helps.