Attempting to add new host fail on xoa and on server, worked on xcp-ng center

Jonathon

Just tried after doing a force clean install, still getting same error. Going to look into it more if there is not any

root@xoa:/home/fpcuser# sudo curl https://raw.githubusercontent.com/Jarli01/xenorchestra_updater/master/xo-update.sh | bash -s -- -f | tee xenrebuild.log
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6896  100  6896    0     0  39116      0 --:--:-- --:--:-- --:--:-- 39181
   installed : v24.11.1 (with npm 11.6.2)
Stopping xo-server...
Checking for Yarn package...
Checking for Yarn update...
E: Malformed entry 1 in list file /etc/apt/sources.list.d/yarn.list (URI parse)
E: The list of sources could not be read.
E: Malformed entry 1 in list file /etc/apt/sources.list.d/yarn.list (URI parse)
E: The list of sources could not be read.
Checking for missing dependencies...
Checking for Repo change...
Checking xen-orchestra...
Current branch master
Current version 5.192.1 / 5.189.0
Current commit 6cfefc91e47db7fb264705bc2def1f1b70bc537b 2025-11-12 18:01:41 +0100
0 updates available
Updating from source...
No local changes to save
No stash entries found.
Already up to date.
Clearing directories...
Installing...
yarn install v1.22.22
(node:1226553) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities.
(Use `node --trace-deprecation ...` to show where the warning was created)
[1/5] Validating package.json...
[2/5] Resolving packages...
success Already up-to-date.
$ husky install
husky - Git hooks installed
Done in 1.57s.
yarn run v1.22.22
$ TURBO_TELEMETRY_DISABLED=1 turbo run build --filter xo-server --filter xo-server-'*' --filter xo-web
turbo 2.5.8

• Packages in scope: xo-server, xo-server-audit, xo-server-auth-github, xo-server-auth-google, xo-server-auth-ldap, xo-server-auth-oidc, xo-server-auth-saml, xo-server-backup-reports, xo-server-load-balancer, xo-server-netbox, xo-server-perf-alert, xo-server-sdn-controller, xo-server-test-plugin, xo-server-transport-email, xo-server-transport-icinga2, xo-server-transport-nagios, xo-server-transport-slack, xo-server-transport-xmpp, xo-server-usage-report, xo-server-web-hooks, xo-web
• Running build in 21 packages
• Remote caching disabled

 Tasks:    30 successful, 30 total
Cached:    30 cached, 30 total
  Time:    1.347s >>> FULL TURBO

Done in 1.55s.
Updated version 5.192.1 / 5.189.0
Updated commit 6cfefc91e47db7fb264705bc2def1f1b70bc537b 2025-11-12 18:01:41 +0100
Checking plugins...
Ignoring xo-server-test plugin
Cleanup plugins...
Restarting xo-server...

So then I updated our seperate vm for xoa that we have used in the past for requests like this, and I am getting this behavior

pool.mergeInto
{
  "sources": [
    "e4cf2039-3547-6574-0e10-96f9d91316f0"
  ],
  "target": "38aea760-cf23-927c-ccf5-90969681e04b",
  "force": true
}
{
  "code": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE",
  "params": [
    "OpaqueRef:151858ec-cd9b-44f5-9cc5-f053685b1b8e",
    ""
  ],
  "call": {
    "duration": 2049,
    "method": "pool.join_force",
    "params": [
      "* session id *",
      "10.2.0.10",
      "root",
      "* obfuscated *"
    ]
  },
  "message": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:151858ec-cd9b-44f5-9cc5-f053685b1b8e, )",
  "name": "XapiError",
  "stack": "XapiError: POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:151858ec-cd9b-44f5-9cc5-f053685b1b8e, )
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Jonathon

After installing packages: https://docs.xcp-ng.org/xostor/#how-to-add-a-new-host-or-fix-a-badly-configured-host

Now I am getting the following on offical

pool.mergeInto
{
  "sources": [
    "e4cf2039-3547-6574-0e10-96f9d91316f0"
  ],
  "target": "38aea760-cf23-927c-ccf5-90969681e04b",
  "force": true
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "Stunnel.Stunnel_verify_error(\"1416F086:SSL routines:tls_process_server_certificate:certificate verify failed\")"
  ],
  "call": {
    "duration": 3104,
    "method": "pool.join_force",
    "params": [
      "* session id *",
      "10.2.0.10",
      "root",
      "* obfuscated *"
    ]
  },
  "message": "INTERNAL_ERROR(Stunnel.Stunnel_verify_error(\"1416F086:SSL routines:tls_process_server_certificate:certificate verify failed\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(Stunnel.Stunnel_verify_error(\"1416F086:SSL routines:tls_process_server_certificate:certificate verify failed\"))
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:60:5)
    at processImmediate (node:internal/timers:454:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

And still getting this on source install

pool.mergeInto
{
  "sources": [
    "e4cf2039-3547-6574-0e10-96f9d91316f0"
  ],
  "target": "38aea760-cf23-927c-ccf5-90969681e04b",
  "force": true
}
{
  "message": "app.getLicenses is not a function",
  "name": "TypeError",
  "stack": "TypeError: app.getLicenses is not a function
    at enforceHostsHaveLicense (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/pool.mjs:15:30)
    at Pools.apply (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/pool.mjs:80:13)
    at Pools.mergeInto (/opt/xen-orchestra/node_modules/golike-defer/src/index.js:85:19)
    at Xo.mergeInto (file:///opt/xen-orchestra/packages/xo-server/src/api/pool.mjs:314:15)
    at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)
    at Task.run (/opt/xen-orchestra/@vates/task/index.js:159:20)
    at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:469:18)"
}

Jonathon

Bummer

Jonathon

Boo

acebmxer

Check out this post for cert error not sure if its related- https://xcp-ng.org/forum/topic/9083/stunnel-sdn-cert-error-when-adding-host-to-pool/7?_=1762983691777

Jonathon

I see, it also says
name ( RO): sdn-controller-ca.pem
host ( RO): <not in database>
Like in the issue, but the file exists.

[11:28 ovbh-pprod-xen05 ~]# xe certificate-list
uuid ( RO)           : afdd9c8e-dcae-17c7-c35c-0fd7cebd387a
           type ( RO): host
           name ( RO): 
           host ( RO): f0cec10f-ad05-48e4-893c-414b3a3e15be
     not-before ( RO): 20251110T23:15:51Z
      not-after ( RO): 20351108T23:15:51Z
    fingerprint ( RO): BF:83:23:BB:7B:E9:30:DE:86:EA:9D:AF:DF:F8:BA:34:39:D0:81:AD:34:E5:C6:AB:0C:49:41:7B:4A:3C:8B:9E


uuid ( RO)           : b8dcd1f0-ef65-e762-f189-46bb78766c6b
           type ( RO): ca
           name ( RO): sdn-controller-ca.pem
           host ( RO): <not in database>
     not-before ( RO): 20200416T00:17:31Z
      not-after ( RO): 20470901T00:17:31Z
    fingerprint ( RO): 63:1F:89:3F:0E:1F:86:52:34:95:3C:6C:3F:9C:C8:B3:5A:61:6B:4D:EE:8F:A7:11:F0:BA:79:8B:C7:15:A0:E0


uuid ( RO)           : e7daedf2-7f35-ba40-093a-e0c011d91633
           type ( RO): host_internal
           name ( RO): 
           host ( RO): f0cec10f-ad05-48e4-893c-414b3a3e15be
     not-before ( RO): 20251110T23:15:46Z
      not-after ( RO): 20351108T23:15:46Z
    fingerprint ( RO): 71:41:B0:25:88:AA:E4:56:EE:F7:A9:8E:0A:A9:FE:C5:6A:0D:D5:37:30:BF:C8:81:C2:D7:B8:20:7A:6C:7F:B7


[13:50 ovbh-pprod-xen05 ~]# ll /etc/stunnel/certs/sdn-controller-ca.pem
-rw-r--r-- 1 root root 1907 Nov 12 09:45 /etc/stunnel/certs/sdn-controller-ca.pem

Removing it did not help, same error

[13:54 ovbh-pprod-xen05 ~]# xe certificate-list
uuid ( RO)           : afdd9c8e-dcae-17c7-c35c-0fd7cebd387a
           type ( RO): host
           name ( RO): 
           host ( RO): f0cec10f-ad05-48e4-893c-414b3a3e15be
     not-before ( RO): 20251110T23:15:51Z
      not-after ( RO): 20351108T23:15:51Z
    fingerprint ( RO): BF:83:23:BB:7B:E9:30:DE:86:EA:9D:AF:DF:F8:BA:34:39:D0:81:AD:34:E5:C6:AB:0C:49:41:7B:4A:3C:8B:9E


uuid ( RO)           : e7daedf2-7f35-ba40-093a-e0c011d91633
           type ( RO): host_internal
           name ( RO): 
           host ( RO): f0cec10f-ad05-48e4-893c-414b3a3e15be
     not-before ( RO): 20251110T23:15:46Z
      not-after ( RO): 20351108T23:15:46Z
    fingerprint ( RO): 71:41:B0:25:88:AA:E4:56:EE:F7:A9:8E:0A:A9:FE:C5:6A:0D:D5:37:30:BF:C8:81:C2:D7:B8:20:7A:6C:7F:B7

I also confirmed that all the certs for the hosts are current and not expired.

olivierlambert

I'm not sure who to ping exactly Maybe @Team-XAPI-Network with some experience on the cert issue?

psafont

I see at least on a couple of instances that the pool join is being forced. Please don't do this unless you understand the checks being ignored.

What's the result of a pool join without the force option?

Do both hosts have tls verification enabled?
If not, run xe pool-enable-tls-verification. This will set up the certificates for pool communication correctly for the hosts in that pool and turn on TLS verification for pool communication.
If yes, run xe host-list --minimal | xargs -I _ xe host-param-get uuid=_ param-name=name-label | xargs -I _ xe host-refresh-server-certificate host=_. This will reset the certificates for all the hosts in the pool.

Try running the normal join command after doing these steps, and report whether it went well, or report the error.

Jonathon

@psafont Sorry was swamped with other things. As listed above I get the same error, forced or not, from xcp-ng center, xcp-ng host, or xoa.

~~tls verification has always been of~~f, and in the past we have not had issue with adding new host to pool.

I have taken no other actions since my last posting.

Jonathon

xe pool-enable-tls-verification Was exactly what I needed, thanks! Worked after that.