"No Stats" Issue

deadman2141

Good day everyone,
I'm racking my brain trying to find a solution for my stats not working on a member of my pool and the pool itself.
Currently using XO Community Edition, set on commit 749f0.

What I can see so far is that the call for the stats is using an IP rather than the FQDN.

{
  "id": "0m7j80y0c",
  "properties": {
    "method": "vm.stats",
    "params": {
      "id": "8cf0448f-8c00-fa2b-8649-da0edda14b39",
      "granularity": "seconds"
    },
    "name": "API call: vm.stats",
    "userId": "c373ef53-7433-450e-814e-74df95dfad68",
    "type": "api.call"
  },
  "start": 1740411424812,
  "status": "failure",
  "updatedAt": 1740411424843,
  "end": 1740411424843,
  "result": {
    "code": "ERR_TLS_CERT_ALTNAME_INVALID",
    "reason": "IP: 192.168.10.81 is not in the cert's list: ",
    "host": "192.168.10.81",
    "cert": {

I'm guessing I've made a mistake in my pool creation, so any insight would be greatly appreciated!

James

P.S. I'm also getting multiple hung Xapi#getResource /rrd_updates tasks, if that's relevant.

deadman2141

Good news everyone!
For anyone that comes across this after the fact. From what I can gather, the issue is in fact it is looking for a valid certificate when going to an IP to get the stats. So even a valid certificate will fail.
The solution that I have found, is the "Enable Unauthorized Certificates" option in Settings > Server.

And to be clear, it is obvious in hindsight...
What tripped me up is the fact that I'm now only connected to the pool master (After joining the second host), which has valid certificates.
So why would I allow unauthorized certificates?

Well this is why, so when the calls go out from the master with an IP address, it doesn't get tripped up.

Cheers!

ph7

@deadman2141
I also had some problem with stats running 749f0
I had no stats for ~ 15hours and then the host rebooted
All backups and Continuous replication ran fine during these ~ 15hours.

https://xcp-ng.org/forum/topic/10520/weird-performance-alert.-start-importing-vm-for-no-reason./2

TS79

@deadman2141 did you change your host's IP address after installation? That TLS certificate error and reason seem to indicate that the cert doesn't match the IP.

I found the link which might be worth looking at: https://docs.xcp-ng.org/guides/TLS-certificates-xcpng/

deadman2141

@ph7 I decided to do a reboot like you did, but it did not resolve the issue. During the time the second host was offline, the stats works as expected.

deadman2141

@TS79 I pawed though that a few times, because I have installed lets encrypt certificated. I should have included the rest of the error message:

  "result": {
    "code": "ERR_TLS_CERT_ALTNAME_INVALID",
    "reason": "IP: 192.168.10.81 is not in the cert's list: ",
    "host": "192.168.10.81",
    "cert": {
      "subject": {
        "CN": "xcp-ng2.disappointnetwork.com"
      },
      "issuer": {
        "C": "US",
        "O": "Let's Encrypt",
        "CN": "R11"
      },
      "subjectaltname": "DNS:xcp-ng2.disappointnetwork.com",
      "infoAccess": {
        "OCSP - URI": [
          "http://r11.o.lencr.org"
        ],
        "CA Issuers - URI": [
          "http://r11.i.lencr.org/"
        ]
      },

It looks like its giving back the valid certificate. But its making the call with an IP, not the host.domain. Unless I overlooked over something?

EDIT: I did not change the IP address after installation.

ph7

@deadman2141
Well, I didn't reboot it.
It did it to itself

deadman2141

@ph7 I suppose if I could read, I would have seen that
I'll check and see if over the next day it corrects itself.

deadman2141

Did some more digging, and found this from 2018.
https://github.com/vatesfr/xen-orchestra/issues/2723
Curious if that is still relevant almost 7 years later

If it is, then I wonder if there is another way to allow the connections other than xe host-emergency-disable-tls-verification
Not going to try that yet though.

jcharaoui created this issue in vatesfr/xen-orchestra

closed SSL certificate verification fails in stats query #2723

deadman2141

Good news everyone!
For anyone that comes across this after the fact. From what I can gather, the issue is in fact it is looking for a valid certificate when going to an IP to get the stats. So even a valid certificate will fail.
The solution that I have found, is the "Enable Unauthorized Certificates" option in Settings > Server.

And to be clear, it is obvious in hindsight...
What tripped me up is the fact that I'm now only connected to the pool master (After joining the second host), which has valid certificates.
So why would I allow unauthorized certificates?

Well this is why, so when the calls go out from the master with an IP address, it doesn't get tripped up.

Cheers!

TS79

@deadman2141 Thanks for sharing the above - I'd forgotten that aspect, as I run XCP-ng in a homelab so have always configured that setting.

BSmithITGuy

In case anyone comes across the "no stats" issue, I was able to solve my problem. I fully updated all my hosts in the pool and also XO. Still, I got "no stats" for the hosts and VMs.

The issue was I had recently set the "default migration network" and "backup network" for the pool. If you are having this issue, go to home>pools, select the pool having the issue, select "advanced", then scroll to the bottom under "Misc". I set them both back to "none" and it works now.

What clued me in was checking the logs (in XO - go to settings>logs), and I saw XO was trying to access my /31 subnet for backup/management traffic between my hosts (this is a P2P connection):

I am guessing that changing this also means XO will attempt to look at these interfaces for all management traffic and stats, and since there was no route to my /31 from my LAN to XO, it couldn't get the data.

I am sure I can get both working, but for now, I will just wait the extra 10 minutes or so for my backups and migrations to complete over gigabit (this is a home lab).

Coming from VMWare, maybe there should be a note about this on the GUI? Also coming from VMware - I want to thank the team (@olivierlambert) for doing a fantastic job; XCP-NG and XO has been rock solid in my homelab on random hardware that I have laying around. This issue was very minor, and maybe this was completely my mistake.

olivierlambert

Thanks for your report @BSmithITGuy

Let me ping @lsouai-vates about this so we can find a way to do better

truongtx8

@BSmithITGuy Thanks for your information. The temporary workaround now is put Xen Orchestra to the same XCP-ng management network.

Update 1: Diving deeper, I discovered that the API was changed the host management address (where the VM is located) to the backup address:
https://github.com/vatesfr/xen-orchestra/blob/92932dd54645838db58ecfdd872895c5ad461608/packages/xen-api/index.mjs#L1010

async _setHostAddressInUrl(url, host) {
...
url.hostname = await this._getHostBackupAddress(host)
}

BSmithITGuy

This post is deleted!