mpiton

@jr-m4

Hi, your journalctl output has the answer: vectura: data plane: could not connect: timed out.

With a vCenter as source, the login and the CBT query go through the vCenter on port 443, but the disk itself is read directly from the ESXi host running the VM, on port 902. Your new XO reaches the vCenter fine, it just can't open a connection to that ESXi on 902. The stream has ended without data error is only the side effect of vectura exiting. CBT has nothing to do with it.

The old instance works because it can reach that host (VDDK needed the same port). I'd check firewall/VLAN rules or DNS on the new VM. From the new XO, using the ESXi name as vCenter shows it for that VM:

  getent hosts <esxi-host>
  nc -vz -w 5 <esxi-host> 902

and compare with the old instance.

~Mathieu

@tmk Confirmed bug on our side; thanks for the report.

The plugin declared its secret with a random default in the config schema. That expression runs when the module is imported, so every xo-server process computes a different one, and xo-server never writes back the values ajv fills in from schema defaults. So if you copied the secret out of the plugin form without saving the configuration, scraping worked until the next restart, and Prometheus started getting 401s.

Fix is here: https://github.com/vatesfr/xen-orchestra/pull/10290. The random default is gone. The secret is generated once on the first load and saved through the plugin configuration, so it survives a restart. If you already have one saved, it's left alone.

Once it's merged it'll be on master for people building from sources, and it'll reach XOA in the next release.

@Mang0Musztarda

To add to what @florent said, here's the complete picture:

Why your command failed:

1. Wrong URL: localhost:9004 is the internal port that only listens on
   127.0.0.1. It's proxied through xo-server at /openmetrics/metrics.

2. Wrong token: The error "Query authentication does not match server setting"
   means your Bearer token doesn't match the one configured in the plugin.

Correct approach:

1. Get your secret from: Settings → Plugins → OpenMetrics → "Prometheus secret"

2. Use the proxied endpoint:

   curl -H "Authorization: Bearer <your-actual-secret>" \                                                                                                                                                                                   
        http://<your-xo-server>/openmetrics/metrics                                                                                                                                                                                         
                                                                                                                                                                                                                                            
   

3. For health checks (no auth required):
   curl http://<your-xo-server>/openmetrics/health
   Returns: {"status":"ok"}

Full documentation is available at:
https://docs.xen-orchestra.com/advanced#openmetrics--prometheus-integration

It covers Prometheus configuration, available metrics, Grafana dashboards,
alerting rules, and troubleshooting.

@MajorP93 Thanks for sharing your Grafana dashboard! It could indeed be a nice
addition to link it from the official docs as a community example.