@tmk Confirmed bug on our side; thanks for the report.

The plugin declared its secret with a random default in the config schema. That expression runs when the module is imported, so every xo-server process computes a different one, and xo-server never writes back the values ajv fills in from schema defaults. So if you copied the secret out of the plugin form without saving the configuration, scraping worked until the next restart, and Prometheus started getting 401s.

Fix is here: https://github.com/vatesfr/xen-orchestra/pull/10290. The random default is gone. The secret is generated once on the first load and saved through the plugin configuration, so it survives a restart. If you already have one saved, it's left alone.

Once it's merged it'll be on master for people building from sources, and it'll reach XOA in the next release.

@Mang0Musztarda

To add to what @florent said, here's the complete picture:

Why your command failed:

1. Wrong URL: localhost:9004 is the internal port that only listens on
   127.0.0.1. It's proxied through xo-server at /openmetrics/metrics.

2. Wrong token: The error "Query authentication does not match server setting"
   means your Bearer token doesn't match the one configured in the plugin.

Correct approach:

1. Get your secret from: Settings → Plugins → OpenMetrics → "Prometheus secret"

2. Use the proxied endpoint:

   curl -H "Authorization: Bearer <your-actual-secret>" \                                                                                                                                                                                   
        http://<your-xo-server>/openmetrics/metrics                                                                                                                                                                                         
                                                                                                                                                                                                                                            
   

3. For health checks (no auth required):
   curl http://<your-xo-server>/openmetrics/health
   Returns: {"status":"ok"}

Full documentation is available at:
https://docs.xen-orchestra.com/advanced#openmetrics--prometheus-integration

It covers Prometheus configuration, available metrics, Grafana dashboards,
alerting rules, and troubleshooting.

@MajorP93 Thanks for sharing your Grafana dashboard! It could indeed be a nice
addition to link it from the official docs as a community example.