@acebmxer thanks, the trace explains it. After the Finalize, the new RPU patched Host 1 and rebooted it at 11:42 your time, then waited for it to come back. When you disabled and re-enabled the pool in XO (I assume Settings > Servers?), XO dropped the connection the RPU was using and opened a new one. The running RPU kept watching the old connection, so it never saw Host 1 come back, and it failed on the 20 minute timeout at 12:02. That's also why it never started on Host 2. The stop you sent at 11:51 didn't cut that wait short either, which we'll look at on our side.

So next time, leave the pool connected during an RPU even if the master takes a few minutes to come back. XO reconnects on its own and gives each host 20 minutes.

The bugtool isn't needed for this run. Can you confirm two things: was "disable and re-enable" the toggle in Settings > Servers, and is Tilton Host 1 the pool master?

@acebmxer good to hear the pool is up to date. Your guess makes sense: the banner shows the last RPU XO recorded on that pool, so if you hadn't touched it since, the timeout is from that earlier run.

One thing I'd like to understand: Finalize only closes the record, it doesn't install anything. Did you start a new RPU after that, and did it stop after Host 1? If so, could you post the error from that
"Rolling pool update" task in XO? Its trace files would help too, they're in /var/lib/xo-server/data/rpu-traces on XOA (the exact path is in the task's traceFile property). The rpu-<poolId>-<timestamp>.ndjson file for that run is enough.

@acebmxer that banner comes from XO, not from the hosts. The previous RPU on Tilton Pool 1 stopped because Host 1 didn't come back within 20 minutes of its reboot (XO waits for the host to be enabled again with a restarted toolstack), and Host 2 is still pending. As long as an RPU is left incomplete, XO won't start a new one on that pool. That's why the button is greyed out.

What I'd do:

1. Check that Host 1 is up, enabled, and running the new packages.
2. Click Finalize. If the failed run left something it had changed (HA, WLB, load balancer, backup schedules, a disabled host, VMs moved or halted), XO refuses and lists it. Put those back, or force it if you're fine leaving them as they are. Forcing only lists them in the task, it doesn't change anything on the pool.
3. Run the RPU again to update Host 2. If the master is already up to date but the other host isn't, XO asks you to confirm first.

If that host is just slow to boot (long POST, RAID init, that kind of thing), you can raise restartHostTimeout in the [xapiOptions] section of the xo-server config. The default is 20 minutes.

If it never came back enabled at all, then the host logs lmolina asked for are still the way to go.

@jr-m4

Hi, your journalctl output has the answer: vectura: data plane: could not connect: timed out.

With a vCenter as source, the login and the CBT query go through the vCenter on port 443, but the disk itself is read directly from the ESXi host running the VM, on port 902. Your new XO reaches the vCenter fine, it just can't open a connection to that ESXi on 902. The stream has ended without data error is only the side effect of vectura exiting. CBT has nothing to do with it.

The old instance works because it can reach that host (VDDK needed the same port). I'd check firewall/VLAN rules or DNS on the new VM. From the new XO, using the ESXi name as vCenter shows it for that VM:

  getent hosts <esxi-host>
  nc -vz -w 5 <esxi-host> 902

and compare with the old instance.

~Mathieu

@tmk Confirmed bug on our side; thanks for the report.

The plugin declared its secret with a random default in the config schema. That expression runs when the module is imported, so every xo-server process computes a different one, and xo-server never writes back the values ajv fills in from schema defaults. So if you copied the secret out of the plugin form without saving the configuration, scraping worked until the next restart, and Prometheus started getting 401s.

Fix is here: https://github.com/vatesfr/xen-orchestra/pull/10290. The random default is gone. The secret is generated once on the first load and saved through the plugin configuration, so it survives a restart. If you already have one saved, it's left alone.

Once it's merged it'll be on master for people building from sources, and it'll reach XOA in the next release.

@Mang0Musztarda

To add to what @florent said, here's the complete picture:

Why your command failed:

1. Wrong URL: localhost:9004 is the internal port that only listens on
   127.0.0.1. It's proxied through xo-server at /openmetrics/metrics.

2. Wrong token: The error "Query authentication does not match server setting"
   means your Bearer token doesn't match the one configured in the plugin.

Correct approach:

1. Get your secret from: Settings → Plugins → OpenMetrics → "Prometheus secret"

2. Use the proxied endpoint:

   curl -H "Authorization: Bearer <your-actual-secret>" \                                                                                                                                                                                   
        http://<your-xo-server>/openmetrics/metrics                                                                                                                                                                                         
                                                                                                                                                                                                                                            
   

3. For health checks (no auth required):
   curl http://<your-xo-server>/openmetrics/health
   Returns: {"status":"ok"}

Full documentation is available at:
https://docs.xen-orchestra.com/advanced#openmetrics--prometheus-integration

It covers Prometheus configuration, available metrics, Grafana dashboards,
alerting rules, and troubleshooting.

@MajorP93 Thanks for sharing your Grafana dashboard! It could indeed be a nice
addition to link it from the official docs as a community example.