install_xen_orchestra v0.9.0 — install optional server plugins with --custom-plugins

Since the v0.7.2 post there's been a safety-net release (v0.8.0) and a new feature (v0.9.0). Everything from v0.8.0 through v0.9.0, in one place.

v0.8.0: VM snapshots before update/rebuild, and every RHEL template is now buildable

--update and --rebuild now take a normal XO API snapshot of the XO VM itself before touching anything, on top of the existing file backup — it shows up in XO's own UI under the VM's Snapshots tab like any other. Old ones get pruned automatically (keeps 3, or 14 days, whichever you set) so they don't pile up and trip XO's own Health-view warnings. This only works when XO itself is a Xen guest; bare metal just keeps the file backup as before.

Also new: a --status command that gives a read-only summary — version, how far behind master, service state, TLS cert expiry, disk/swap, backup and snapshot counts — and a --list-backups to see what's there without going through a restore. TLS certs now get a warning once they're under 30 days from expiring, wherever that check would previously only surface as a failed connection.

Rocky Linux 8, 9 and 10 dropped their "Coming Soon..." tag and build now — same tpl_prep_rhel script the AlmaLinux and CentOS Stream rows already use, since it's the same family. Every row in the template catalogue builds something now.

Rest of v0.8.0 was a security hardening pass — checksum-verifying the Node.js download, no more passwords on the command line for the XO Proxy helper, tightened file permissions on config.toml and the swap file — worth a look at the CHANGELOG if you're running this on anything you care about.

v0.9.0: --custom-plugins — install optional xo-server plugins with one command

XO has a plugin system (xo-server-auth-ldap, xo-server-load-balancer, that kind of thing) but nothing installs one for you — you're expected to drop it under node_modules by hand and restart the service. --custom-plugins (new menu entry too) does that part: pick a plugin, it gets copied to /usr/local/lib/node_modules/<name> — outside /opt/xen-orchestra, so --update's rebuild never touches it — and xo-server restarts to pick it up. Run it again and anything already installed shows pre-checked; uncheck to remove, check something new to install, leave one checked to refresh it if this repo's copy has changed since. Configuration itself still happens the normal way, in XO's own Settings > Plugins.

Two plugins ship to start:

• xo-server-nanokvm — power control for a host fitted with a Sipeed NanoKVM, over its REST API.
• xo-server-host-power-manager — powers an extra pool host on when CPU or memory gets tight, and powers it back off (evacuated first, same path as XO's own maintenance mode) once it isn't needed. Power-on can go through NanoKVM or XO's built-in iLO/DRAC/Wake-on-LAN.

Docs: docs/custom-plugins.md

The host-power-manager plugin got the most testing this round, and threw up a few real bugs against a live pool: saving its settings while it was already running silently killed every rule's timer until the next xo-server restart, its CPU/memory numbers disagreed with XO's own dashboard because it was excluding the managed host from the pool total, and there was no way to build a CPU-only or memory-only rule since both thresholds were required. All three are fixed, and power-off is now HA-aware — it defers to XAPI's own failover check and just leaves the host running and retries later rather than forcing an evacuation that would break your HA plan.

Repo: https://github.com/acebmxer/install_xen_orchestra

As always, let me know if you try either plugin out or hit anything.

@lem2405 said:

@acebmxer Excellent! Thanks a lot!

Welcome, enjoy.. Please let me know if you have any issues or suggestions to help improve.

@lem2405 said:

@acebmxer Is there a way to make the update feature completely unattended so it can be configured to run as a cron job?

Regards

Sure... Make sure you configure xo-config.cfg correctly. ./install-xen-orchestra.sh --update. It would only prompt for sudo passwor d.

Available Functions

Function	CLI Flag	Description
Deploy	--deploy	Create a Debian VM on a XenServer/XCP-ng pool and install XO into it
Build Templates	--build-templates	Build cloud-init VM templates on a XenServer/XCP-ng pool
Install	--install	Fresh install of Xen Orchestra
Update	--update	Update existing installation (with backup)
Restore	--restore	Restore from a previous backup (verified for completeness first; add --list-backups to just list them)
Rebuild	--rebuild	Fresh clone + clean build, preserves settings
Reconfigure	--reconfigure	Apply config changes without rebuilding
XO Proxy	--proxy	Deploy XO Proxy to a Xen pool master
Adjust Memory	--adjust-memory	Raise the heap memory allocated to the xo-server process
Status	--status	Read-only health report: version, service, TLS cert, disk/swap, backups/snapshots, git state
Edit Config	(menu only)	Open xo-config.cfg in your preferred editor
Rename Config	(menu only)	Rename sample-xo-config.cfg to xo-config.cfg

Running without flags launches an interactive menu. All flags also work directly:

./install-xen-orchestra.sh           # interactive menu
./install-xen-orchestra.sh --update  # run update directly
./install-xen-orchestra.sh --help    # show all options

@julienXOvates

Yes verified links working correctly now.

Vates tech responded to my ticket - Ticket#7762089 stated there was patch to apply to my XOA. After it was applied the rep stated to reboot XOA..

From then XOA would not connect to either remote proxy with error about unknown state (logs in ticket) XOA keep running into OOM issues after about 5 min idle trying to reconnect to remotes... Multiple reboots of xoa or proxy would not let them reconnect or memory issue to stop...

I had issues trying to revert back to previous snapshot taken during on of the upgrades... Eventualy got to to revert back to 6.8.1 proxies reconnected instantly...

v0.8.0–v0.9.1 — multi-user accounts, self-update, built-in HTTPS, and date ranges. Three releases since the last update, so bundling them here.

Date ranges. Collections, extractions, findings runs and the support package can now be scoped to a window instead of always covering the whole bundle — last 24h/7d/30d, since last reboot, or a custom range. XO's own routes still have no date filter, so the first download is unchanged size — the range only narrows what gets kept and reported afterwards.

Redact on demand. A collection no longer has to redact immediately with whatever rules happen to be on at that moment. There's now a checkbox to store the raw bundle only, and a "Redact now" button on the card afterwards using whichever rules are switched on when you press it.

Multiple user accounts, roles, and an activity log. This was on the "what is coming" list in the first post and it's in now. Any number of accounts, three roles — admin, operator, viewer (read-only, but can still download what's already stored) — and an Activity page logging logins, settings changes, and job runs.

Optional 2FA. Per-user TOTP, off by default, each account turns it on for itself. QR code setup, backup codes.

Self-update from the UI, opt-in and off by default. Checking needs nothing extra; applying an update needs the Docker socket mounted in explicitly, since that's effectively host root, so it's a separate deliberate step — uncomment the socket mount and group_add block in docker-compose.yml, and it needs its own .env file (just DOCKER_GID=..., next to docker-compose.yml, not the same file as xcp-pulse.env) or docker compose up -d fails with unable to find group. .env.example has the one-liner to generate it.

Built-in HTTPS, no reverse proxy required. Bundles nginx into the image to terminate TLS — self-signed cert on first start, or upload your own from Settings. A reverse proxy still works fine too.

A user manual in the app itself, under /help — no need to leave XCP Pulse or check out the repo to read how something works.

Fixed

• The account-requirements note from the first two posts was wrong in a way I only found by testing it properly: export:logs — the privilege log downloads actually need — isn't in any built-in XO role, but it can be granted through a custom role. "Test connection" was reading a catalogue that can't tell you that, and told every restricted account it needed full admin regardless. It now actually probes the download endpoint, and the docs walk through creating the custom role via three REST API calls (no UI for it yet on either XO version).
• A real concurrency bug: the shared SQLite connection wasn't locked across fetches, only execute, so two requests landing close together could crash a page reading jobs — reproduced it under the test suite's own concurrency test.
• Base-OS CVEs patched in the image build (perl-base, libc6, others); CI now runs a Trivy scan on every push.

Still not fixed: the truncated-download-behind-a-reverse-proxy issue from the last post. Haven't found the actual cause yet. Also looking for anyone who can test against a remote proxy in a lab setup. Also open to any other suggestions, features, improvements, UI changes, etc...

If any chance someone on Vates would test on their own time. Things like the support bundle and or the logs themselves are not being manipulated in any unwanted ways (for Vates or the project)

https://github.com/acebmxer/xcp_pulse

v0.7.0 — findings, category extraction, and a Vates support package.

Since the last update, log collection has been through a few fixes and gained three new pieces built on top of it.

Findings, from two sources. A new Findings page reads the XO API (messages, alarms, tasks, missing patches, backup/restore runs) and also scans a collected bundle for storage failures, multipath issues, XAPI exceptions, HA fencing, OOM kills and clock skew — six log-based rules in total now. When both reports describe the same incident (matched by condition and, if both are timestamped, within an hour of each other) each finding gets a small "confirmed by" badge instead of showing up as two unrelated entries.

Individual log categories, pulled out of a bundle already on disk instead of downloading again — XAPI, storage, audit, security, kernel, HA, xenstore, RRD, network, system. Tick categories on the Collect form before collecting, or against an already-stored bundle afterwards.

A Vates support package. One .tgz — redacted bundle, findings (MD + JSON), redaction report, inventory, and a manifest of what's inside and what was masked — instead of downloading each piece separately.

Dashboard status panels now show findings severity, how many redaction rules are off, storage used, and recent job activity at a glance.

Known issue — still unsolved

The truncated-download-behind-a-reverse-proxy bug from the last post is not fixed. logs.tgz has no Content-Length (XCP-ng builds it on the fly), and on my own Nginx Proxy Manager deployment the proxy's connection to XO intermittently closes before the last chunk arrives. Two things tried, neither fixed it:

• proxy_buffering off / proxy_set_header Connection "" — reduced how often it happened, didn't eliminate it. Identical config, run twice back to back, produced one good download and one truncated at the same offset as before.
• Suppressing httpx's Accept-Encoding: gzip, deflate header (it was asking the proxy to transport-encode an already-gzipped file) — same result, only reduced frequency, not shipped since it doesn't actually fix anything.

proxy_http_version 1.1; broke the proxy outright when I tried it, for a reason I don't understand — so that's not the next thing to try either.

What XCP Pulse now does about it: reads the archive in streaming order and keeps whatever it read before the stream broke, rather than losing the whole analysis to the last few missing bytes, and says on the report when it ended early. If a collection tells you the bundle ended early, the honest advice is to retry it — not that it's fixed. Documented in docs/installation.md in case someone with more insight into the Nginx/httpx side of this has a fix.

Also fixed since v0.6.0

• The restricted-account warning on the Collect page was showing on every connection, including admin ones — wrong template condition, not a real permission check. Fixed.
• The default collection no longer downloads the XAPI audit trail unless you ask for it — xen-bugtool already includes it in the bundle, so it was 770 MiB of duplicate data on every run. Default collection is now the bundle only, ~870 MB / ~3 minutes.
• XO's /messages and similar routes were being limited from the oldest end, not the newest — a pool with a lot of history could return findings from a month ago and miss everything recent. Now filtered by time window server-side instead.

https://github.com/acebmxer/xcp_pulse

I have received a very long update back from Veeam on my backup issues....

There appears to be another user with similar setup / issue not sure if that user is the OP this post specifically...

Hello,

Thank you for your patience. The QA team has finished the analysis, and I am going to outline the details as below:

1. CBT Inconsistency Issue

Whenever you see the Warning about CBT showing:

2026-08-16 17:37:27.459 00079 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 291af3d7-28c1-15a9-7f13-c6a1a12283e9 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated].

We can go to the previous run and see that we get reports that the CBT of the previous snapshot is inconsistent and thus removed by XCP:

2026-08-16 09:54:34.120 00004 ERROR | [XenBackupManager]: Failed to retain the data for the snapshot 5a17fb31-2616-4e72-a85c-e235883c8c91
Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 39c0f25f-e811-ee45-1b0b-e4a9ca9839ae (Async.VDI.data_destroy) failed: . VDI_NO_CBT_METADATA. OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed

The team has been reviewing and testing, and this is what they see. We send the request for the snapshot to be created, and it is sent to the coordinator: xcp-ng-vyadytkn

Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] vdi_clone: introduced VDI: OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed (5a17fb31-2616-4e72-a85c-e235883c8c91)

CBT shows as open and reading fine:

Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] ['/usr/sbin/cbt-util', 'set', '-n', '/var/run/sr-mount/7911c9c5-5f20-01e1-8b8d-39c6a98a2704/5a17fb31-2616-4e72-a85c-e235883c8c91.cbtlog', '-f', '1']
Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread]   pread SUCCESS

But look at how on xcp-ng-host2 the CBT gets marked as inconsistent by XCP, even though it's open and reading:

Aug 16 09:51:05 xcp-ng-host2 SM: [1683689][MainThread] Changed Block Tracking metadata is inconsistent for disk 5a17fb31-2616-4e72-a85c-e235883c8c91.

The Breakdown:

• xcp-ng-vyadytkn was the coordinator—the one we talk to, who then passes everything around to the hosts.
• xcp-ng-host2 was the host that the VM resided on at that time.
• xcp-ng-host2 is marking the CBT as inconsistent and deleting the snapshot CBT log, which means we cannot reference it on the next run. We do not see anything else interacting with the CBT besides that host.

This matches exactly what we see with another client running the same setup. The team successfully replicated the environment, which is configured as follows:

• VM storage is NFS.
• The VM is running on a host that is not the coordinator.

They have been able to reproduce this behavior occasionally, and the working theory is that the VM host keeps its own tracking separate from the coordinator. Part of the backup process requires the VM host to issue a pause/resume via a process called tapdisk. When it resumes, it pushes a data cache (likely inside the NFS cache), overwriting the CBT reference held by the coordinator server. It acts as a race condition—whoever pushes the CBT data last wins. If the VM host pushes last, it breaks what the coordinator is sending.

Next Steps for CBT:
The team is working to raise this issue directly with Vates so they can address the race condition. We ask that you also open a Vates ticket if possible to help draw more attention to the bug. We are trying to find ways to code around this race condition in the future, but there are currently no ETAs or guarantees.

---

2. Synthetic Full Failures (Delilah_ArcFS01)

In addition to the CBT bug, the team discovered a separate issue. Recently, the synthetic fulls for Delilah_ArcFS01 have been failing. This appears to be related to the NFS repository:

[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)    Failed to execute full transform task
[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)    Agent: Failed to process method {Transform.CompileFIB}: NfsFileEx was already stopped. File: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]] (Veeam.Backup.Common.CCppComponentException)
[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)       in c++: Failed to execute command Command: READ, Offset: 2523136, Data size: 659456, Chunk size: 131072
[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)       in c++: Failed to read file: Offset: 2523136, Block size: 659456, File: Path: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]], Handle: [01000702080061030000000007fffd5f65840fb20000000000000000150061038a2f7c0b0400610379207c0b], Read chunk size: 131072, Write chunk size: 131072, Read only: true

Because it continuously fails during the synthetic full, it eventually causes the snapshot to be lost when retries fail. The team will change this logic in a future update.

Action Items for the NFS Issue:
To help prevent these snapshot loss failures, could you please provide the logs from the repository NFS (192.168.20.91)?

1. Export the logs from the Veeam server and select the repository host.
2. Provide the results of running this command directly on the Repository host:

journalctl --since "30 days ago" > journal_repo.log

Temporary Workaround:
If you can, please temporarily switch to active fulls instead of synthetic fulls to help stabilize the job.

Please let me know if you have any questions, and if you are able to raise that ticket with Vates.

They also just responded back with this statment...

Regarding the second part of the last email with the noticed Synthetic full issue, I actually would like you to also make this registry entry on the Veeam server and keep synthetic fulls enabled to see if it helps with that issue:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: Nfs3CommandWaitTimeoutSec
Type: DWORD
Value (In Decimal): 86400

SDN controller documentation link - page not found...

XO from sources latest commit - 280c0

So i just tried to apply the xcp-ng host updates release today.. I had an issue unable to apply updates as a vm could not migrate...

The problem was i had Nested Virtualization enabled on the vm. This was enabled for testing and forgot to disabled. However the error was not clear.

I am noting this to come back to when my xcp-pulse is further along.

vm.migrate
{
  "vm": "fb72a8d7-a039-849f-b547-24fc56f056ba",
  "migrationNetwork": "172b1b75-18cf-eed1-0a8c-0aca5e16f4e9",
  "targetHost": "c1372ec6-3651-4481-808b-34293e53e144"
}
{
  "code": "VM_IS_IMMOBILE",
  "params": [
    "OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0"
  ],
  "task": {
    "uuid": "2b031b83-6b30-4a26-29c7-a19bf963ceb0",
    "name_label": "Async.VM.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260908T20:23:39Z",
    "finished": "20260908T20:23:39Z",
    "status": "failure",
    "resident_on": "OpaqueRef:119546e4-e0fd-adc7-d42a-b17c4efe789d",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VM_IS_IMMOBILE",
      "OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 758))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1657))((process xapi)(filename fun.ml)(line 33))((process xapi)(filename fun.ml)(line 38))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1825))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1656))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2584))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "VM_IS_IMMOBILE(OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0)",
  "name": "XapiError",
  "stack": "XapiError: VM_IS_IMMOBILE(OpaqueRef:1571e2f7-c235-dc10-d766-8883ddd99fd0)
    at XapiError.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1329:24)
    at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1363:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1353:12)
    at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1560:14)"
}

v0.6.0 — log collection is in. This was the piece the first post said was next, and it works end to end now.

A new Collect page runs a per-host collection as a background job: it pulls logs.tgz and the XAPI audit trail from Xen Orchestra, keeps the raw copies, and produces a redacted copy of each. On my own XCP-ng 8.3 pool that came to 434 MiB for the bundle and 769 MiB for the audit trail, 2.3 GiB stored across the five files, with 2,542,805 values masked — almost all of them session tokens.

The download is streamed a megabyte at a time and never held in memory, and every chunk is a cancellation point, so Cancel actually stops a transfer rather than waiting it out. A failed or cancelled download deletes its partial file instead of leaving a half-written bundle sitting there looking like a collection.

Both copies are kept and marked redacted — safe to send or raw — unmasked. Redaction is lossy, and the raw copy is the only thing that can answer "what did that used to say?" afterwards. Each collection writes the same redaction report the preview page produces, so you can see which rules fired — and which were switched off — before anything leaves the box. In the screenshot I had IPv4, MAC, UUIDs and hostnames turned off, and they show as off rather than as zero hits. That is deliberate: a partly-masked bundle should be obvious before you attach it to a ticket.

There is also a retention policy. The newest n collections are kept whatever their age, and only what is left is judged on age — a long gap in collecting can't empty the store. It shows you what a cleanup would delete and how much space that returns before you press the button.

Expect bugs. This is still very much a work in progress and it has only ever run against my own pool. If you try it in a different environment, assume things will break — please tell me what did. Known issues right now:

• The red "This connection uses a restricted account" warning on the Collect page shows for every connection, including an administrator one. It is a wrong condition in the template, not a real permission check — ignore it if your account is admin. The screenshot below has it, over a collection that succeeded.
• The Collect page says to expect "433 MB and 100 seconds per host". The 100 seconds is the download only; a full collection also fetches the audit trail and redacts both copies, which took 203 seconds on my pool. Budget three to four minutes.

Two more things worth flagging if you try this.

logs.tgz sends no Content-Length, so a reverse proxy that buffers will happily hand you a truncated archive with HTTP 200. Mine did. It now detects that and reports the bundle as cut short rather than as a protocol error, and repacks whatever did arrive instead of throwing the lot away. If you are behind Nginx Proxy Manager, proxy_buffering off; and proxy_max_temp_file_size 0; are what fixed the stall for me.

The account requirement from the first post has not changed — the log download needs export:logs on host, and on the instance I measured that is only reachable via Administrator. Inventory still works fine on Read only.

Next up: individual log categories pulled out of the cached bundle (~35 MB instead of 434 MiB), date ranges, and then findings.

https://github.com/acebmxer/xcp_pulse

While this project is more for myself it is open to others to use. Please use at your own risk. As always review the code before using in a production environment. Please leave any feedback or suggestions. https://github.com/acebmxer/xcp_pulse

XCP Pulse collects XCP-ng and Xen Orchestra logs, bundles them for download, analyses them, and reports findings — for your own troubleshooting or to attach to a Vates support ticket. Everything goes through the Xen Orchestra REST API — nothing is installed on your hosts, and it only ever reads.

It runs as a container with a web UI. Being built in stages, and v0.5.1 is where it is today — so this is an early look rather than a finished tool. Log collection itself is the next big piece.

What works today

Version	What it does
v0.1.0	Container, login, sessions, healthcheck
v0.2.0	Connect to Xen Orchestra — URL and API token, token encrypted at rest, "Test connection"
v0.3.0	Pools and hosts listed on the dashboard, grouped by pool
v0.4.0	Background jobs with live progress and history, and stored results
v0.5.0	Redaction — masks addresses, tokens and credentials out of log text, with a preview page
v0.5.1	Each redaction rule can be switched on or off

Redaction came before any download button on purpose. A real log bundle is full of internal addresses, usernames and session tokens, and the point of the download is sending that file to Vates — shipping collection first would have meant a headline feature that leaks credentials. One real xensource.log I tested against had 8,359 lines matching password, secret or session patterns.

What is coming

• Collect the full log bundle per host, as a background job, redacted, with a download. Measured on my own XCP-ng 8.3 pool: about 433 MB and 100 seconds per host, 603 files.
• Collect individual categories — XAPI, storage, audit, security, kernel and the rest — pulled out of the cached bundle instead of downloading again. Current logs only comes to roughly 35 MB instead of 433 MB.
• Date ranges — of that 433 MB, 418 MB is rotated history, so asking for the last three days is a large saving.
• Findings — failed tasks, alarms, missing patches, storage errors, HA fencing, clock skew, with the evidence behind each one.
• A Vates support package — one file with the redacted bundle, findings and a manifest of what was masked.
• Multiple users, docs in the web UI, and self-update.

Full list with status: https://github.com/acebmxer/xcp_pulse/blob/main/docs/roadmap.md

Quick start

No clone and no build — the image is published to GHCR, so the compose file and an env file are the whole deployment:

mkdir xcp-pulse && cd xcp-pulse
curl -o compose.yaml https://raw.githubusercontent.com/acebmxer/xcp_pulse/main/compose.yaml.example
curl -o xcp-pulse.env https://raw.githubusercontent.com/acebmxer/xcp_pulse/main/xcp-pulse.env.example

Generate the admin password hash — XCP Pulse stores a hash, never a password, and refuses to start without one:

docker compose run --rm xcp-pulse python -m app.hashpw

Paste the printed XCP_PULSE_ADMIN_PASSWORD_HASH=... line into xcp-pulse.env, then:

docker compose up -d

Open http://localhost:8080 and sign in with admin and the password you chose.

The env file has to be called xcp-pulse.env and not .env — compose treats a file of that name as its own variable source and mangles the Argon2 hash.

A note on the XO account

Inventory and API-based findings work fine with a Read only role. Downloading logs does not — /hosts/{id}/logs.tgz requires export:logs on host, and on the instance I measured (XO CE, @xen-orchestra/rest-api 0.39.0) that privilege was not in the grantable catalogue at all. The only role that could download logs was Administrator.

So for log collection you currently need an admin token. XCP Pulse reads the privilege catalogue from your instance and tells you what it can actually grant, rather than assuming — and asks which account type you gave it so it can name the missing privilege instead of showing a bare 403.

Security

XCP Pulse holds a token that can read every log on your pool, and stores files containing session tokens and internal network topology. Run it on a trusted management network behind a reverse proxy — the compose file binds to 127.0.0.1 by default for that reason. Not exposed to the internet.

MIT licensed. This is an independent tool and is not affiliated with or endorsed by Vates.

install_xen_orchestra v0.7.2 — twelve templates, and the builder can now use XO's own API

Since the v0.5.0 post the template library has gone from one distro to twelve, and it picked up a second way of talking to the pool. Everything from v0.5.0 through v0.7.2, in one place.

Templates: twelve buildable, three still to come

v0.5.0 shipped with Debian 13 and I asked here which distros to do next. That list is now:

• Debian 12, 13
• Ubuntu 22.04, 24.04, 26.04 LTS
• AlmaLinux 8, 9, 10
• CentOS Stream 9, 10
• Fedora 43, 44

Rocky Linux 8/9/10 are the only rows still marked Coming Soon... in the menu.

Every one of these is a row in the catalogue table rather than new build code, which is what I said in the v0.5.0 post I was aiming for. Each image did get read individually rather than assumed to match its neighbour — default user, disk size, kernel type, partition table — because those differ inside a family. Ubuntu 22.04 expands to 2.2 GiB and 24.04 to 3.5 GiB, so copying one figure across the family would have inflated every VM cloned from the smaller one.

Two things that had to be solved along the way and now apply to every image:

• qcow2 images import. Debian publishes raw; almost everyone else publishes qcow2, so the builder converts before importing.
• The shipped password actually works. The RHEL-family and Fedora images declare lock_passwd: True for their default user, and cloud-init re-locks that account on every boot — so the password the build set was already locked by the time a clone came up. The build now drops a file in /etc/cloud/cloud.cfg.d turning that off. Delete it once you've put your own SSH keys on. Key auth was never affected.

Arrow keys in the template menu now skip the Coming Soon rows instead of stepping through them.

The builder can now go through XO's API instead of SSH to dom0

Template building has always driven xe over SSH to the pool master, which means having the pool master's root password to hand. Everything it does that way, XO exposes over its own API — create the VM, import the disk, build the cloud-init drive — so v0.7.0 added that as a second path.

Which one makes sense depends on where you run it from. If you installed with --deploy, this repo is sitting inside the XO VM and that's where you run --update from, so XO is on localhost and you already have an API token in xo-config.cfg for the pre-update task check. Same token, no root password. Run it from your workstation against a pool with no XO on it yet and none of that is true, so SSH stays.

TEMPLATE_BUILD_METHOD picks:

auto   # default — API if it's reachable and authenticated, else SSH, and it says which and why
api    # API only, no fallback
ssh    # exactly what it did before

The choice is made by a preflight check before anything gets created on the pool, so a fallback costs you nothing but a different password prompt. On the API path the image streams from its mirror straight into XO without being staged to a file, and XO builds the cloud-init drive itself — so no ISO writer needed on the machine you're running from.

Two new config keys: XO_API_TOKEN and XO_URL. XO_API_TOKEN replaces XO_TASK_CHECK_TOKEN in name only — the old key is still read, so an existing config keeps working untouched. XO_URL can be left unset on the XO VM itself, and falls back to PUBLIC_URL if you've set that.

--deploy fixes

• The appliance now gets the XCP-ng guest tools. A deployed VM had no guest agent at all, so the XO it was itself hosting couldn't report its IP, memory or disk usage, and couldn't shut it down cleanly. It ran fine while looking half-blank in the very interface it served.
• It now boots UEFI, not BIOS. Nothing was ever setting the firmware parameter, so a VM from --deploy and a VM cloned from a --build-templates template came out of the same script with different firmware. It's probed from the disk now, same as the builder does.
• The temporary deployment SSH key is actually removed. It wasn't being deleted, leaving a working key on the VM.

Progress bars

curl --progress-bar redraws the whole line every update and leaves the cursor sitting in it, which over SSH is a cursor visibly scribbling back and forth instead of a bar filling up. All four transfers now draw a bar that doesn't repaint, including the one that stages the image on the pool master — which is the longest step of a deploy and was the one people would actually be watching.

Also

• Templates get sensible platform settings instead of inheriting them from the scaffolding: vga=std with 16 MiB video memory (the stock 4 MiB cirrus is what leaves the console at 640x480), viridian off (it's Hyper-V enlightenment, meant for Windows), and cores-per-socket matching the vCPU count.
• Checksum verification works for every origin now. AlmaLinux, CentOS Stream and Fedora would each have 404'd on the wrong filename and imported unverified with a warning.
• Debian 11 and Ubuntu 22.04 are deprecated with dates on them, and the menu says so rather than the row silently vanishing one day.
• The README's menu screenshots no longer need a horizontal scrollbar on GitHub.

353 unit tests, and CI smoke-tests the install on 10 distros.

Repo: https://github.com/acebmxer/install_xen_orchestra
Changelog: https://github.com/acebmxer/install_xen_orchestra/blob/main/CHANGELOG.md
Templates doc: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

install_xen_orchestra v0.5.0 — build your own cloud-init VM templates

As I am sure most of you know that XOA Hub is not accessible from Xen Orchestra from sources.

So v0.5.0 of my install script adds a VM Template Library — a new
--build-templates flag (and menu entry) that builds the equivalent on your own pool, from each distribution's own published cloud image.

./install-xen-orchestra.sh --build-templates

It runs from your workstation over SSH against the pool master, like --deploy does. Nothing is installed locally and your XO install isn't touched.

What you get

A sealed template that shows up next other vm templates you have, with:

• cloud-init — supply an SSH key or a full cloud-config at VM creation.
• XCP-ng guest tools, installed from guest-tools.iso on the pool. Without these XO never reports the VM's IP. On Debian 13 there's no
  xe-guest-utilities package, and a failed package install doesn't stop cloud-init, so an apt-first approach silently gives you a template that looks fine and is useless. The ISO is what the XCP-ng docs prescribe anyway.
• A scrubbed identity — machine-id, SSH host keys, cloud-init state and logs are all wiped before sealing, so clones don't collide on DHCP leases or share an SSH fingerprint.
• growroot, so the disk size you ask for at creation is actually filled.
• One template for both BIOS and UEFI — the Debian generic images carry an ESP and a BIOS boot partition, so you just flip "Boot firmware" in XO's advanced settings. The build itself isn't prompted for storage or network —
  it uses your pool's default SR (or the emptiest one, if you haven't set a default) and the management network. Both are just build-time choices; you pick whatever you want when you actually create a VM from the template.

Images are pulled from the distribution's own mirror (nothing redistributed), downloaded onto the pool master so the ~3 GB never crosses your workstation's link, and verified against the origin's published SHA512SUMS at build time
—so new upstream releases get picked up without me editing anything.

Allow roughly five to ten minutes per template; it has to boot the VM once to
install the guest agent and run the scrub.

Distros

Debian 13 (Trixie) is the only one for now. That's a starting point, not the limit — the catalogue is a table with one row per distro, and adding another is a row in that table, not a change to the build.

So: comment here if there's a specific distro you'd want to see sooner rather than later and I'll prioritise accordingly.

Repo: https://github.com/acebmxer/install_xen_orchestra
Docs for this feature: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

@flakpyro The patch was pushed though to XO from Sources. Hasnt hit XOA yet i dont beleive.

https://github.com/vatesfr/xen-orchestra/commit/003cc6f9cc7de6bff9eaee9fa0bc14a1f7b64cc0

I am not seeing this error after patch applied to XOA...

fix(proxy): fetch failed  (#10346)
* fix(backups): fetch failed

disable compresion between xoa on proxy on xoa side

* fix(xo-proxy): disable compression on the proxy

it was disable with hrp, replaced by #10038 since it was never
sending accept-encoding header

compression can be put back later on a route by route basis after
careful checks ( for example : file restore, backup listing )

v0.4.1 and v0.4.2 are out.

v0.4.1 — mostly credential encryption (ENCRYPT_REDIS_CREDENTIALS)

• The preflight check now catches a Xen guest that's missing xenstore-read / xenstore-write. Before, that guest passed the systemd-detect-virt check and then failed later at xo-server startup, because there was nowhere to store the XenStore half of the key. It now names the missing tools and the package that provides them.
• --backup now warns that it contains neither half of the encryption key, and points at the passphrase-protected XO config export as the actual recovery artifact.
• --uninstall warns before deleting this host's on-disk key half — doing so while leaving Redis in place turns the stored records into unreadable ciphertext.
• The docs (README, sample config, generated config.toml comments) now explain why the XAPI credential in Redis is stored reversibly instead of hashed: it's replayed on every connect and auto-reconnect, so it can't be a hash. They also spell out the recovery caveats — the key is split between XenStore and /var/lib/xo-server/data, and losing either half while encryption is on makes the records permanently undecryptable.
• Also flagged in the sample config: pointing REDIS_URI at an off-host Redis sends the pool credentials over the network in cleartext.

v0.4.2 — docs and packaging only, no behaviour change:

• The README had grown past 700 lines, with the --deploy walkthrough alone taking up a third of it. The large reference sections moved out to docs/deployment.md, docs/configuration.md, docs/authentication.md and docs/troubleshooting.md; the README is back to ~330 lines with a table pointing at them. Nothing was removed.
• Expanded the badge row — release tag, last commit, open issues, unit-test count, number of distros the CI container matrix covers, and ShellCheck status.

---

Where the project's got to

I started this in February as a script to save myself doing the from-source install by hand every time I rebuilt my homelab. Seven months and twelve tagged releases later it's grown into something a fair bit bigger:

• --install does the from-source build on whatever distro you run it on — the Debian/Ubuntu, RHEL/Alma/Rocky/CentOS and Fedora families, eight distributions in all, each one smoke-tested in CI alongside 118 unit tests.
• --deploy (added in v0.3.0) starts a step earlier: point it at a XCP-ng / XenServer pool and it builds the VM, pulls a Debian cloud image straight onto the pool, seeds it with cloud-init and runs the install inside it. It's there for people who have a pool but no Linux VM to put XO on.
• --proxy deploys an XO Proxy VM the same way — that one came straight out of requests in this thread.
• Around that: a config file so nothing needs editing in the script itself, --update / --rebuild with a local build cache, --backup / --restore / --uninstall, non-root operation with encrypted Redis credentials, firewall handling, and a lot of --deploy security work in v0.4.0 — image checksums, pool host-key pinning, the deploy SSH key destroyed at the end of the run, a required admin password, passwordless sudo revoked once the install finishes.

Since the v0.3.0 / v0.4.0 post a couple of weeks ago the repo has had 263 clones from 49 unique cloners.

Thanks

To everyone in this thread who's installed it, tested it and come back with feedback — genuinely, thank you. The proxy support, the credential-encryption questions that turned into proper docs, the config-file suggestions, the distro edge cases on systems I don't run myself: several of the releases above have something in them that started as a post here. Keep it coming.

Repo: https://github.com/acebmxer/install_xen_orchestra
Changelog: https://github.com/acebmxer/install_xen_orchestra/blob/main/CHANGELOG.md

@pdonias said:

Hello everyone! We need you!

We're currently designing the XO 6 UI for non-admin users, and some choices are genuinely hard to make. If you'd like to give us your opinion, here's a 30-second survey with 2 questions we couldn't settle ourselves: https://survey.vates.tech/s/cms4nrqb4005wrw01021ru6dy

Thanks!

Thank you with presenting us with a choice. I have made my comments and offered a 3rd option.

To sum up... It appears the patch has worked for me but has shed some light on miss configuration with the network design at this remote location, that I will need to take a real closer look at.

@florent thank you very much for your continue assistance with my issues.

@florent

Backups ran after patch applied...

1 pool had zero issues on backup 100% complete...
Other pool while it complete did have some errors...

Started a second backup on same pool and giving same results.

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "compactMjml",
    "hideSuccessfulItems": true
  },
  "id": "1788428437610",
  "jobId": "c0e31730-7d45-4f3a-8733-8cea444b127d",
  "jobName": "Salem - Delta Backups",
  "message": "backup",
  "proxyId": "7840039e-a3cb-41ed-acee-3690f89a1426",
  "scheduleId": "ce3c801c-4384-4726-bda1-f6bae788e2f4",
  "start": 1788428437610,
  "status": "success",
  "tasks": [
    {
      "id": "0mtlc477z-gdvpj68kva",
      "start": 1788428438927,
      "status": "success",
      "tasks": [
        {
          "id": "0mtlc478f-guqsurb38qa",
          "start": 1788428438943,
          "status": "success",
          "end": 1788428440120,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        },
        {
          "id": "0mtlc48cx-c1y79k93vl",
          "start": 1788428440401,
          "status": "success",
          "end": 1788428442225,
          "result": "67677eec-494b-6ab6-0447-ac4dba8932a1",
          "message": "snapshot"
        },
        {
          "id": "0mtlc49rm-ou0nslm2c7l",
          "start": 1788428442227,
          "status": "success",
          "tasks": [
            {
              "id": "0mtlc724s-62ae1bzdupa",
              "start": 1788428572300,
              "status": "success",
              "end": 1788428932605,
              "result": {
                "size": 20385362944
              },
              "message": "transfer",
              "data": {
                "progress": 99
              }
            },
            {
              "id": "0mtlceupa-f476k7iq2u4",
              "start": 1788428935918,
              "status": "success",
              "tasks": [
                {
                  "id": "0mtlceutz-yztgqk0sv0c",
                  "start": 1788428936087,
                  "status": "success",
                  "end": 1788429285986,
                  "result": {
                    "size": 20380123136,
                    "id": "5fb5f6f1-709b-b149-4aa6-31514ef2c7f2"
                  },
                  "message": "transfer"
                },
                {
                  "id": "0mtlcmctg-8z00dmmisds",
                  "start": 1788429285988,
                  "status": "success",
                  "end": 1788429291289,
                  "message": "vmstart"
                }
              ],
              "end": 1788429296722,
              "message": "health check"
            },
            {
              "id": "0mtlcml5b-ywe7qa02f7q",
              "start": 1788429296783,
              "status": "success",
              "end": 1788429300723,
              "result": {
                "merge": false,
                "size": 0
              },
              "message": "clean-vm"
            }
          ],
          "end": 1788429300727,
          "message": "export",
          "data": {
            "id": "19270323-baaa-4a7b-bf1f-e152ae44f263",
            "isFull": false,
            "type": "remote"
          }
        }
      ],
      "warnings": [
        {
          "message": "can't compute delta OpaqueRef:afcf0af9-64c9-845d-207c-789c9c5de78f from OpaqueRef:13ce290f-2194-280e-72c5-9e16271505df, fall back to a full"
        },
        {
          "message": "can't connect through NBD, fall back to stream export"
        },
        {
          "message": "Backup fell back to a full"
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:afcf0af9-64c9-845d-207c-789c9c5de78f"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1788429300728,
      "message": "backup VM",
      "data": {
        "id": "0618e9fe-854e-7f01-1f22-6fb1667806d4",
        "type": "VM",
        "name_label": "Salem XO-CE",
        "progress": 0
      }
    },
    {
      "id": "0mtlc4783-s6m3pnf32pl",
      "start": 1788428438931,
      "status": "success",
      "tasks": [
        {
          "id": "0mtlc478g-epd65nptnrn",
          "start": 1788428438944,
          "status": "success",
          "end": 1788428439927,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        },
        {
          "id": "0mtlc486t-mq755lvlxi",
          "start": 1788428440181,
          "status": "success",
          "end": 1788428443614,
          "result": "d52e8853-1102-8bcd-b588-adbe5b411a6b",
          "message": "snapshot"
        },
        {
          "id": "0mtlc4au7-q4va6ntph6c",
          "start": 1788428443615,
          "status": "success",
          "tasks": [
            {
              "id": "0mtlc9pax-xyje1fo25lf",
              "start": 1788428695641,
              "status": "success",
              "end": 1788429704445,
              "result": {
                "size": 44032329216
              },
              "message": "transfer",
              "data": {
                "progress": 100
              }
            },
            {
              "id": "0mtlcvdy1-pxmgi1rx7tk",
              "start": 1788429707353,
              "status": "success",
              "tasks": [
                {
                  "id": "0mtlcvdzc-uodtj65grrn",
                  "start": 1788429707400,
                  "status": "success",
                  "end": 1788430481457,
                  "result": {
                    "size": 44021317632,
                    "id": "ed4d4ce7-2685-a35a-b1e4-c54a1385364e"
                  },
                  "message": "transfer"
                },
                {
                  "id": "0mtldbz96-dsndwbb93hv",
                  "start": 1788430481466,
                  "status": "success",
                  "end": 1788430486658,
                  "message": "vmstart"
                }
              ],
              "end": 1788430492516,
              "message": "health check"
            },
            {
              "id": "0mtldc7vt-oopf5wcg74e",
              "start": 1788430492649,
              "status": "success",
              "end": 1788430501065,
              "result": {
                "merge": false,
                "size": 0
              },
              "message": "clean-vm"
            }
          ],
          "end": 1788430501070,
          "message": "export",
          "data": {
            "id": "19270323-baaa-4a7b-bf1f-e152ae44f263",
            "isFull": false,
            "type": "remote"
          }
        }
      ],
      "warnings": [
        {
          "message": "can't compute delta OpaqueRef:82b55a23-b13e-df37-c9f7-11373a42c40b from OpaqueRef:9824e8f1-0199-885c-e69a-d9275cd62d2a, fall back to a full"
        },
        {
          "message": "can't connect through NBD, fall back to stream export"
        },
        {
          "message": "Backup fell back to a full"
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:82b55a23-b13e-df37-c9f7-11373a42c40b"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1788430501071,
      "message": "backup VM",
      "data": {
        "id": "64476a1a-20f5-32a4-6352-47916b2162d8",
        "type": "VM",
        "name_label": "Salem UnifiOS",
        "progress": 0
      }
    },
    {
      "id": "0mtlcmo92-ezud8mmu26",
      "start": 1788429300806,
      "status": "success",
      "tasks": [
        {
          "id": "0mtlcmoaf-ah3ohsdoy7r",
          "start": 1788429300855,
          "status": "success",
          "end": 1788429304315,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        },
        {
          "id": "0mtlcmrdk-xm266uwj1ni",
          "start": 1788429304856,
          "status": "success",
          "end": 1788429307794,
          "result": "dd0d0918-efdd-1a1d-ce94-85afbf0fd25c",
          "message": "snapshot"
        },
        {
          "id": "0mtlcmtn7-clidkdhy2ht",
          "start": 1788429307795,
          "status": "success",
          "tasks": [
            {
              "id": "0mtlcs7g8-3k59s9tshcf",
              "start": 1788429558968,
              "status": "success",
              "end": 1788430829213,
              "result": {
                "size": 43868711424
              },
              "message": "transfer",
              "data": {
                "progress": 100
              }
            },
            {
              "id": "0mtldjhbz-rfnh54dr3y",
              "start": 1788430831487,
              "status": "success",
              "tasks": [
                {
                  "id": "0mtldjhd9-g6rihy05ec7",
                  "start": 1788430831533,
                  "status": "success",
                  "end": 1788431285549,
                  "result": {
                    "size": 43857739776,
                    "id": "e3215b6b-9b5a-0d49-3392-491cc39c19e4"
                  },
                  "message": "transfer"
                },
                {
                  "id": "0mtldt7ox-r81abwrax4",
                  "start": 1788431285553,
                  "status": "success",
                  "end": 1788431290565,
                  "message": "vmstart"
                }
              ],
              "end": 1788431296314,
              "message": "health check"
            },
            {
              "id": "0mtldtg0h-8kvnbox5dg3",
              "start": 1788431296337,
              "status": "success",
              "end": 1788431297610,
              "result": {
                "merge": false,
                "size": 0
              },
              "message": "clean-vm"
            }
          ],
          "end": 1788431297611,
          "message": "export",
          "data": {
            "id": "19270323-baaa-4a7b-bf1f-e152ae44f263",
            "isFull": false,
            "type": "remote"
          }
        }
      ],
      "warnings": [
        {
          "message": "can't compute delta OpaqueRef:4b3d9930-d795-03f8-c887-f8b7d2a95524 from OpaqueRef:894ead1a-dae6-f156-d8c7-c7f5ba2200ef, fall back to a full"
        },
        {
          "message": "can't connect through NBD, fall back to stream export"
        },
        {
          "message": "Backup fell back to a full"
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:4b3d9930-d795-03f8-c887-f8b7d2a95524"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1788431297611,
      "message": "backup VM",
      "data": {
        "id": "ddf296c8-20a0-6aa9-0d74-ebc029d0d1b3",
        "type": "VM",
        "name_label": "Salem UnifiOS",
        "progress": 0
      }
    }
  ],
  "end": 1788431297621,
  "infos": [
    {
      "data": {
        "vms": [
          "0618e9fe-854e-7f01-1f22-6fb1667806d4",
          "64476a1a-20f5-32a4-6352-47916b2162d8",
          "ddf296c8-20a0-6aa9-0d74-ebc029d0d1b3"
        ]
      },
      "message": "vms"
    }
  ]
}

@florent I sent pm with support tunnel. Are you able to apply patch or will patch be pushed live soon?