Tag-Based Automation Plugin: Tag-Based VM Performance & Permission Management via assigned tag(s)

johnnezero

UPDATE: Tag-Based-Automation Plugin

WHAT: The xo-tag-automation plugin is a native xo-server plugin that enforces VM performance tiers, manages group permissions, and synchronizes VM metadata -- all driven by VM tags and NFS-hosted CSV files (see below for more details). In general only apply one performance and/or permission action tag per VM. Note: Currently XO only allows one ACL assignment per VM (e.g. assigning "Dept1-Operator", and" Dept3-Admin" to the same VM does not work).

So brace yourself as there's a lot to chew below,, but I promise if your actively involved with a sizable migration project like we are (a.k.a. "Who can afford VMware anymore?"), it will be well worth your time once you have the plugin setup and running (especially in the "Set-it and Forget-it" Permissions AutoPiolot mode). Enjoy....

FEATURES

PERFORMANCE SYNC
Automatically applies CPU weights and IO priorities (sched-pri) based on VM performance tier tags:

0-core   -- CPU weight: 2048  IO priority: 7
1-high   -- CPU weight: 1024  IO priority: 7
2-normal -- CPU weight: 512   IO priority: 5
3-low    -- CPU weight: 256   IO priority: 2

All tier weights and IO priorities are fully configurable in the plugin UI. Note: Under normal conditions these performance settings do not affect anything. It's only when system resources are under high-load that the tiering levels ensure important VMs get priority over non-critical workloads, etc.

PERMISSION SYNC
VM tags ending in -Admin, -Operator, or -Viewer (e.g. "Dept1-Operator", "Dept2-Admin", etc.) automatically trigger appropriate XO Group creation and ACL assignments for the designated VM.

CSV PERMISSION MANAGEMENT FILES
Two CSV files on your NFS share drive the workflow:

current-vms.csv
A live export of your entire VM inventory. Contains UUID, Name, CurrentTags, NewTags, CurrentNotes, and NewNotes columns. Edit the NewTags and NewNotes columns and run the plugin (e.g. push the [Test plugin] button) to apply changes in bulk. The CSV auto-refreshes after each run. A staleness warning fires if the CSV has not been updated within a configurable number of days.

preload-vms.csv
Pre-stage tag and notes configurations for VMs that do not exist yet -- before they are migrated or created. The moment a VM matching a preload entry appears in XO, the plugin applies (on next scheduled interval) its tags and notes automatically and removes the entry from the preload file.

PERMISSION AUTOPILOT
Designed for active migration and onboarding projects. Automatically applies permission settings on scheduled interval, based on preload-vms.csv contents. Note: Should be disabled when not actively involved in migration projects.

DRY-RUN / EXPORT-CSV MODE
When Dry-Run is ON (the default), the plugin previews all changes in the XO logs without applying any changes, and simultaneously exports a fresh copy of all VM metadata to current-vms.csv with blank NewTags and NewNotes columns ready to fill in. Turn Dry-Run OFF to apply changes for real.

RUN NOW (e.g. The [Test plugin] button)
Trigger a full enforcement cycle instantly from the XO plugin UI without waiting for the next scheduled run.

NFS LOGGING
All activity is written to structured log files on your NFS share:

xo-tag-automation.log         -- full run log (auto-rotates at 2MB)
xo-tag-automation.log.1      -- previous log backup
xo-tag-automation-summary.log -- run summary entries only
daily-summary.log             -- nightly VM count and new VM report

Logs are also available via xo-cli API methods (see below).

LEGACY MIGRATION
If you have an existing vm_metadata.csv from an older version, the plugin automatically renames it to current-vms.csv on first run. No manual migration needed.

SECURITY -- PLEASE READ CAREFULLY

This plugin automates infrastructure changes. Security is not optional. Please take both of the following seriously before enabling Permission Sync or Autopilot.

-- REST API SERVICE ACCOUNT --

This plugin uses the XO JSON-RPC API internally. Use a dedicated service account (never your personal admin credentials). Note: REST API is only required for the Permissions Sync functions, it's not required if you just want to use the Performance Sync plugin functionality.

Recommended setup (generic -- adapt to your environment):

Create a dedicated XO user account for the service (e.g. a non-admin account with only the minimal access permissions required)
Generate a scoped API token via xo-cli:
xo-cli --register
xo-cli token.create
Store the token securely -- treat it like a password!
Refer to the official Vates REST API documentation for full token management guidance:
https://xen-orchestra.com/docs/restapi.html

-- NFS SHARE SECURITY --

The NFS share hosts your CSV files and logs. Anyone with write access to the share can modify VM tags and permissions.

STRONGLY RECOMMENDED

Run the NFS share from a dedicated VM -- not a general-purpose NFS server.
Restrict NFS exports to the XOA IP address only:
/srv/nfs/share<XOAIP>/32(rw,sync,no_subtree_check,no_root_squash)
Do NOT expose the NFS share to the general network or to end-user access.
Admins who need to edit CSV files can SCP them to/from the XOA:
EXAMPLE:
Download CSV from XOA to your workstation
scp <xoa-user>@<xoa-ip>:/path/to/current-vms.csv ./

Upload edited CSV back to XOA
scp ./current-vms.csv <xoa-user>@<xoa-ip>:/path/to/current-vms.csv
Use firewall rules to enforce NFS access at the network level in addition to the exports configuration

Failure to secure the NFS share is a serious security risk.

UPGRADING FROM THE STANDALONE SCRIPT

If you installed the old standalone set-performance.sh script, you can remove it before enabling the plugin as follows:

Remove the script:

sudo rm /usr/local/bin/set-performance.sh

Remove the crontab entry:

  crontab -e
  (delete the line referencing set-performance.sh)

Note: The plugin handles its own scheduling via the XO UI. No manual cron configuration is required.

INSTALLATION

Download the latest airgap release tarball from GitHub:
https://github.com/johnezero/xo-tag-automation_plugin/releases

SCP the tarball to your XOA:

scp xo-tag-automation-airgap-vX.X.X.tar.gz <xoa-user>@<xoa-ip>:/tmp/

Create the plugin folder:

sudo mkdir -p /usr/local/lib/node_modules/xo-server-tag-automation

Extract directly into the plugin directory:

sudo tar -xzvf /tmp/xo-tag-automation-airgap-vX.X.X.tar.gz -C /usr/local/lib/node_modules/xo-server-tag-automation/ --strip-components=1

Restart xo-server:
```
sudo systemctl restart xo-server
```

Verify registration:

sudo journalctl -u xo-server -n 100 --no-pager | grep -A3 "tag-automation"

You should see:

[INFO] xo-tag-automation: Plugin factory called -- xo context: YES
[INFO] xo-tag-automation: Plugin loaded -- waiting for core started.
xo:plugin INFO successfully register tag-automation

Enable and configure the plugin options in XO:
Settings -> Plugins -> tag-automation -> Enable

XO-CLI API METHODS

The plugin exposes several API methods accessible via xo-cli:

xo-cli xo-server-tag-automation.exportCsv
Export current VM inventory to current-vms.csv

xo-cli xo-server-tag-automation.downloadCsvApi
Print CSV content to stdout

xo-cli xo-server-tag-automation.uploadCsvApi content@./current-vms.csv
Push an edited CSV back to the NFS share

xo-cli xo-server-tag-automation.getLog lines=100
View the last N lines of the plugin log

xo-cli xo-server-tag-automation.getDailySummary
View the nightly VM count summary

xo-cli xo-server-tag-automation.getFilePaths
Show all configured file paths

LINKS

GitHub: https://github.com/johnezero/xo-tag-automation_plugin
Vates REST API docs: https://xen-orchestra.com/docs/restapi.html

STANDARD DISCLAIMER HERE

This software is provided "AS-IS" without any express or implied warranty. While this plugin is being used in a production environment managing live VMs, you should always review the code and test it in a non-production environment before full deployment.
Note: The plugin is designed to ONLY take action on VMs with specific tags assigned (e.g. untagged VMs are never modified).
Use Dry-Run mode to preview all changes (via log file output) before applying them.

Feedback, input and suggestions and always welcome...

poddingue

This is a really nice bit of work, thanks for building it and writing it up.
I'm not deep enough in xo-server plugin internals to give you useful feedback on the implementation, but the tag-driven approach to CPU weight and IO priority is clever; XO exposes those knobs manually in the VM advanced view, so automating them off tags is a neat layer on top.
The one-ACL-per-VM limitation you mention is interesting; I think that's a genuine gap rather than something you're missing, though I could be wrong.
If multiple ACL assignments per VM would help others too, it might be worth a short entry on https://feedback.vates.tech so the XO folks can see the demand, and @Team-XO-Backend might find the plugin itself interesting.

johnnezero

@poddingue Thank you for the feedback, and suggestion. I will post a request regarding the one-ACL-per-VM issue, as I know it's an issue with our current 600+ VM VMware-to-XCP-ng migration project. Happy Day

johnnezero

@poddingue FYI: I posted the "Feature Request" to the feedback channel here as you suggested.
Hopefully if enough people were used to the way VMware allowed "multiple granular permissions to be set per VM", it will get some traction. Thanks again for pointing me in the right direction.
Happy Day

poddingue

Thanks a lot. I voted for it.

MathieuRA

Hi @johnnezero and thanks for the post!

Just wanted to talk about PERMISSION SYNC.

In the REST API the "permission sync" pattern is actually handled natively by the RBAC system using selectors.

For example, if you want a role that allows a user to manage VM power state only for VMs tagged dev:

Start from the built-in role template “VMs power state manager” (just to speed up role creation, but totally optional)
Create or customize a role with the required VM power privileges (read, start, stop, reboot, etc.)
Scope each privileges using a selector like:
tags:dev
Then assign the role to your user or group

Once done, the access is fully dynamic:

Any VM with the dev tag is included in the scope
Removing the tag immediately revokes access
Adding the tag grants access instantly
No need to maintain per-VM ACL entries

The key point is that RBAC evaluates privileges at request time based on selectors.
You can also base selectors on other VM properties, not only tags (for example power state, name patterns ...).

You can find the doc here
and a dedicated forum thread here

PS: For the moment the XO6 UI does not support the RBAC system, but we are working on it

julienXOvates

@johnnezero
To add one more info on top of Mathieu's post : ACL (XO5 only) and RBAC (in REST API + XO6 soon) are completely separate, so better use RBAC as it gives much more flexibility !
Interesting plugin by the way !!!

johnnezero

@MathieuRA Wow, and wow! Thanks for the detail on RBAC, I'll have give it a look. Unfortunately the current plugin doesn't get any where near the "deep-weeds" of all that, and basically just uses the built-in "Admin, Operator and Viewer" roles to auto-provision (via Autopilot) new VMs as they show up (i.e. during a migration project, etc. ). We needed something for our current VMware-to-XCP-ng migration project, and '"Where's there's a will, there's a way" kicked in and the plugin came to be.
Thanks again for your input, and let me know if anything else comes to mind...
Happy Day

johnnezero

@julienXOvates Noted, thank you!

fohdeesha

@johnnezero Hi! Thank you for your contribution to the vates / XCP ecosystem! It's certainly appreciated. However there seems to be a few "vibe-code-isms" in the plugin source that has caused us some support tickets on Vates side - the main issue is that you're doing two synchronous NFS operations (fs.existsSync & fs.appendFileSync) for every single log entry with no buffering or batching. Being synchronous, the entire node event loop locks up dead until the NFS call completes. So your plugin writing ~500 log lines whenever it runs at the top of the hour (or whenever else) is doing 2000+ NFS sync calls, locking up the node event loop for nearly 60 full seconds - meaning XOA loses the ability to contact hosts, perform operations, etc. This is made exponentially worse by the logger writing to XOA backup mounts, as when an XOA backup is running, the NFS share is typically quite loaded with backup traffic, making the logging sync operations take many times longer.

We've seen a couple customers whose backups started failing due to this which was interesting to track down - it appears as just the XOA appliance losing network connectivity to the pool, when in fact it's because xo-server is completely locked up waiting on logging activity from this plugin while the backup run is trying to call XAPI on the customer pool.

there's some other stuff like the enforcePerformance performing four writes over xapi on every single VM unconditionally every single call, instead of checking if the value is already set correctly and doesn't need a write. Stuff like vm.VCPUs_params and vm.other_config which XOA already intelligently maintains a cache of, so 99% of these routine calls should theoretically require zero xapi writes, or even reads. Here's a full analysis from Claude (I am not a dev, and our current XO devs are swamped so I didn't want to bother them with this )

The hourly enforcement cycle blocks the Node.js event loop for tens of seconds.
xo-server runs every pool connection in that same single threaded process, so
the block starves xen-api's event watcher. Its event.from long poll has a
60.1 second client side deadline, the timer cannot fire on time, and when it
does xen-api treats the connection as dead and reconnects. Reconnecting
flushes the XO object cache. Backup jobs starting in that window fail with
no such object <pool-uuid>.

The block is synchronous NFS I/O, one call per log line, inside loops over
hundreds of rows.

Evidence

Four _watchEvents TimeoutError events fired within the same second, one second
after the cycle's preload phase ended:

call	deadline	actual
1	60.1 s	99.4 s
2	60.1 s	67.1 s
3	60.1 s	66.4 s
4	60.1 s	66.2 s

Timers with deadlines spread across 33 seconds do not batch like that unless the
loop was starved and then released. The pool master logged no XAPI errors during
the window, and its session.login_with_password from the XOA address matches
the reconnect exactly.

Over a longer window, 18 timeout events since 1 August, 9 of them within 90
seconds of the top of an hour.

Findings

1. Synchronous NFS I/O in the logging path (critical)

writeLog (line 279) makes two blocking filesystem calls per log line, both
against the NFS mount:

if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });  // stat RPC
fs.appendFileSync(logPath, message + "\n", "utf8");               // open+write+close

logInfo (290) and logWarn (298) both call it, twice when summary is set.
About 270 log lines per cycle at roughly four round trips each is about 1100
blocking calls. Measured block was 39 seconds, implying about 35 ms per call,
consistent with the share being under backup write load at the time.

Other synchronous calls on the same mount: rotateLogs (231, 234, 244, 245,
258, 261), readLogTail (310, 311) which reads up to 10 MB, CSV reads (587,
619, 662, 750), and full CSV writes (645, 849).

console.log is not a contributor. Node writes to a pipe asynchronously.

2. Unconditional XAPI writes on every VM, every hour

enforcePerformance, lines 450 to 453:

await xapi.call("VM.remove_from_VCPUs_params", vm._xapiRef, "weight");
await xapi.call("VM.add_to_VCPUs_params",      vm._xapiRef, "weight", String(matched.weight));
await xapi.call("VM.remove_from_other_config", vm._xapiRef, "sched-pri");
await xapi.call("VM.add_to_other_config",      vm._xapiRef, "sched-pri", String(matched.ioPri));

No check for whether the value is already correct, though the cache already
holds vm.VCPUs_params and vm.other_config. Four writes per tagged VM per
hour, each journaled by XAPI and broadcast through event.from to every client,
including the process that is currently frozen. The plugin generates the event
storm it then cannot drain.

The remove-then-add pair is also non-atomic. Between the calls the VM has
neither weight nor sched-pri.

Same pattern for tags at 688 and 794, where VM.add_tags is called blind and a
duplicate key error is caught afterwards. The cached vm.tags already answers
that.

3. `xo.getAllGroups()` inside per-VM and per-tag loops

Line 501 in enforcePermissions, line 552 in applyPermissionTag. The latter is
called inline from runCsvSync (696) and processPreloadVms (831), multiplying
the cost.

4. `xo.getXapi(vm)` unguarded

Lines 681 and 786. It throws no connection found for object <uuid> when the
pool is disconnected, which is the state the plugin creates for itself. Neither
is wrapped, so one disconnected pool aborts the whole cycle through
runEnforcementCycle (891, which rethrows). enforcePerformance handles this
correctly at 446; the other two do not.

5. The preload queue never drains

processPreloadVms (742) removes a row only on a successful match (839). A row
naming a VM that does not exist, was decommissioned, lives on an unconnected
pool, or was mistyped is retried forever. The observed instance held 216 such
rows, producing 216 synchronous NFS writes per hour to report that nothing
happened. That is roughly 80 percent of the plugin's blocking I/O.

The message "not found or filtered" also conflates a missing VM with one
rejected by isRealVm, and the lookup at 774 searches across every connected
pool, taking whichever the index returns first on a name collision.

6. Repeated full object scans and quadratic lookups

Object.values(xo.getObjects({ type: "VM" })) is rebuilt at 430, 487, 583, 653,
749, 860, 996, and 1018, several in the same cycle. VM resolution is a linear
Array.find at 601, 678, and 774. The CSV is regenerated from the VM list, so
row count tracks VM count and runCsvSync is quadratic in pool size.

7. No guard against concurrent cycles

runEnforcementCycle is reachable from the cron job (905, 941), the test
action (1000), and xo-server-tag-automation.runSync (1005). None check whether
a cycle is running. A manual "Run Now" during the hourly tick doubles everything.

8. Scheduling

getCron (182) maps hourly to 0 * * * *, and createSchedule (905, 914,
941, 950) is called with no timezone, so it fires on the hour in appliance local
time. That collides with every other scheduled job, including backups.

9. Dead code in `isRealVm`

Lines 346 to 349. All three exact matches contain the substring tested on 346,
so the last three branches are unreachable.

10. Unverified object model

location	code	issue
337, 338	`vm.$type` and `vm.type` both tested	only one exists
404 to 413	four property fallback chain for notes	only one is the real field
601, 678	`(v.uuid \\|\\| v.id)`	both exist, with defined meanings

vm._xapiRef (450 to 453, 688, 720, 794, 813) is an internal field that can go
stale after a reconnect, and nothing revalidates it.

11. Minor

rotateLogs (229) rotates only FILE_LOG; summary and daily logs grow without
bound.
writeRefreshedCsv (640) never quotes fields while parseCsvLine (194)
handles quotes, so a value containing a double quote round trips incorrectly.
The CSV is read twice per cycle (619, 662), three times with autopilot enabled
(587).
configure() (925) reassigns _config, but a cycle already in flight holds
the old reference.