@johnnezero if you copy/paste all those findings to a claude fable/opus or openAI Astra model, it's scarily good at fixing everything, just tell it to test test test

@johnnezero Hi! Thank you for your contribution to the vates / XCP ecosystem! It's certainly appreciated. However there seems to be a few "vibe-code-isms" in the plugin source that has caused us some support tickets on Vates side - the main issue is that you're doing two synchronous NFS operations (fs.existsSync & fs.appendFileSync) for every single log entry with no buffering or batching. Being synchronous, the entire node event loop locks up dead until the NFS call completes. So your plugin writing ~500 log lines whenever it runs at the top of the hour (or whenever else) is doing 2000+ NFS sync calls, locking up the node event loop for nearly 60 full seconds - meaning XOA loses the ability to contact hosts, perform operations, etc. This is made exponentially worse by the logger writing to XOA backup mounts, as when an XOA backup is running, the NFS share is typically quite loaded with backup traffic, making the logging sync operations take many times longer.

We've seen a couple customers whose backups started failing due to this which was interesting to track down - it appears as just the XOA appliance losing network connectivity to the pool, when in fact it's because xo-server is completely locked up waiting on logging activity from this plugin while the backup run is trying to call XAPI on the customer pool.

there's some other stuff like the enforcePerformance performing four writes over xapi on every single VM unconditionally every single call, instead of checking if the value is already set correctly and doesn't need a write. Stuff like vm.VCPUs_params and vm.other_config which XOA already intelligently maintains a cache of, so 99% of these routine calls should theoretically require zero xapi writes, or even reads. Here's a full analysis from Claude (I am not a dev, and our current XO devs are swamped so I didn't want to bother them with this )

The hourly enforcement cycle blocks the Node.js event loop for tens of seconds.
xo-server runs every pool connection in that same single threaded process, so
the block starves xen-api's event watcher. Its event.from long poll has a
60.1 second client side deadline, the timer cannot fire on time, and when it
does xen-api treats the connection as dead and reconnects. Reconnecting
flushes the XO object cache. Backup jobs starting in that window fail with
no such object <pool-uuid>.

The block is synchronous NFS I/O, one call per log line, inside loops over
hundreds of rows.

Evidence

Four _watchEvents TimeoutError events fired within the same second, one second
after the cycle's preload phase ended:

call	deadline	actual
1	60.1 s	99.4 s
2	60.1 s	67.1 s
3	60.1 s	66.4 s
4	60.1 s	66.2 s

Timers with deadlines spread across 33 seconds do not batch like that unless the
loop was starved and then released. The pool master logged no XAPI errors during
the window, and its session.login_with_password from the XOA address matches
the reconnect exactly.

Over a longer window, 18 timeout events since 1 August, 9 of them within 90
seconds of the top of an hour.

Findings

1. Synchronous NFS I/O in the logging path (critical)

writeLog (line 279) makes two blocking filesystem calls per log line, both
against the NFS mount:

if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });  // stat RPC
fs.appendFileSync(logPath, message + "\n", "utf8");               // open+write+close

logInfo (290) and logWarn (298) both call it, twice when summary is set.
About 270 log lines per cycle at roughly four round trips each is about 1100
blocking calls. Measured block was 39 seconds, implying about 35 ms per call,
consistent with the share being under backup write load at the time.

Other synchronous calls on the same mount: rotateLogs (231, 234, 244, 245,
258, 261), readLogTail (310, 311) which reads up to 10 MB, CSV reads (587,
619, 662, 750), and full CSV writes (645, 849).

console.log is not a contributor. Node writes to a pipe asynchronously.

2. Unconditional XAPI writes on every VM, every hour

enforcePerformance, lines 450 to 453:

await xapi.call("VM.remove_from_VCPUs_params", vm._xapiRef, "weight");
await xapi.call("VM.add_to_VCPUs_params",      vm._xapiRef, "weight", String(matched.weight));
await xapi.call("VM.remove_from_other_config", vm._xapiRef, "sched-pri");
await xapi.call("VM.add_to_other_config",      vm._xapiRef, "sched-pri", String(matched.ioPri));

No check for whether the value is already correct, though the cache already
holds vm.VCPUs_params and vm.other_config. Four writes per tagged VM per
hour, each journaled by XAPI and broadcast through event.from to every client,
including the process that is currently frozen. The plugin generates the event
storm it then cannot drain.

The remove-then-add pair is also non-atomic. Between the calls the VM has
neither weight nor sched-pri.

Same pattern for tags at 688 and 794, where VM.add_tags is called blind and a
duplicate key error is caught afterwards. The cached vm.tags already answers
that.

3. xo.getAllGroups() inside per-VM and per-tag loops

Line 501 in enforcePermissions, line 552 in applyPermissionTag. The latter is
called inline from runCsvSync (696) and processPreloadVms (831), multiplying
the cost.

4. xo.getXapi(vm) unguarded

Lines 681 and 786. It throws no connection found for object <uuid> when the
pool is disconnected, which is the state the plugin creates for itself. Neither
is wrapped, so one disconnected pool aborts the whole cycle through
runEnforcementCycle (891, which rethrows). enforcePerformance handles this
correctly at 446; the other two do not.

5. The preload queue never drains

processPreloadVms (742) removes a row only on a successful match (839). A row
naming a VM that does not exist, was decommissioned, lives on an unconnected
pool, or was mistyped is retried forever. The observed instance held 216 such
rows, producing 216 synchronous NFS writes per hour to report that nothing
happened. That is roughly 80 percent of the plugin's blocking I/O.

The message "not found or filtered" also conflates a missing VM with one
rejected by isRealVm, and the lookup at 774 searches across every connected
pool, taking whichever the index returns first on a name collision.

6. Repeated full object scans and quadratic lookups

Object.values(xo.getObjects({ type: "VM" })) is rebuilt at 430, 487, 583, 653,
749, 860, 996, and 1018, several in the same cycle. VM resolution is a linear
Array.find at 601, 678, and 774. The CSV is regenerated from the VM list, so
row count tracks VM count and runCsvSync is quadratic in pool size.

7. No guard against concurrent cycles

runEnforcementCycle is reachable from the cron job (905, 941), the test
action (1000), and xo-server-tag-automation.runSync (1005). None check whether
a cycle is running. A manual "Run Now" during the hourly tick doubles everything.

8. Scheduling

getCron (182) maps hourly to 0 * * * *, and createSchedule (905, 914,
941, 950) is called with no timezone, so it fires on the hour in appliance local
time. That collides with every other scheduled job, including backups.

9. Dead code in isRealVm

Lines 346 to 349. All three exact matches contain the substring tested on 346,
so the last three branches are unreachable.

10. Unverified object model

location	code	issue
337, 338	vm.$type and vm.type both tested	only one exists
404 to 413	four property fallback chain for notes	only one is the real field
601, 678	(v.uuid \|\| v.id)	both exist, with defined meanings

vm._xapiRef (450 to 453, 688, 720, 794, 813) is an internal field that can go
stale after a reconnect, and nothing revalidates it.

11. Minor

• rotateLogs (229) rotates only FILE_LOG; summary and daily logs grow without
  bound.
• writeRefreshedCsv (640) never quotes fields while parseCsvLine (194)
  handles quotes, so a value containing a double quote round trips incorrectly.
• The CSV is read twice per cycle (619, 662), three times with autopilot enabled
  (587).
• configure() (925) reassigns _config, but a cycle already in flight holds
  the old reference.

@gb.123 those scsi messages can be expected and ignored when a USB enclosure is connected, some USB enclosures do not emulate SCSI Enclosure Services (SES) very well, so the kernel complains when it queries them and gets nonsense back. USB passthrough devices are still visible and enumerated by dom0's kernel. If you remove the drive the messages will go away, but they can be safely ignored.

@r0ssar00 hi, that issue would arise if you ran this script with python3, but it's interpreter is set as /usr/bin/python - How did you call this script, did you manually call it with python3? It should be ran by just running the command on the CLI eg interface-rename

@jubin3 As this is totally unrelated to XOA and XCP-ng, you'll (hopefully) get a better response in the cloud-init community, as it's their project which has (once again) been broken by an OS update. I gave up chasing them some time ago, especially with brand new OS releases.

@rtjdamen Copying my reply to your official support ticket (any reason for duplicating support tickets on the forum as well?):

given XOA is built on standard debian, you can assign multiple IPs to the same interface quite easily by just duplicating another "iface eth1 inet static" line. Also keep in mind XOA does not add extra interfaces under the main /etc/network/interfaces file, but in files under the /etc/network/interfaces.d/ directory. So in your case given it was eth1 you wanted a second IP on, you can add your required second IP in this file like so:

[09:43 12] xoa:~$ cat /etc/network/interfaces.d/eth1
allow-hotplug eth1
iface eth1 inet static
 address 192.168.1.80
 netmask 255.255.255.0

#second IP
iface eth1 inet static
 address 172.16.100.5
 netmask 255.255.255.0

@abelaguilar indeed you do not have to fill out the dns and gateway fields - in fact as you surmised you shouldn't. Where you getting an error or something when leaving them blank? The only mandatory fields are IP and netmask.

@SNSNSN Indeed, these would typically at least be isolated via vlans at least (one vlan for iscsi traffic, one for iscsi). There's no point in having them in two different subnets if they're in the same network and vlan, the traffic isn't isolated at all. You might as well have them in the same subnet if you're doing that, in which case you only need 1 IP on the XCP-ng management NIC.

@SNSNSN Hi, this isn't possible, at least not without a lot of manual workarounds. It's not recommended anyhow, why do you need to assign another subnet to an adapter already in a different subnet? These should typically be isolated either physically via different connections, or via VLANs.

@olivierlambert never done it myself, but this is indeed exactly what the feature "Copy host BIOS strings to VM" was intended for as @Andrew mentioned. Hopefully the BIOS strings this feature copies are enough for the ROK installer to recognize the "authorized" dell hardware

Indeed, to properly edit iptables rules on xcp-ng, you need to add rules to /etc/sysconfig/iptables. I would copy something like the ssh allow line to another line directly below it, and change the port to 161 for example (and change protocol to udp, which I'm pretty sure your card uses, if it's just doing plain snmp). After verifying that fixes it, you can lock the rule down further by allowing this traffic only from the IP of the management card.

Example of added line below ssh line:

-A RH-Firewall-1-INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m conntrack --ctstate NEW -m udp --dport 694 -j ACCEPT
##UPS rule
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m udp --dport 161 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m conntrack --ctstate NEW -m tcp --dport 80 -j ACCEPT
etc
etc

Note that anytime you edit this file, you must restart iptables for it to take effect with service iptables restart

Thinking about this further though I don't think this should be necessary, as the ups daemon in dom0 is reaching out to the UPS card, not the other way around, so an explicit open port shouldn't be necessary with the default iptables in dom0 (which allows outbound conns)

@brm Hmm, I actually am not sure if we ever added support for this specifically (specifying an IP from IP pools in a cloud-init configuration). I've never seen IP variables used or referenced so I don't think it's currently possible. @olivierlambert who was it on the team that implemented the IP Pools feature?

@MrXeon So, the actual root issue here I believe, is opnsense installs come with an IP and dhcp server already assigned and enabled on the lan interface (I believe it's 192.168.1.1, but don't quote me). If your existing home network already uses 192.168.1.x/24 and already has a dhcp server, booting an opnsense install with it's virtual lan nic set to your existing home lan, there will be a lot of conflicts. Virtual nic order can be whatever you'd like (you can change and move around assignments in opnsense), but if it's preconfigured lan interface gets set to your preexisting lan network, there will be conflicts

Also note the text at the top of your screenshot: to continue you need to select a boot device. There might be a way in that menu (or partition creation submenu) to mark that created partition as bootable, or maybe you just need to highlight/select the partition under "used devices" before hitting "done"

@encryptblockr yup, welcome to cloud-init hell. Your issue is definitely ubuntu related though, if I had to guess, the installer wants/requires a swap partition. Just create a 1 or 2gb swap partition as well, but put it first in the partition table, so the root partition after it has room to grow. You'll also run into some network issues probably when trying to use your new template, as ubuntu has moved to new netplan crap to manage networking in the OS, and cloud-init has a ton of bugs with it

@furyflash777 I'm assuming you're on Ubuntu? Indeed as Olivier said this is tested on Debian and doesn't cause issues, but it seems on the newer Ubuntu versions with cloud-init, the new Netplan based network manager and how it interacts with cloud-init breaks/gets wiped if no cloud-init drive is found. Yet another cloud-init bug to track down

@jcdick1 I run opnsense on xcp-ng personally as well and use their packaged tools without issue, the only time I've gotten this behavior is when I hot-added interfaces and it changed the order of interfaces. If that's not it, I'm really not sure what would be causing this

One last thing you can try in case it's a weird cash issue is (inside XOA) go to settings > servers, click the green connected button next to your xcp-ng server to disconnect it from xoa - then wait a couple seconds and click it again to reconnect it

@jcdick1 Hi, have you hot-added any new network interfaces to this VM by chance? I've noticed when doing this with *bsd based guests like the *sense projects, the order can get quite messed up, if you've added any new interfaces, changed any MAC addresses, etc, can you please shut the VM down entirely (not just issue a reboot) - once the power state of the VM is completely off, start it again.

Note that if you did hot-add interfaces and hadn't rebooted yet, the interface order will probably change into its "final" order (the ordering seems to be affected when hot adding interfaces, eg when I hot add interfaces into *bsd VMs, sometimes the new interface will show up as xn0 in the VM, so the existing xn0 will get moved to xn3 etc). I've avoided this by just no longer hot adding interfaces and doing it when the VM is off instead

@tekwendell xen carefully manages CPU power management to match VM load and vCPU count, I would not manually try to adjust things with xenpm in the meantime as it's likely you'll make things worse (don't try to outsmart xen power management unless you have a VERY specific use case). Xen is designed for paralleled workloads (more than a single VM), so there's many tunables for VMs that are set with this in mind (like CPU affinity). So by default I'm sure the CPU affinity for your single windows VM is still set somewhere in the "middle", so it's not going to be allowed to schedule the full CPU time versus what dom0 is also using.

I'm not an expert in AMD/Epyc power management, but I believe it's pretty typical that CPU power/clock management boosts based on overall CPU load, and running a benchmark on only a single VM using something like 8 cores on a 64 core processor is not going to demand a lot CPU time, so I'm not surprised to see it's not boosting very far. Spin up 6 more of those VMs and benchmark them all at the same time, I wouldn't be surprised if you see it start boosting higher

475 cpu-z versus 501 bare metal is very good and indicates pretty clearly there's no issue here, you're getting 94% bare metal performance on windows under a large virtualization stack (historically the OS with the most overhead to virtualize). I would be very happy about this

If you really want to dig further, ensure your bios power management is set to "OS-controlled", this will hand more control over turbo and c-states to the xen power manager and is what is recommended on AMD processors, and then you can use some commands listed here to check actual turbo status. But again, note that I won't be surprised if you can't get a 64-core processor to enter its highest turbo states when only stressing 1/10th of its cores: https://support.citrix.com/article/CTX200390/power-settings-in-citrix-hypervisor-cstates-turbo-and-cpu-frequency-scaling