@pierrebrunet Tested and working again, thanks

Small update — I split the custom plugins out of this project into their own repo, for anyone who just wants the plugins without pulling in the whole install script: https://github.com/acebmxer/xo-plugins Same deal as always — use at your own risk, review the code before running it on anything that matters. The two plugins in there are also still shipped inside this install script's Custom Plugins menu, kept in sync automatically — this repo just exists for people who don't want the rest of the project. xo-server-nanokvm This one's probably the more useful of the two for a lot of people. If you've got a host with no iLO/DRAC/IPMI — most consumer/prosumer boards, a lot of homelab gear — and you've wired up a Sipeed NanoKVM to the power header, this plugin lets Xen Orchestra power that host back on through the NanoKVM's own API. Same interface the NanoKVM web UI itself uses to press the button, just done from XO. On its own it doesn't decide when to turn a host on, it just gives XO a way to do it. Pairs with the other plugin below for that, or you could call it from your own automation if you wanted. Worth knowing: it can only press the button, it has no way to know if the host is actually on or off, so it only handles power-on. Powering off goes through XO's normal shutdown, which is a clean OS shutdown and evacuates VMs first — no reason to route that through the NanoKVM. Setup is a config entry per host: label, the NanoKVM's URL, a login, and which XO host it's wired to. Recommend making it a dedicated user-role account on the NanoKVM rather than admin — that role already has power/reset access without giving the plugin anything to storage/network settings on the KVM itself. xo-server-host-power-manager This is the one that actually decides when to act. Point it at an "extra" host in the pool and give it CPU and/or memory thresholds — when the rest of the pool is under pressure it powers that host on, and once things calm down for a while it powers it back off. Power-on can go through XO's built-in methods or through the NanoKVM plugin above, your choice per rule. Powering off always goes through XO's own host shutdown — it evacuates the running VMs first, and if HA is on and doesn't have room to cover it, XAPI just refuses and the plugin backs off and tries again later rather than forcing anything. It's deliberately quick to scale up and slow to scale down (needs both CPU and memory comfortable for a full cooldown period before it'll power a host off) so it's not flapping a host on and off over a short spike. Both have a Test button in their config page that actually tells you something useful, unlike XO's own generic "test plugin" popup — check journalctl -u xo-server right after clicking it to see what it found. As always, happy to hear feedback or find out I've broken something.

@olivierlambert said: Hi, It's nice to suggest something, but we can't communicate just today about it, but be assured that we have stuff in the pipes to answer exactly all of this Hello Olivier, Understood completely on the timing—I appreciate that communications like this need to be perfectly aligned with the development cycle. It’s incredibly reassuring to hear that there is already a solution in the pipes! As someone who has been around the community since 2021, I’m more than happy to help keep the forum threads constructive and patient in the meantime. When the time comes, if you need any early testing, feedback, or GitHub contributions to help vet whatever you have planned, please feel free to loop me in. I’d be glad to help out. Looking forward to the announcement! Best, John

@dinhngtu Yes, I've submitted a false positive report here: https://www.bitdefender.com/en-us/business/submit

Now for my question that I would love input on. This DRBD dashboard states that there are some things out of sync. [image: a1be50f7-d656-40db-a572-e2fb6d68a68a-image.jpeg] Which is fed from this metric [image: 37b05625-285e-496d-acb1-39184b0c3bd8-image.jpeg] However linstor says differently. jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │ jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 v l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ InUse │ UpToDate │ Established(2) │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │ What is this pv? K8s worker vdi. Cool jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv s xcp-sr-linstor_group_thin_device | grep d07a41d6-0aa2-49a5-a241-b755c6daa3d8 │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/metadata │ {"name_label": "OVBH-VPROD-K8S01-WORKER02", "name_description": "", "is_a_snapshot": false, "snapshot_of": "", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │ │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/not-exists │ 0 │ │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/volume-name │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ I notice it only states that some xcp-volumes-* are out of sync, and none of the pvs created by piraeus operator. Curious to know if anyone has any thoughts on this?

@jr-m4 Hi, We have found this issue in another thread, it is now merged

@Emmanuel-V In my opinion, it would make more sense if a standalone disk exported in VMDK format were exported directly as monolithicSparse, so that it would not need to be converted. This makes more sense to me because when only the disk is exported, rather than the entire VM (OVA), it can be assumed that the disk will be attached directly to some VM.

Have there been any updates to this? I'm haven't seen any activity here for a while.

@jacob.becker Hi Jacob, I made a fix and it should be available on the next release

I just did a quick test to check. And it does indeed seem that this (small) issue has now been resolved. Tested on Windows Server 25, Management agent 9.2.385-0. Cheers!

Apologies for the long silence on my end — real life got in the way and I couldn't dig into this for a while. Finally found the time now to go through it again properly, helped a lot by the details others have already posted here in the meantime. I don't think this adds a new root cause, but it does corroborate what @kagbasi-wgsdac described — same "mass collapse onto a single anchor UUID" pattern, and in my case too that anchor is a real, still-in-use base disk, not a dangling/non-existent OpaqueRef (referencing vatesfr/xen-orchestra#9578). Environment: 2-host pool, shared NFS SR, XO recently updated to the latest version (didn't change anything regarding this issue, as expected since it's XAPI-side). Ran a read-only xe vdi-list params=uuid,name-label,is-a-snapshot,snapshot-of,snapshot-time directly against the pool master. Result: 47 of 196 VDIs match the pattern (is-a-snapshot: false but snapshot-of populated). All 47 point to the exact same single anchor UUID — which resolves to one specific, actively-used base VDI, not a missing reference. snapshot-time on most affected VDIs is the epoch default (1970-01-01T00:00:00Z), but a handful show plausible real dates (e.g. late 2025 / early 2026) — suggesting those entries originally had a legitimate snapshot-of relationship that got overwritten by whatever corrupted the metadata. Currently-running VMs affected (100% of each VM's disks affected in every case): VM OS Disks Sizes VM-1 Windows Server 2016 1 50 GB VM-2 Windows Server 2016 3 50 / 200 / 450 GB VM-3 Windows Server 2019 3 100 / 100 / 100 GB VM-4 Windows 11 Pro 2 50 / 100 GB VM-5 (decommissioned) Windows Server 2016 2 100 / 50 GB VM-6 Ubuntu 24.04 2 10 / 10 GB VM-7 Ubuntu 24.04 2 10 / 10 GB VM-8 Windows Server 2025 3 25 / 25 / 64 GB VM-9 Ubuntu 24.04 2 10 / 10 GB On top of that, ~28 more affected VDIs are orphaned/unattached objects ("base copy" leftovers, old ISO references) — same pattern, no VBD attached. Not touching any of this (no snapshot-fixer.py, no manual vdi-param-set) given the risk of severing legitimate snapshot relationships that's already been flagged here. Happy to provide a full anonymized xe vdi-list dump if that's useful for tracking down the root cause. Big thanks to everyone who kept digging into this and shared their findings here — especially @kagbasi-wgsdac for the detailed write-up that pointed me in the right direction, and of course @poddingue and the whole Vates team for staying on top of this and keeping us updated despite no clear timeline yet. Really appreciate the effort that goes into this, especially for something as tricky as a metadata corruption bug across production pools. Kind Regards and thx again Alex

@olivierlambert We are still seeing similar issues on fully updated xcp-ng 8.3 host. We do reboot every time it happens. Is there anything we can do to fix this otherwise or prevent this from happening (beside manually disconnecting the cifs iso storage before the san goes for a reboot)?

Hello @mike.potapov , We found a race-condition happening on Qcow2 driver that explains this issue. We developed a fix that will be ship in the next release of blktap.

@acebmxer Very interesting. Thanks for sharing! I will wait until Veeam issued a new (hotfix) release before evaluating Veeam as a replacement for Xen Orchestra Backup.

@florent & @poddingue I now have two heap snapshots. Could you provide somewhere to upload them, please? And I will do so as soon as I can.