6.8.2 went out, and it carries florent's proxy patch, so the thing that was stopping you switching channels has cleared. Before you do, though, the 6.8.2 changelog lists exactly one bug fix, the fetch failed proxy one, with xo-server at 5.208.3 and the proxy at 0.31.10, and there's nothing in it about metadata backups or BodyTimeoutError. Your own post above points the same way, since you're seeing the stall on sources at the latest commit, and that commit already has the proxy fix in it. So I'd treat the metadata timeout as a separate bug that 6.8.2 won't touch, and I couldn't find an open issue covering it, though I might well have searched the wrong words. Might be worth a mention to @Team-XO-Backend, mostly so the metadata half doesn't quietly get filed as fixed by 6.8.2.

You already spotted this over on 12468, so this is just to close the loop here. That one's fixed: Pierre Brunet put the fix on a branch, @Andrew and @abudef both tested it, and it's merged now. Since you're running from source on master, you should be able to pull past that merge rather than wait for a release. One thing to keep in mind, though: the same error string showed up in 11875 for a completely different cause, so if it comes back after you update, it's probably not this bug again. I'd let 12468 be the thread to follow since that's where the people who fixed it are.

You're not blind, and your fingerprint is right. I pulled packer-plugin-xenserver_v0.11.4_SHA256SUMS.sig off the GitHub release and ran gpg --list-packets on it, and it reports issuer fpr v4 0DD2E92A7A0F682BC29F87073084D82948625D89, which is exactly what you pasted. It doesn't match the XCP-ng key because it isn't ours. The plugin our docs point at is ddelnano/packer-plugin-xenserver, maintained outside Vates, so it's signed with the maintainer's key while 34AC2EB6... only signs XCP-ng RPMs. I couldn't find that key published anywhere either. It's not on keys.openpgp.org or keyserver.ubuntu.com, it isn't in the repo, and it isn't among the 11 keys at github.com/ddelnano.gpg. That looks like something worth raising on the plugin's own issue tracker, and you'd file it better than I would since you're the one trying to verify a download. I'd also say our IaC docs could stand to mention the plugin isn't ours, which is what sent you looking at the wrong key in the first place.

@abudef asked almost exactly this a few days ago over on 12466, and @pierrebrunet answered it there with a link to the doc describing the layout, so that thread's probably your quickest route. For the specific "which VM is this folder" question, the UUID in the directory name is the VM's own UUID, so xe vm-list uuid=<that-uuid> params=name-label on a host should give you the name back. One thing that bite me when I tried it: if the VM has since been deleted, that command prints nothing at all and still exits cleanly, so silence means "no such VM here" rather than "the command didn't work". Those silent ones are probably where your space has gone, if I had to guess. I'm not the backup expert here so someone may well have a tidier way of doing it.

Thanks for doing that, @john.c. The sponsorship side isn't mine to speak to; I forwarded to the right person The CFP I can say something about: nor(DEV):con is already in the conference pipeline I keep an eye on, and their Sessionize call closes on 30 September, so your timing is better than you probably realised. I can't promise you'll see us in Norwich, but the nudge landed in front of the right people.

Forwarded to the right team, thanks!

Small update — I split the custom plugins out of this project into their own repo, for anyone who just wants the plugins without pulling in the whole install script: https://github.com/acebmxer/xo-plugins Same deal as always — use at your own risk, review the code before running it on anything that matters. The two plugins in there are also still shipped inside this install script's Custom Plugins menu, kept in sync automatically — this repo just exists for people who don't want the rest of the project. xo-server-nanokvm This one's probably the more useful of the two for a lot of people. If you've got a host with no iLO/DRAC/IPMI — most consumer/prosumer boards, a lot of homelab gear — and you've wired up a Sipeed NanoKVM to the power header, this plugin lets Xen Orchestra power that host back on through the NanoKVM's own API. Same interface the NanoKVM web UI itself uses to press the button, just done from XO. On its own it doesn't decide when to turn a host on, it just gives XO a way to do it. Pairs with the other plugin below for that, or you could call it from your own automation if you wanted. Worth knowing: it can only press the button, it has no way to know if the host is actually on or off, so it only handles power-on. Powering off goes through XO's normal shutdown, which is a clean OS shutdown and evacuates VMs first — no reason to route that through the NanoKVM. Setup is a config entry per host: label, the NanoKVM's URL, a login, and which XO host it's wired to. Recommend making it a dedicated user-role account on the NanoKVM rather than admin — that role already has power/reset access without giving the plugin anything to storage/network settings on the KVM itself. xo-server-host-power-manager This is the one that actually decides when to act. Point it at an "extra" host in the pool and give it CPU and/or memory thresholds — when the rest of the pool is under pressure it powers that host on, and once things calm down for a while it powers it back off. Power-on can go through XO's built-in methods or through the NanoKVM plugin above, your choice per rule. Powering off always goes through XO's own host shutdown — it evacuates the running VMs first, and if HA is on and doesn't have room to cover it, XAPI just refuses and the plugin backs off and tries again later rather than forcing anything. It's deliberately quick to scale up and slow to scale down (needs both CPU and memory comfortable for a full cooldown period before it'll power a host off) so it's not flapping a host on and off over a short spike. Both have a Test button in their config page that actually tells you something useful, unlike XO's own generic "test plugin" popup — check journalctl -u xo-server right after clicking it to see what it found. As always, happy to hear feedback or find out I've broken something.

Have there been any updates to this? I'm haven't seen any activity here for a while.

@olivierlambert said: Hi, It's nice to suggest something, but we can't communicate just today about it, but be assured that we have stuff in the pipes to answer exactly all of this Hello Olivier, Understood completely on the timing—I appreciate that communications like this need to be perfectly aligned with the development cycle. It’s incredibly reassuring to hear that there is already a solution in the pipes! As someone who has been around the community since 2021, I’m more than happy to help keep the forum threads constructive and patient in the meantime. When the time comes, if you need any early testing, feedback, or GitHub contributions to help vet whatever you have planned, please feel free to loop me in. I’d be glad to help out. Looking forward to the announcement! Best, John

@jacob.becker Hi Jacob, I made a fix and it should be available on the next release

Now for my question that I would love input on. This DRBD dashboard states that there are some things out of sync. [image: a1be50f7-d656-40db-a572-e2fb6d68a68a-image.jpeg] Which is fed from this metric [image: 37b05625-285e-496d-acb1-39184b0c3bd8-image.jpeg] However linstor says differently. jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │ jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 v l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ InUse │ UpToDate │ Established(2) │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │ What is this pv? K8s worker vdi. Cool jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv s xcp-sr-linstor_group_thin_device | grep d07a41d6-0aa2-49a5-a241-b755c6daa3d8 │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/metadata │ {"name_label": "OVBH-VPROD-K8S01-WORKER02", "name_description": "", "is_a_snapshot": false, "snapshot_of": "", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │ │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/not-exists │ 0 │ │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/volume-name │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ I notice it only states that some xcp-volumes-* are out of sync, and none of the pvs created by piraeus operator. Curious to know if anyone has any thoughts on this?

I just did a quick test to check. And it does indeed seem that this (small) issue has now been resolved. Tested on Windows Server 25, Management agent 9.2.385-0. Cheers!

Apologies for the long silence on my end — real life got in the way and I couldn't dig into this for a while. Finally found the time now to go through it again properly, helped a lot by the details others have already posted here in the meantime. I don't think this adds a new root cause, but it does corroborate what @kagbasi-wgsdac described — same "mass collapse onto a single anchor UUID" pattern, and in my case too that anchor is a real, still-in-use base disk, not a dangling/non-existent OpaqueRef (referencing vatesfr/xen-orchestra#9578). Environment: 2-host pool, shared NFS SR, XO recently updated to the latest version (didn't change anything regarding this issue, as expected since it's XAPI-side). Ran a read-only xe vdi-list params=uuid,name-label,is-a-snapshot,snapshot-of,snapshot-time directly against the pool master. Result: 47 of 196 VDIs match the pattern (is-a-snapshot: false but snapshot-of populated). All 47 point to the exact same single anchor UUID — which resolves to one specific, actively-used base VDI, not a missing reference. snapshot-time on most affected VDIs is the epoch default (1970-01-01T00:00:00Z), but a handful show plausible real dates (e.g. late 2025 / early 2026) — suggesting those entries originally had a legitimate snapshot-of relationship that got overwritten by whatever corrupted the metadata. Currently-running VMs affected (100% of each VM's disks affected in every case): VM OS Disks Sizes VM-1 Windows Server 2016 1 50 GB VM-2 Windows Server 2016 3 50 / 200 / 450 GB VM-3 Windows Server 2019 3 100 / 100 / 100 GB VM-4 Windows 11 Pro 2 50 / 100 GB VM-5 (decommissioned) Windows Server 2016 2 100 / 50 GB VM-6 Ubuntu 24.04 2 10 / 10 GB VM-7 Ubuntu 24.04 2 10 / 10 GB VM-8 Windows Server 2025 3 25 / 25 / 64 GB VM-9 Ubuntu 24.04 2 10 / 10 GB On top of that, ~28 more affected VDIs are orphaned/unattached objects ("base copy" leftovers, old ISO references) — same pattern, no VBD attached. Not touching any of this (no snapshot-fixer.py, no manual vdi-param-set) given the risk of severing legitimate snapshot relationships that's already been flagged here. Happy to provide a full anonymized xe vdi-list dump if that's useful for tracking down the root cause. Big thanks to everyone who kept digging into this and shared their findings here — especially @kagbasi-wgsdac for the detailed write-up that pointed me in the right direction, and of course @poddingue and the whole Vates team for staying on top of this and keeping us updated despite no clear timeline yet. Really appreciate the effort that goes into this, especially for something as tricky as a metadata corruption bug across production pools. Kind Regards and thx again Alex

@olivierlambert We are still seeing similar issues on fully updated xcp-ng 8.3 host. We do reboot every time it happens. Is there anything we can do to fix this otherwise or prevent this from happening (beside manually disconnecting the cifs iso storage before the san goes for a reboot)?

@dinhngtu Yes, I've submitted a false positive report here: https://www.bitdefender.com/en-us/business/submit

@pierrebrunet Tested and working again, thanks

@jr-m4 Hi, We have found this issue in another thread, it is now merged

Hello @mike.potapov , We found a race-condition happening on Qcow2 driver that explains this issue. We developed a fix that will be ship in the next release of blktap.