@lem2405 said: @acebmxer Is there a way to make the update feature completely unattended so it can be configured to run as a cron job? Regards Sure... Make sure you configure xo-config.cfg correctly. ./install-xen-orchestra.sh --update. It would only prompt for sudo passwor d. Available Functions Function CLI Flag Description Deploy --deploy Create a Debian VM on a XenServer/XCP-ng pool and install XO into it Build Templates --build-templates Build cloud-init VM templates on a XenServer/XCP-ng pool Install --install Fresh install of Xen Orchestra Update --update Update existing installation (with backup) Restore --restore Restore from a previous backup (verified for completeness first; add --list-backups to just list them) Rebuild --rebuild Fresh clone + clean build, preserves settings Reconfigure --reconfigure Apply config changes without rebuilding XO Proxy --proxy Deploy XO Proxy to a Xen pool master Adjust Memory --adjust-memory Raise the heap memory allocated to the xo-server process Status --status Read-only health report: version, service, TLS cert, disk/swap, backups/snapshots, git state Edit Config (menu only) Open xo-config.cfg in your preferred editor Rename Config (menu only) Rename sample-xo-config.cfg to xo-config.cfg Running without flags launches an interactive menu. All flags also work directly: ./install-xen-orchestra.sh # interactive menu ./install-xen-orchestra.sh --update # run update directly ./install-xen-orchestra.sh --help # show all options

@olivierlambert said: Hi, It's nice to suggest something, but we can't communicate just today about it, but be assured that we have stuff in the pipes to answer exactly all of this Hello Olivier, Understood completely on the timing—I appreciate that communications like this need to be perfectly aligned with the development cycle. It’s incredibly reassuring to hear that there is already a solution in the pipes! As someone who has been around the community since 2021, I’m more than happy to help keep the forum threads constructive and patient in the meantime. When the time comes, if you need any early testing, feedback, or GitHub contributions to help vet whatever you have planned, please feel free to loop me in. I’d be glad to help out. Looking forward to the announcement! Best, John

@jacob.becker Hi Jacob, I made a fix and it should be available on the next release

Now for my question that I would love input on. This DRBD dashboard states that there are some things out of sync. [image: 1789689457331-a1be50f7-d656-40db-a572-e2fb6d68a68a-image.jpeg] Which is fed from this metric [image: 1789689530955-37b05625-285e-496d-acb1-39184b0c3bd8-image-resized.jpeg] However linstor says differently. jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │ jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 v l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ InUse │ UpToDate │ Established(2) │ │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ xcp-sr-linstor_group_thin_device │ 0 │ 1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │ What is this pv? K8s worker vdi. Cool jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv s xcp-sr-linstor_group_thin_device | grep d07a41d6-0aa2-49a5-a241-b755c6daa3d8 │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/metadata │ {"name_label": "OVBH-VPROD-K8S01-WORKER02", "name_description": "", "is_a_snapshot": false, "snapshot_of": "", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │ │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/not-exists │ 0 │ │ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/volume-name │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ I notice it only states that some xcp-volumes-* are out of sync, and none of the pvs created by piraeus operator. Curious to know if anyone has any thoughts on this?

I just did a quick test to check. And it does indeed seem that this (small) issue has now been resolved. Tested on Windows Server 25, Management agent 9.2.385-0. Cheers!

Apologies for the long silence on my end — real life got in the way and I couldn't dig into this for a while. Finally found the time now to go through it again properly, helped a lot by the details others have already posted here in the meantime. I don't think this adds a new root cause, but it does corroborate what @kagbasi-wgsdac described — same "mass collapse onto a single anchor UUID" pattern, and in my case too that anchor is a real, still-in-use base disk, not a dangling/non-existent OpaqueRef (referencing vatesfr/xen-orchestra#9578). Environment: 2-host pool, shared NFS SR, XO recently updated to the latest version (didn't change anything regarding this issue, as expected since it's XAPI-side). Ran a read-only xe vdi-list params=uuid,name-label,is-a-snapshot,snapshot-of,snapshot-time directly against the pool master. Result: 47 of 196 VDIs match the pattern (is-a-snapshot: false but snapshot-of populated). All 47 point to the exact same single anchor UUID — which resolves to one specific, actively-used base VDI, not a missing reference. snapshot-time on most affected VDIs is the epoch default (1970-01-01T00:00:00Z), but a handful show plausible real dates (e.g. late 2025 / early 2026) — suggesting those entries originally had a legitimate snapshot-of relationship that got overwritten by whatever corrupted the metadata. Currently-running VMs affected (100% of each VM's disks affected in every case): VM OS Disks Sizes VM-1 Windows Server 2016 1 50 GB VM-2 Windows Server 2016 3 50 / 200 / 450 GB VM-3 Windows Server 2019 3 100 / 100 / 100 GB VM-4 Windows 11 Pro 2 50 / 100 GB VM-5 (decommissioned) Windows Server 2016 2 100 / 50 GB VM-6 Ubuntu 24.04 2 10 / 10 GB VM-7 Ubuntu 24.04 2 10 / 10 GB VM-8 Windows Server 2025 3 25 / 25 / 64 GB VM-9 Ubuntu 24.04 2 10 / 10 GB On top of that, ~28 more affected VDIs are orphaned/unattached objects ("base copy" leftovers, old ISO references) — same pattern, no VBD attached. Not touching any of this (no snapshot-fixer.py, no manual vdi-param-set) given the risk of severing legitimate snapshot relationships that's already been flagged here. Happy to provide a full anonymized xe vdi-list dump if that's useful for tracking down the root cause. Big thanks to everyone who kept digging into this and shared their findings here — especially @kagbasi-wgsdac for the detailed write-up that pointed me in the right direction, and of course @poddingue and the whole Vates team for staying on top of this and keeping us updated despite no clear timeline yet. Really appreciate the effort that goes into this, especially for something as tricky as a metadata corruption bug across production pools. Kind Regards and thx again Alex

@olivierlambert We are still seeing similar issues on fully updated xcp-ng 8.3 host. We do reboot every time it happens. Is there anything we can do to fix this otherwise or prevent this from happening (beside manually disconnecting the cifs iso storage before the san goes for a reboot)?

@dinhngtu Yes, I've submitted a false positive report here: https://www.bitdefender.com/en-us/business/submit

@pierrebrunet Tested and working again, thanks

@jr-m4 Hi, We have found this issue in another thread, it is now merged

Hello @mike.potapov , We found a race-condition happening on Qcow2 driver that explains this issue. We developed a fix that will be ship in the next release of blktap.

@acebmxer Very interesting. Thanks for sharing! I will wait until Veeam issued a new (hotfix) release before evaluating Veeam as a replacement for Xen Orchestra Backup.

@Emmanuel-V In my opinion, it would make more sense if a standalone disk exported in VMDK format were exported directly as monolithicSparse, so that it would not need to be converted. This makes more sense to me because when only the disk is exported, rather than the entire VM (OVA), it can be assumed that the disk will be attached directly to some VM.

@florent & @poddingue I now have two heap snapshots. Could you provide somewhere to upload them, please? And I will do so as soon as I can.

I do have active directory setup in my lab environment. I just need to set aside some time to try it out between two domain joined hosts. Thanks for trying that! That's some good debugging.