Apologies for the long silence on my end — real life got in the way and I couldn't dig into this for a while. Finally found the time now to go through it again properly, helped a lot by the details others have already posted here in the meantime. I don't think this adds a new root cause, but it does corroborate what @kagbasi-wgsdac described — same "mass collapse onto a single anchor UUID" pattern, and in my case too that anchor is a real, still-in-use base disk, not a dangling/non-existent OpaqueRef (referencing vatesfr/xen-orchestra#9578). Environment: 2-host pool, shared NFS SR, XO recently updated to the latest version (didn't change anything regarding this issue, as expected since it's XAPI-side). Ran a read-only xe vdi-list params=uuid,name-label,is-a-snapshot,snapshot-of,snapshot-time directly against the pool master. Result: 47 of 196 VDIs match the pattern (is-a-snapshot: false but snapshot-of populated). All 47 point to the exact same single anchor UUID — which resolves to one specific, actively-used base VDI, not a missing reference. snapshot-time on most affected VDIs is the epoch default (1970-01-01T00:00:00Z), but a handful show plausible real dates (e.g. late 2025 / early 2026) — suggesting those entries originally had a legitimate snapshot-of relationship that got overwritten by whatever corrupted the metadata. Currently-running VMs affected (100% of each VM's disks affected in every case): VM OS Disks Sizes VM-1 Windows Server 2016 1 50 GB VM-2 Windows Server 2016 3 50 / 200 / 450 GB VM-3 Windows Server 2019 3 100 / 100 / 100 GB VM-4 Windows 11 Pro 2 50 / 100 GB VM-5 (decommissioned) Windows Server 2016 2 100 / 50 GB VM-6 Ubuntu 24.04 2 10 / 10 GB VM-7 Ubuntu 24.04 2 10 / 10 GB VM-8 Windows Server 2025 3 25 / 25 / 64 GB VM-9 Ubuntu 24.04 2 10 / 10 GB On top of that, ~28 more affected VDIs are orphaned/unattached objects ("base copy" leftovers, old ISO references) — same pattern, no VBD attached. Not touching any of this (no snapshot-fixer.py, no manual vdi-param-set) given the risk of severing legitimate snapshot relationships that's already been flagged here. Happy to provide a full anonymized xe vdi-list dump if that's useful for tracking down the root cause. Big thanks to everyone who kept digging into this and shared their findings here — especially @kagbasi-wgsdac for the detailed write-up that pointed me in the right direction, and of course @poddingue and the whole Vates team for staying on top of this and keeping us updated despite no clear timeline yet. Really appreciate the effort that goes into this, especially for something as tricky as a metadata corruption bug across production pools. Kind Regards and thx again Alex

@julienXOvates said: @acomav We will actually propose to change the date and time format in XO6 settings, so you would be able to choose between : YYYY-MM-DD MM/DD/YYYY DD/MM/YYYY and 12h or 24h time format. Hopefully in one of the next 3 months ! I hope that will answer your need, otherwise let me know ! Thanks for putting this one the roadmap, for one of the next 3 months. That date order issue, has been bugging me as if you are rushed off your feet, may not remember that it’s the US date order. Also fits in more with the rest of European software used, or configured for the European localisation.

@abudef Hi, Do you think this doc can help you? Link to doc I think it needs some improvements, feel free to enhance it if you feel it is necessary!

Thanks for the feedback, @dvinni !

@AtaxyaNetwork Thank you for sharing the latest version. It works very well except I'm getting error when trying to install any plugin. Did you get this as well ? [image: 1788936195627-d8c58b99-0330-4c85-830c-998b37004d59-image.jpeg]

On one machine in my lab, /etc/rsyslog.d/xenserver.conf defines 17 outchannels, every one of them at 104857600 bytes, and /var/log/messages isn't one of them; it's written by the stock *.info;mail.none;authpriv.none;cron.none line in /etc/rsyslog.conf and rotated by /etc/logrotate.d/syslog, so nightly and with no size trigger. The docs at https://docs.xcp-ng.org/guides/logs#rsyslog describe the 100 MiB rule but don't list which files it actually covers, which is probably why this is hard to work out without reading the config the way you did. Whether messages can realistically outrun a nightly rotation I don't know, and I'd be guessing if I said either way; the one thing in your favour is that /var/log is its own filesystem, 3.9 GB on the box I looked at (small lab machine, I haven't searched in the code, so I don't know where that size is decided), so it can't take the rest of dom0 with it. Whether that omission is deliberate or just inherited (or even something fancier) is really a question for whoever owns that file, so it might be worth a mention to @Team-OS-Platform-Release.

If you are running XOA, not XO from sources, then we could take a look remotely using the support tunnel. Otherwise, make sure that you have patched and rebooted each pool member.

@mpiton Thanks for looking into this, I guess it wasn't apparent that I needed to click the Save Configuration button for that plugin. I did that and confirmed that the secret now survives an xo-server restart. Thanks again!

I measured this on a two-NIC 8.3 host. As far as I understand, there's no way to do it from XO: the call that edits a network doesn't take a bond mode, every bondMode in the codebase sits on a create path, and the REST API on a running XOA offers create_bonded_network plus get and delete on a network, with nothing that edits one. The CLI route looks cheap though. xe bond-set-mode on a live bond cost zero dropped packets in both directions I tried, pinging every 200ms across the change, and the command returned in about 1.4 seconds. Two caveats, and the second is about your case specifically: creating and destroying the bond did interrupt the host (roughly 8 and 16 seconds), and I only went active-backup to balance-slb and back, never touching lacp, because my switch port isn't configured as a LAG and I'd have dropped the box. So I can't tell you your lacp to active-backup move is free, only that mode changes in general didn't cost me anything. Worth putting on https://feedback.vates.tech either way so the votes have somewhere to gather.

I went digging and three threads turned up people who already have this hardware. @yeopil21 in https://xcp-ng.org/forum/topic/11799 had an LPe37102-D on a Dell R770 that never showed up at all, @jinju0504 in https://xcp-ng.org/forum/topic/11946 ended up on the -alt package on an R670, and @lukas in https://xcp-ng.org/forum/topic/8847 was running an LPe31002-M6-D with NVMe-oF back on 8.3 beta2. That last one is the older generation, so I'd guess it's the side where you most want to hear nothing regressed. Tagging them in case the cards are still on their benches.

Thanks for the feedback.

I just did a quick test to check. And it does indeed seem that this (small) issue has now been resolved. Tested on Windows Server 25, Management agent 9.2.385-0. Cheers!

@olivierlambert We are still seeing similar issues on fully updated xcp-ng 8.3 host. We do reboot every time it happens. Is there anything we can do to fix this otherwise or prevent this from happening (beside manually disconnecting the cifs iso storage before the san goes for a reboot)?

@Andrew said: pcie_aspm=disable Just to help anyone who would run into it. Disabling ASPM via dom0 settings/kernel did not resolve the issue. Had to disable it in BIOS (NUC13) After kernel level disable it did show: lspci -vv -s 55:00.0 | grep -E 'LnkCap|LnkCtl|LnkSta' LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L0s <2us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+ LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Only after BIOS disable it showed lspci -vv -s 55:00.0 | grep -E 'LnkCap|LnkCtl|LnkSta' LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L1, Exit Latency L0s <2us, L1 <4us LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- Running on NUC13ANBi7 Hope it helps anyone running into this problem.

@olivierlambert I'm seeing a recurrence of the snapshot_of / hidden-disks issue after patching to xapi-26.1.16-1.2 (both hosts, rebooted). Unusual element this time: snapshot-fixer.py dry-run flags ~260 VDIs across the SR that all cite a single OpaqueRef which resolves to no VDI (xe vdi-list | grep <ref> returns nothing). Holding off on rewrite until I understand whether that single-dangling-ref pattern is safe, given the set appears to include base VDIs of legitimate snapshots. Full technical detail and questions posted on GitHub: https://github.com/xcp-ng/xcp/issues/844.

@andriy.sultanov I finally got around to fixing my capture card and I gotta say, it's a real pain to be using both methods and doesn't feel "correct" especially when there are warnings about possibly losing dom0 configurations after updates and the knowledge base saying that the old hide PCI from dom0 method is now considered obsolete. Perhaps the best solution would just be to provide users with a CLI command that would add that particular device to the xl pci-assignable-list if it doesn't fall into classes 01XX, 02XX or 03XX. Because I had added my GPU to my VM using the webUI, but then I also wanted to add the capture card through the console, but it appears that in the end you basically need to undo what's been done in the webUI and do it all through the console, otherwise you'd add the capture card manually with command line and then that would remove the GPUs... So I think some sort of manual override is necessary, even if it's a CLI option and then once that's done we could go into the webUI to enable it once the device has been marked as safe to passthrough.