@acebmxer that is exactley what i was saying before. if you disable the remote before nas shutdown you are good on the XO side and will work indefinitely, but if you don't execute umount -l /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70 you will have the mount stuck on the os side. You will be able to reboot XO vm eventually but you will have to wait for a long timeout during XO shutdown. if you want to simulate my issue just disable the remote 5 minutes after shutting down the synology, get the pid from service xo-server status then ps -T pid after a while you'll have 4 "rpc_wait_bit_killable" and XO will stop responding

Your reading of the code matches what I see on master, as far as I can tell. The object.enabled === true check in _getHosts() seems to have come in with PR #10209, released in XO 6.7.1 in August, to stop the balancer migrating VMs onto disabled hosts. If I follow the code right, a host that XO shuts down goes through disableHost first, so the power-on step in performance-plan.js no longer sees it. That looks like a side effect of the fix to me. The docs still say the performance plan powers halted hosts back on (https://docs.xen-orchestra.com/manage-your-infrastructure/load_balancing#plan-modes), so I think one of the two is wrong. I haven't reproduced it and there may be a reason to filter halted hosts that I can't see from the code, so @Team-XO-Backend is better placed to say if your powerState === 'Halted' exception is the right fix.

@tjkreidl said: @john.c Note that a two-host pool running HA can run into the split-brain issue if one of the hosts fails; it's highly recommended to have at least three hosts in an HA pool (or look into HA-Lizard if running a pool with two hosts and HA). Right but also considering the in development Twinstor for when it goes stable and in production!

@vmpr Popping it behind an instance of BunkerWeb (https://www.bunkerweb.io/) or similar, with Xen Orchestra on a new IP address would help. As BunkerWeb is what’s known as a Web Application Firewall. It will help to protect your instance of Xen Orchestra, especially if you change the IP address of the appliance, and only give that friend the one that goes through BunkerWeb. As always don’t expose Xen Orchestra to the Internet, but if it’s absolutely necessary always use something like BunkerWeb (with BunkerNET) and only provide the BunkerWeb based IP address! Also run a check on all of your hosts and may be best to roll them a fresh (in case there’s an Advanced Persistent Threat). As well as a fresh instance of Xen Orchestra, and maybe encourage your friend while also doing yourself to start using hardware security keys, preferably ones that support biometrics and PINs. Alternatively other secure enclaves with cryptographic credentials held there. Additionally opt into the new encrypted datastore for Xen Orchestra on the new fresh Xen Orchestra and XCP-ng. Also make sure it’s completely up to date! Also if the hosts support it ensure that Intel TME or similar is enabled on the fresh installs of XCP-ng. Finally rotate all passwords associated with that system, including your Vates account password, especially if it’s linked to the Xen Orchestra instance hit.

@jr-m4 I did not took time to test Podman yet. But we expect this to work. You'd have to expose a socket, like the docker daemon does. Probably systemctl enable --now podman.socket would be enough. More in this page https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html/building_running_and_managing_containers/using-the-container-tools-api and this one https://docs.podman.io/en/latest/markdown/podman-system-service.1.html

Thanks for the information, @ph7 . That's a typo, for sure.

Thanks @cocoon, and I owe a correction: when I said in https://xcp-ng.org/forum/post/104865 that 6.4 would make these alerts disappear for good, I overstated it. If I'm reading the code right this time, the XO server sends a ping every 30 seconds stamped with its own clock (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/rest-api/src/events/event.class.mts), and the browser compares that stamp with its own clock and shows the banner once the gap passes 40 seconds (https://github.com/vatesfr/xen-orchestra/blob/master/@xen-orchestra/web-core/lib/packages/remote-resource/sse.store.ts). So https://github.com/vatesfr/xen-orchestra/pull/9681 widened the margin from 32 to 40 seconds, but a browser whose clock runs ahead of the XO server by more than about 10 seconds would still see it. That might be why @GregBinSD saw it on the Windows PC and not on the tablet, though that's only a guess on my part. Comparing two different clocks looks fragile to me, but @Team-XO-Frontend will know far better than I do whether there's a reason it's done that way.

@acebmxer I haven't run your script, so I can't vouch for the log lines, but the systemctl status check looks like the safer of the two to me. It shows the binary the service is really running. Whichever node comes first in your shell can be a different one if more than one Node is installed. For XOA I'd go with what @Danp said above.

@bogdantomasciuc Glad to hear. I was hoping it was just something missed or miss typed. If you can replicate the issue let me know. Its possible i might have missed that little nuance or something.

Update - changes are live in the dev branch now. So figured out the update / switching branches for the proxy by connecting the proxy to my xoa free account. That triggered the available update to next version. From there I found the command that needs to be ran on the proxy to switch back and forth... [image: Screenshot_20261009_143546.png] Latest branch [image: Screenshot_20261009_144414.png] stable branch [image: Screenshot_20261008_202716.png] Backup job ran sucessful. 2026-10-09T17_48_25.447Z - backup NG.txt After the proxy has been deployed... The proxy has been registered with your Xen Orchestra instance. You can manage it from the Xen Orchestra web interface. Next: register the proxy's updater and move it to the 'latest' channel. On the pool master (IP), set a password for the proxy's 'xoa' user (choose your own) and reboot the proxy VM: xe vm-param-set uuid=UUID xenstore-data:vm-data/system-account-xoa-password='<your password>' xe vm-reboot uuid=UUID SSH to the proxy as 'xoa' (IP) and run: sudo xoa-updater register - This registers it with your vates account. sudo xoa-updater configure-channel xo-proxy-appliance-latest - set to latest or stable sudo xoa-updater upgrade - This can be ran via the UI or this command. Both need to be done twice to do the upgrade and/or downgrade. register needs a free xen-orchestra.com account; see the README. And host connected via proxy. [image: Screenshot_20261009_150011.png]

@jp13232 @peek have you tried following this: https://docs.xcp-ng.org/compute/#pci-passthrough ? Also, you can have a look there: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers if you finally manage to do the passthrough but have "vbios issues" like explained in the thread.

Data point from a 2-host XCP-ng 8.3 pool that supports @tosh's diagnosis, plus a no-patch workaround that gave us ~2.7x until the TCP_NODELAY fix lands. Setup XCP-ng 8.3, xapi-core / vhd-tool 26.1.16-1.2.xcpng8.3, dom0 kernel 4.19.0+1 Two hosts, local ext SRs on NVMe (Samsung 970 EVO Plus / WD Red SN700) Dedicated migration network: 2.5 GbE (Intel i226-V, igc), MTU 9000, no errors Offline storage migration (halted VM, VM.migrate_send) of 2 VDIs (disk + snapshot), 16 GiB virtual / 7773 MiB allocated each; sparse_dd ... -prezeroed ... -dest-proto nbd over TLS Baseline (stock) 132 s per VDI, i.e. ~59 MiB/s on the wire, flat line, same in both directions Link at ~20 % of line rate, source SR read latency 0.3 ms, dom0 total <= 0.5 vCPU, no single dom0 vCPU above 0.08 (RRD, 60 s averages) Sending host with turbo or capped at 2.5 GHz gave the same 132 s, so not CPU-bound here Workaround: disable delayed ACKs on the receiver, only for the migration-network route # on the destination host; adapt prefix/dev/src to the output of: ip route show <migration-net> ip route change 10.10.10.0/24 dev xenbr1 proto kernel scope link src 10.10.10.1 quickack 1 With the receiver ACKing immediately, Nagle on the sender only waits one RTT instead of the ~40 ms delayed-ACK timer. Result (same VM, same direction, same data) stock quickack 1 on receiver per VDI 132 s / 132 s 50 s / 46 s throughput ~59 MiB/s ~155-169 MiB/s whole migration 4:50 2:00 dom0 on the receiver went up to ~1.1 vCPU total, still nothing saturated. Caveats One run per arm, one pool. I did not capture ss -ti during the runs, so the notsent:512 signature is inferred, not observed here. It does not address the reply gating @TeddyAstie described, it only removes the delayed-ACK half. Consistent with that, we land at ~2.7x, below the ~4.3x reported above for TCP_NODELAY (different rig, so not directly comparable). Not persistent: lost on reboot and when xapi re-plugs the PIF. We re-apply it from a small systemd unit at boot as a stopgap and will remove it once vhd-tool ships TCP_NODELAY. Scope is only connections routed via the migration network (storage and live migration). More ACK packets on that link, nothing else changes. Is there a PR or issue for the TCP_NODELAY patch in xapi-project/xen-api that we can follow? I could not find one.

@MathieuRA We reverted back to 6.8.2 for now. We can wait for patch, otherwise, I have opened a tunnel #38081.

Forwarded to the right team, thanks!

@olivierlambert said: Hi, It's nice to suggest something, but we can't communicate just today about it, but be assured that we have stuff in the pipes to answer exactly all of this Hello Olivier, Understood completely on the timing—I appreciate that communications like this need to be perfectly aligned with the development cycle. It’s incredibly reassuring to hear that there is already a solution in the pipes! As someone who has been around the community since 2021, I’m more than happy to help keep the forum threads constructive and patient in the meantime. When the time comes, if you need any early testing, feedback, or GitHub contributions to help vet whatever you have planned, please feel free to loop me in. I’d be glad to help out. Looking forward to the announcement! Best, John

@jacob.becker Hi Jacob, I made a fix and it should be available on the next release

I just did a quick test to check. And it does indeed seem that this (small) issue has now been resolved. Tested on Windows Server 25, Management agent 9.2.385-0. Cheers!

Apologies for the long silence on my end — real life got in the way and I couldn't dig into this for a while. Finally found the time now to go through it again properly, helped a lot by the details others have already posted here in the meantime. I don't think this adds a new root cause, but it does corroborate what @kagbasi-wgsdac described — same "mass collapse onto a single anchor UUID" pattern, and in my case too that anchor is a real, still-in-use base disk, not a dangling/non-existent OpaqueRef (referencing vatesfr/xen-orchestra#9578). Environment: 2-host pool, shared NFS SR, XO recently updated to the latest version (didn't change anything regarding this issue, as expected since it's XAPI-side). Ran a read-only xe vdi-list params=uuid,name-label,is-a-snapshot,snapshot-of,snapshot-time directly against the pool master. Result: 47 of 196 VDIs match the pattern (is-a-snapshot: false but snapshot-of populated). All 47 point to the exact same single anchor UUID — which resolves to one specific, actively-used base VDI, not a missing reference. snapshot-time on most affected VDIs is the epoch default (1970-01-01T00:00:00Z), but a handful show plausible real dates (e.g. late 2025 / early 2026) — suggesting those entries originally had a legitimate snapshot-of relationship that got overwritten by whatever corrupted the metadata. Currently-running VMs affected (100% of each VM's disks affected in every case): VM OS Disks Sizes VM-1 Windows Server 2016 1 50 GB VM-2 Windows Server 2016 3 50 / 200 / 450 GB VM-3 Windows Server 2019 3 100 / 100 / 100 GB VM-4 Windows 11 Pro 2 50 / 100 GB VM-5 (decommissioned) Windows Server 2016 2 100 / 50 GB VM-6 Ubuntu 24.04 2 10 / 10 GB VM-7 Ubuntu 24.04 2 10 / 10 GB VM-8 Windows Server 2025 3 25 / 25 / 64 GB VM-9 Ubuntu 24.04 2 10 / 10 GB On top of that, ~28 more affected VDIs are orphaned/unattached objects ("base copy" leftovers, old ISO references) — same pattern, no VBD attached. Not touching any of this (no snapshot-fixer.py, no manual vdi-param-set) given the risk of severing legitimate snapshot relationships that's already been flagged here. Happy to provide a full anonymized xe vdi-list dump if that's useful for tracking down the root cause. Big thanks to everyone who kept digging into this and shared their findings here — especially @kagbasi-wgsdac for the detailed write-up that pointed me in the right direction, and of course @poddingue and the whole Vates team for staying on top of this and keeping us updated despite no clear timeline yet. Really appreciate the effort that goes into this, especially for something as tricky as a metadata corruption bug across production pools. Kind Regards and thx again Alex