Follow-up on the questions left open in this thread. Three things came out differently from what I said above, so corrections first, and my earlier headline numbers need revising upward. My earlier ratios were too low: the rig was the bottleneck Everything I posted before ran NVMe to NVMe, where the destination disk saturates around 250 to 300 MB/s and degrades as it fills. That capped the faster arms, arm C worst of all. Rebuilt with RAM-backed SRs on both hosts, no storage ceiling, full four-arm matrix on one rig, 3 runs per arm, interleaved. arm build MiB/s range vs stock stalls B/seg A stock control 71.3 71.2-71.4 1.00x 78.3% 1441.6 B TCP_NODELAY only +socket 303.8 303.1-305.0 4.26x 1.5% 1426.6 C NODELAY + pipelining both 526.6 522.8-528.5 7.39x 1.1% 1425.7 D pipelining only +pipeline 528.3 523.0-531.9 7.41x 16.1% 1442.0 [image: 1788093035722-af369410-36dc-4a28-b822-35a00f7098f3-image.jpeg] Per-arm spread is 0.3% to 1.7%. The full stack is worth 7.4x, not 5.51x, and TCP_NODELAY alone is worth 4.26x, not 2.80x. The two changes are substitutes, not complements: TCP_NODELAY alone: 4.26x pipelining alone: 7.41x both together: 7.39x So: Pipelining alone captures the whole win. Adding the socket option on top of it is 0.997x, with overlapping ranges. Nothing. The reverse is not true: pipelining on top of NODELAY is still worth 1.73x. NODELAY does not substitute for pipelining. Pipelining substitutes for NODELAY, on throughput. [image: 1788093043264-2ab335d6-fe4b-4019-a8f9-c32d12e971bd-image.jpeg] What the socket option still does after pipelining is remove the stall, 16.1% of samples down to 1.1%, for 1.14% more segments. On this rig that buys no throughput. See the RTT caveat before writing it off. Correction 1: I said Chunked might regress. It does. xe vdi-copy between two host-local SRs, 100 GiB, interleaved A/B/A/B. @TeddyAstie was right. arm transfer data_segs_out B/seg stalls A control 442.1 s 74,746,948 1438.5 0.0% A2 control 439.4 s 74,677,058 1439.9 0.0% B NODELAY 449.5 s 75,205,400 1430.1 0.0% B2 NODELAY 455.5 s 75,260,858 1428.7 0.0% [image: 1788093061682-872deeb0-4514-4f82-96c0-ca3f96564f07-image.jpeg] Both controls beat both patched runs with no overlap, so this is not drift. TCP_NODELAY costs +2.7% wall clock and +0.70% segments here, and buys nothing. Nothing, because the stall it fixes does not occur on this path: Sub-MSS stalls are 0.0% in every arm, including the unpatched control. Chunked writes a 12 byte header then data, with no per-request reply. So the send queue stays backed up (median notsent about 3.3 MB, rwnd_limited 99.7%). Nagle never faces a small-segment decision. Packing is already 0.993x MSS before the patch. Where the extra packets come from, and it is not mainly the headers: The disk is fully allocated, so expand_copy (f.ml:2745) splits at its 2 MiB cap: 51,200 chunks. That is only 9.8% of the 521,126 extra segments. The rest is TLS record boundaries. The channel is unbuffered (channels.ml:129), so each 16 KiB TLS record is its own 16,413 byte write, 11.33x MSS, leaving a sub-MSS remainder. With Nagle those coalesce with the next record. With NODELAY they go out uncoalesced whenever the send queue drains. strace confirms both shapes: 41 byte writes are the headers (12 plus 29 of AES-GCM), 16,413 byte writes are the bulk records. Data is correct either way: the patched copy md5s identical to the source, ce647d9436b48401cd4b489c955ef0f7. Protocol confirmed two ways, for anyone reproducing: An https destination not advertising transfer-encoding: nbd yields [Chunked; NoProtocol] (impl.ml:1084), and the code takes List.hd. At runtime the receiver runs vhd-tool serve --source-format raw --source-protocol chunked --source-fd 8. Correction 2: my reasoning for "NODELAY is a no-op after pipelining" was wrong I said that with 8 requests in flight there is almost always an MSS queued, so the sub-MSS condition rarely arises. Not what happens. Arm D is pipelining with the socket patch removed, and it still stalls on 16.1% of samples against 1.1% with it. Depth 8 does not keep the queue above MSS. The conclusion survives, the reasoning does not: NODELAY after pipelining gives no throughput gain because the stall stops being the limiting factor, not because the stall goes away. Correction 3: the export path needs nothing Export runs stream_raw (impl.ml:335) : export_raw_vdi.ml:56 passes "none" as the destination protocol. No headers, no replies, so no write-write-read pattern. The patch could not cover it anyway: --destination-fd becomes fd://N, then File_descr, then Channels.of_raw_fd (impl.ml:991), never touching the socket helper. It does not need to. http_svr.ml:624 already sets TCP_NODELAY on the listening socket and Linux passes it to accepted sockets. Verified on the 4.19 kernel these hosts run, with a listener without the option as a control returning 0. stunnel sets it independently too (xapi_stunnel_server.ml:66-68). The cost of TCP_NODELAY, every test we ran path Nagle on TCP_NODELAY cost NBD 107 GiB, NVMe 1441.7 (0.996x MSS) 1425.2 (0.984x) +1.16% segments NBD pipelined, RAM 1442.0 (0.996x) 1425.7 (0.984x) +1.14% segments NBD 6-pair repeat, RAM 1441.9 (0.996x) 1424.6 (0.984x) +1.20% segments Chunked 100 GiB 1438.5-1439.9 (0.993x) 1428.7-1430.1 (0.988x) +0.70% segments [image: 1788093077310-865bccc1-8e7e-4948-b329-50ad84a56e25-image.jpeg] About 1.2% more segments, reproducible to three digits across four independent experiments. Small, and it is the real permanent cost of the option. Where the ceiling is now At 7.4x we are at about 525 MiB/s, and it is not the obvious things: Not the network: 10G link, roughly 46% utilised. Not the sender: sparse_dd at mean 57%, peak 64% of one core. On the receiving host, tapdisk is roughly 72% of all busy CPU (python3 9%, xapi 6%, stunnel 6%). That sample spans more than one leg and sums several tapdisk processes, so read it as dominance rather than a precise figure. It is where we would look next. Our reading Both changes are worth having, they are independent, and they are not additive, so the order is a real decision. TCP_NODELAY Strongest argument is precedent, not the benchmark: vhd-tool is the only NBD client in the toolstack that leaves Nagle on. QEMU sets it unconditionally (nbd/client-connection.c:143) and forwards it through TLS. blktap treats failing to set it as fatal (drivers/block-nbd.c:793). xapi and stunnel both set it. 12 lines, 4.26x on code that ships today. Costs, now measured rather than assumed: about 1.2% more segments everywhere, and 2.7% wall clock on vdi-copy between two host-local SRs where it gains nothing. Pipelining 7.41x on its own, and it makes the socket option irrelevant to throughput. Less work than it looks: the multiplexer already exists and is unused (nbd/lib/client.ml:78), so this is not the NBD redesign it first appears to be. The real obstacle is buffer ownership. expand_copy hands out slices of one shared 2 MiB buffer, so a naive pipeline silently corrupts data while reporting success. Our prototype works around it with a local buffer pool and a memcpy per block. The proper fix belongs inside expand_copy in f.ml, a shared library with other consumers. The prototype is not mergeable as it stands: progress counts issued rather than completed work, and a failed write leaves its siblings unawaited. We will follow whichever route the XAPI team prefers and are happy to do the work either way. Our own order would be the socket option first, then pipelining done properly rather than our prototype. That is a sequencing preference, not a claim that the gains compose. If you would rather go straight to pipelining and skip the socket change, our data supports that: it reaches 7.41x on its own. One caveat that cuts in favour of the socket option, and our rig cannot measure it: Every number here is from a 0.2 ms RTT link. The Nagle stall costs a delayed-ACK round trip, so its cost scales with RTT. 0.2 ms is therefore the regime least favourable to fixing it. On a higher-RTT link (cross-rack, cross-site, DR replication) the same 16.1% stall rate that costs nothing here should cost real throughput. So read 4.26x as a floor for TCP_NODELAY, and "pipelining makes it redundant" as a statement about short links specifically. We have not measured a long-RTT link and would like to hear from anyone who has. Open questions Is the 2.7% on vdi-copy acceptable, or should the option be scoped to NBD? Unconditional is simpler and matches every other component. Scoping avoids a measured regression on a path that cannot benefit. Is the Chunked header worth batching regardless? It is a separate unbuffered 12 byte write before every chunk, which under TLS becomes its own 41 byte record. Writing header and payload together removes that independently of any socket option, and helps the Nagle-on case too. Should expand_copy own a buffer pool? Any pipelining implementation needs it. Fixing it in f.ml fixes it for every consumer, but that is a wider blast radius than we wanted to take unilaterally. Is the receiving tapdisk the next real ceiling? At 7.4x the sender and the network both still have headroom and tapdisk dominates destination CPU. We have not dug into why. Does the picture change on a high-RTT or faster link? Both conclusions, that NODELAY is worth 4.26x and that pipelining makes it redundant, are properties of a short fast link that never saturated. Rig Two host XCP-ng 8.3 pool, xapi 26.1.16, 10G, 0.2 ms RTT. Arms differ by exactly one setsockopt where relevant, same tree and toolchain. Every run verified by binary sha256 before it started, transfers checksum verified. RAM SRs are ext4 on a brd ramdisk. tmpfs does not work: no O_DIRECT, so VDIs can be written onto such an SR but never migrated back off.

Nice, thanks for the feeder entry and the explanation, @CAPS!

@poddingue You're right. The RPU task shown in the list was a previous one. Even that one finished fine. So both RPU tasks didn't show at all today! My bad. The evacuation and remirroring tasks were all shown. Latest XOA 6.7.1

I measured this on a two-NIC 8.3 host. As far as I understand, there's no way to do it from XO: the call that edits a network doesn't take a bond mode, every bondMode in the codebase sits on a create path, and the REST API on a running XOA offers create_bonded_network plus get and delete on a network, with nothing that edits one. The CLI route looks cheap though. xe bond-set-mode on a live bond cost zero dropped packets in both directions I tried, pinging every 200ms across the change, and the command returned in about 1.4 seconds. Two caveats, and the second is about your case specifically: creating and destroying the bond did interrupt the host (roughly 8 and 16 seconds), and I only went active-backup to balance-slb and back, never touching lacp, because my switch port isn't configured as a LAG and I'd have dropped the box. So I can't tell you your lacp to active-backup move is free, only that mode changes in general didn't cost me anything. Worth putting on https://feedback.vates.tech either way so the votes have somewhere to gather.

v0.4.0 Release - https://github.com/acebmxer/install_xen_orchestra/releases 0.4.0 - 2026-08-23 Changed The cloud image is staged on the pool master by default instead of being streamed. Staging is the only path that can resume a broken download, retry a transient failure, and check the downloaded size before anything reaches the VDI; streaming can do none of those, because a pipe already feeding a fixed-Content-Length PUT cannot be rewound. On a link that drops the occasional TLS record — which any multi-gigabyte transfer eventually meets — streaming failed every attempt while staging rode it out. Streaming is now what it should always have been: the fallback for a host without the few gigabytes of scratch space staging needs. Fixed A stalled streaming import no longer has to be interrupted by hand. When the download end died, the upload sat waiting for a response XAPI would never send. --speed-time could not help — by that point curl is waiting, not transferring, so the speed meter has stopped ticking and only --max-time 3600 would eventually fire. The two transfers now run as separate processes with the download's exit status watched, and the upload is killed the moment it fails. A pool master short on scratch space no longer fails the whole deploy. The staged path signalled "no room" with a plain non-zero return, which under set -e took the script down before the fallback could be reached. A failed cloud-image download no longer looks like a successful import. The streaming import piped one curl into another, and the remote shell reported only the upload side's exit status. A download that died partway — a transient SSL_read ... bad record mac on a 3 GB transfer is the usual cause — therefore produced a truncated disk that the deploy reported as imported, and a VM that booted into a corrupt filesystem. The pipeline now runs under bash -o pipefail. A broken image download no longer hangs the deploy for an hour. When the download end died, the upload curl had already promised XAPI an exact Content-Length and sat waiting to send bytes that were never coming, with XAPI waiting alongside it until --max-time 3600 expired — the visible symptom being a XAPI task frozen at partial progress and a script that had to be interrupted. Both ends now abort after 60s below 1 KiB/s. The staged image download resumes instead of starting over. It now uses -C - with --retry 5 --retry-delay 3 --retry-all-errors, so the transient TLS failures that a multi-gigabyte single-connection download eventually hits are ridden out rather than failing the deploy. The streaming path deliberately does not retry: curl re-issues from byte 0, and piped into a fixed-Content-Length PUT those bytes would be appended to the ones already sent, corrupting the image while appearing to succeed. A short staged download is refused rather than imported. The file's size is now checked against the length the server advertised before anything is written into the VDI. The staged download reports progress. It was silent for several minutes on the longest step of the deploy, which reads as a hang worth killing. A failed deploy names the VM it left behind, with the xe vm-destroy command, instead of leaving a half-built VM to be rediscovered later in the pool's VM list. Nothing is destroyed automatically. --update/--reconfigure/--rebuild no longer abort on a root install. Reading User= from a systemd unit that has no such line (which is what a root install looks like) failed the pipeline under set -o pipefail and took the script down before the fallback could run. Deploy prompts validate values, not just their shape. 999.999.999.999 was accepted as an address and 70000 as a port; both were only rejected after the VM existed, by an unreachable guest or by the installer inside it. Prompted settings are no longer lost when the base config omits the key. The generated xo-config.cfg was patched with sed, which silently does nothing for a key that is not there — so the VM installed on the default while the summary showed the value you typed. Missing keys are now appended. An $EDITOR with arguments works. code --wait passed the availability check and then failed with "No such file", since the whole string was treated as one executable path. Values edited into the config are validated. An unusable port or branch was silently ignored, leaving the summary showing one thing and the VM installing another. Troubleshooting commands point at a key that still exists. The ssh -i lines printed on a failed install and on a non-200 health check named the temporary key, which the exit trap had already deleted. A second VM with the same hostname no longer overwrites the first one's SSH key, which was the only way into that machine. The disk-space check before staging an image on the pool master is derived from the image's actual size instead of a hard-coded 4 GiB, which rejected small images and let large ones fill /var/tmp mid-download. tests/probe-xapi-deploy.sh acquires its XAPI session from the pool master, the way --deploy does, so a firewall that blocks port 443 from your workstation no longer skips the HTTP transport probes that matter. It also validates --host, --user, --sr, --image and --payload-mb before they reach a shell, drops the eval in the workstation-side transport, and exits non-zero when any probe failed rather than whenever one transport worked. The menu example in the README and the layout comment above MENU_NAMES described the old fixed 5/4/centered grid; with ten items the menu draws five entries in each column. Security The cloud image is verified against its published checksum. The size check catches a download that was cut short; it cannot catch one that arrived complete from the wrong place, because a substituted image has a perfectly consistent Content-Length. The staged image is now checked against the SHA512SUMS its origin publishes beside it — which is what Debian ships — and a mismatch aborts before anything reaches the disk. An origin that publishes no sums warns and continues, so a custom XO_DEPLOY_IMAGE_URL keeps working. Set XO_DEPLOY_IMAGE_SHA512 to require a specific digest instead: that makes the check mandatory, aborting rather than continuing unverified, and refuses the streaming import outright because a pipe fed straight into the VDI leaves no file to hash. Fetching sums over the same connection as the image is not a detached signature — it defends against a bad mirror or a stale cache, not an attacker holding the TLS session for both requests. A pinned pool-master fingerprint is now enforced instead of advised. deploy_verify_host_key fingerprinted the host key and then returned success on every path that could not complete the check — so with XO_DEPLOY_POOL_FINGERPRINT set, a ssh-keyscan that timed out meant the host password was sent to whatever answered on that address, which is exactly what pinning exists to prevent and the easiest outcome for an on-path attacker to arrange. A pin that cannot be checked is now a hard failure. The verified host key is bound to the connection that carries the password. The scanned key was fingerprinted, shown, and then discarded, while dom0_exec connected with StrictHostKeyChecking=accept-new against the default known_hosts — verifying one transaction and trusting another. Nothing stopped a different key, or the host's RSA key when the ED25519 one had been displayed, being accepted at connect time. The whole scan is now pinned into a run-scoped known_hosts that dom0_exec enforces with StrictHostKeyChecking=yes, the same way deploy_wait_for_guest already treated the guest. A hostile pool master can no longer run commands on the workstation. The free-space probe in deploy_import_vdi_staged fed the host's reply straight into (( )), which expands an array subscript before evaluating it — so an answer of PATH[$(...)] executed locally rather than being rejected. It is now checked against ^[0-9]+$ first, matching the guards already applied to size and got in the same function. Note that set -euo pipefail does not cover this: set -u blocks only the unbound-variable form of the payload. This mattered more after staging became the default import path, because the probe went from rarely reached to running on every deploy. The pool master's root password is no longer visible in ps. Three calls predating dom0_exec still used sshpass -p "$HOST_PASSWORD", putting the password in the process list where any other user on the workstation could read it. They now use sshpass -e with $SSHPASS, as dom0_exec does. The admin password hash is kept out of XO_DEBUG=1 output. deploy_harden_guest_sudo and deploy_build_config_drive were missing the local - / set +x guard the rest of the script uses, so the hash was printed by xtrace. Previously masked on automated runs only because --non-interactive left the hash empty; requiring a password made it reachable on every deploy. Revoking the deployment key can no longer empty authorized_keys. A grep failure — no space for the temporary file, an unreadable source — was swallowed by || true and the empty result written back, taking the operator's own key with it. grep's "nothing matched" (a legitimate empty result) is now distinguished from a real error, which aborts and leaves the file untouched. The streaming import's FIFO is created inside a private directory. mktemp -u returns a name without creating anything, leaving a window in dom0's world-writable /tmp. The FIFO now lives in a mktemp -d directory. DSA public keys are rejected. ssh-dss was accepted by deploy_load_pubkey, but OpenSSH has refused DSA since 7.0 and removed it in 9.8, so it only installed a key that silently never worked. The cloud-init cache scrub covers cloud-config.txt. The rendered config holds hashed_passwd just as the raw user-data does; only the latter was being redacted. The deployment SSH key is destroyed at the end of a deploy. It used to be...

I'm late to this, but I've been building lately a JetBrains plugin against the REST API and ACL v2 turned out to decide its whole design, so here's some feedback. Everything below is just one appliance, one pool (my small homelab), on a plan 4 trial, with an admin control call taken in the same breath as every scoped one. What I experienced is that selectors narrow reads (tags: and id: both, 1 VM against an admin control of 11), they're re-evaluated per request rather than fixed when the privilege is created, deny composes the way your Carol example says (allow-all plus deny on a tag gave 9, which is 11 minus the 2 tagged), and they scope the power verbs too, not just reads. The event stream is, to me, scoped as well, which was the thing I most wanted to check, because a scoped read next to an unscoped feed would have been a nasty trap. It's not. With two streams open at once, a change to an out-of-scope VM produced an update on the admin stream and nothing at all on the scoped one, so it's genuinely filtered rather than merely quiet. And the bit I'd underline: the same single change is update to the admin and add or remove to the scoped user. The verb is computed per subscriber, not per object. Across that run the admin only ever saw update, and I originally wrote that an admin never sees add or remove at all. That was wrong and I've since measured it: an admin does get add and remove when an object is genuinely created or deleted. So the rule is that a scope change reads as update and an existence change as add or remove, which supports your design better than what I first wrote did. "From the user's perspective, not XOA's" turns out to be literal rather than a figure of speech, and that's a nicer piece of design than the sentence let me imagine. Now the three things that I didn't see in the post above. vm-snapshot is a separate privilege resource and vm doesn't imply it. Maybe that's obvious, but it was not obvious to me. With all six VM privileges granted, GET /vm-snapshots came back empty while admin saw 6, including a snapshot the scoped user had just taken. Anything with a restore or revert screen gets an empty list and no error. Inherited snapshot tags look like a snapshot-time copy rather than a link. Tag a VM and it's in scope immediately, but its existing snapshots keep tags: [] and stay invisible. So someone onboarded into a tag scope after their snapshots exist sees the VM and not its history. Subscribing delivers no initial dump. Both streams sat on init and keepalives until something changed, so it's a delta feed and a client has to fetch the collection over REST and maintain it from events. Worth a line, since the natural assumption (at least to me, don't make that the rule for everyone) is the other one. While I'm here: two smaller ones. Privilege action names aren't REST action names: shutdown:clean grants clean_shutdown, revert-snapshot grants revert_snapshot, and a mistyped action quietly gives you a privilege that grants nothing. And the event: init frame's field is id, not connectionId (why did I think it was connectionId, no idea, I thought it was "natural"), which cost me a while of thinking the stream was dead when I was posting to /events//subscriptions. Yes, I know, I should have read the documentation instead of experimenting in the dark, sending made-up field names in the wild. One last thing: selector is optional, so a privilege created without one reads back as {id, resource, action, effect, roleId} with no hint the field exists. I granted allow read on vm, saw all 11 VMs, read the object back, and (falsely) concluded the REST API had no object dimension at all. It's all in the previous post and it's in the swagger, and of course, in the official documentation. Once again, I'm an innocent victim because I didn't RTFM. I just never saw a privilege that had one. If a privilege echoed selector: null, or if the first example anyone met were a scoped one, I don't think I would have spent much time on that. Once again, my bad, didn't RTFM. This was a small, targeted test, at best. I didn't have the intent to test what was brought up in this very thread, I just happened to tinkle with the REST API and ACL V2 for my PoC, so lots of things got untested. Please, don't take it from me as settled: only tags: and id: selector forms, nothing on a second pool or a real multi-user deployment, and I have not checked what happens to a live subscription when the privilege itself changes rather than the VM's tags. If you read me until there, you're brave, or have too much time on your hands.

Hey, that sucks after catching up on 23 commits. Looks like something broke in the log module during the update. I’d try a clean reinstall of the dependencies first, or drop back to an older Node version for a bit since Node 24 can be fussy with these packages. When you need a quick mental break from troubleshooting npm issues, checking out TonyBet is a great way to enjoy some online gaming and sports betting. Hope getting those modules reinstalled gets your build running smoothly again soon!

Hi bikemuch, For unregistered XOA the template auto-update is limited. Easiest way is via CLI: download the latest Debian 13 and CentOS Stream 10 ISOs, then use xe vm-import or create new templates from them. You can also check the XOA “Templates” section and force a refresh if available. Works fine for me this way.

If the list is only complete after you've been through the Storage view first, that smells like the VM view not loading the whole SR collection, so the dialog only knows about the SR the disk already sits on. I could easily be wrong about the mechanism though. 6.7.0 went out on 30 July, so it would help to know whether it still does this there, and whether anyone else on 6.6.2 sees the same thing. Might be worth a mention to @Team-XO-Frontend, since they'd know straight away whether that dropdown is meant to come from a shared collection or get fetched per view. The nearest thing I found in the tracker is https://github.com/vatesfr/xen-orchestra/issues/7392, but that one is ISO SRs turning up in the wrong list rather than SRs going missing, so probably not the same thing.

@neal https://github.com/vatesfr/xen-orchestra/issues/10260

@poddingue Fistst of all I appreciate your answer and your position. The thing is that, even though the proxy code itself is opensource, the functionality of the plugin is basicaly behind a paywall. We are not talking about support. Actual functioning of the plugin after compiling from sources depends on license availability and there is no option to select no support or something along the lines "I built it myself from sources". Without patching the code even though the proxy is otherwise functional the backups won't work because of missing license. Hopefully the powers that can will provide an acceptable albeit community supported way to use the proxy cleanly, without touching license checks. Best regards!

@acebmxer No problem, that's excellent news, thanks for the feedback!

The July updates batch that went out on 28 July carries xapi-26.1.11-1.3.xcpng8.3, and @kagbasi-wgsdac has since confirmed on https://github.com/xcp-ng/xcp/issues/825 that three days after patching his disks are all visible again and reverting a snapshot no longer duplicates VDIs. The blog entry for that batch names the fix as non-snapshotted VBDs staying attached after VM.revert, a regression from xapi-26.1.4-3.3: https://xcp-ng.org/blog/2026/07/28/july-2026-updates-1-for-xcp-ng-8-3-lts/. Host reboots are needed. That stops new damage, but it does not unstamp VDIs that were already hit, so if disks are still hidden after you patch you probably still want the repair script at https://xcp-ng.org/forum/post/105564. My post above also had the mechanism wrong, and @kagbasi-wgsdac corrected it: the field being wrongly written is snapshot-of on base disks, not is-a-snapshot. If anyone is still watching disks disappear on a fully updated pool, please say so here, because that would be something new rather than the tail of this one.

RESOLVED — root cause found, three years later. Leaving a full write-up for anyone who lands here from a search. Short version: this was never an XO bug, and it was never intermittent. The answer was sitting in the very first test-cli.js output I posted back in May 2023, and I misread it — as did everyone else in this thread, myself very much included. The line that mattered failed to bind as CN=Agbasi\, Kismet,...: 80090308: LdapErr: DSID-0C090434, comment: AcceptSecurityContext error, data 569, v4f7c We all pattern-matched AcceptSecurityContext error to "bad credentials" and moved on. But the meaning is entirely carried by the data field, which is the underlying Win32 status in hex: data 52e = 0x52E = 1326 = ERROR_LOGON_FAILURE — this is the "wrong password" one data 525 = 1317 = ERROR_NO_SUCH_USER data 532 = 1330 = password expired data 775 = 1909 = account locked out data 569 = 0x569 = 1385 = ERROR_LOGON_TYPE_NOT_GRANTED I was getting 569, not 52e. My password was correct all along. AD validated it, then refused the logon type. Why that happens xo-server-auth-ldap verifies a password the only way LDAP allows — it re-binds to the directory as the user. Against Active Directory, an LDAP simple bind to a DC is processed as a Type 3 (network) logon on that DC. So if an account is caught by "Deny access to this computer from the network" (SeDenyNetworkLogonRight) in the Default Domain Controllers Policy (or any other WINNING GPO, for that matter), it cannot complete an LDAP bind — no matter how correct the password is, and no matter which LDAP client is asking. My environment uses a tiered admin model. Non-domain-admin admin groups are explicitly denied network logon to the DCs. My admin account is in those groups. Hence 569, every single time, by design. Why it looked intermittent It wasn't. I sampled it either side of a config change. I could prove a bind had succeeded recently, because my LDAP-only XO account (no local password on the record at all) minted an API token on 28 July. Then on 31 July I restored RBAC settings on the Default Domain Controllers Policy that had drifted at some point — I found that during unrelated PKI work. GptTmpl.inf last-write confirms it. The token's last successful use is about eleven hours before that edit. Two deterministic states, one config change in the middle. That's the whole "intermittency." My 2023 "seven security groups" theory was wrong For the record, since it's still up there and someone will find it: I removed group memberships one at a time until auth worked, and concluded there was a membership count limit. There isn't. My own control test disproved it at the time — adding fifteen groups never reproduced the failure — and I should have taken that seriously instead of filing it under "weird." The variable was never the count. It was which group. One of the removals happened to drop the account out of a denied group. My other closing theory in this thread — special-character handling in the username or password — was also wrong. Getting 569 back proves AD parsed the escaped DN (CN=Agbasi\, Kismet), found the object, and got as far as evaluating the password. A mangled DN gives you 525 or a DN syntax error, not a logon-rights rejection. ldapts and passport were behaving correctly throughout. How to check this in 60 seconds Run the plugin test CLI and note the data value. Convert hex → decimal, look it up in Microsoft's System Error Codes list. On the DC, look for Security event 4625 with Sub Status 0xC000015B (STATUS_LOGON_TYPE_NOT_GRANTED). Fastest test of all — from a workstation, as the affected account: net use \\dc01\sysvol. If network logon to the DC is denied, this fails too, and you've confirmed it without touching XO at all. Check the policy directly: $p = "\\mydomain.net\SYSVOL\mydomain.net\Policies\{6AC1786C-016F-11D2-945F-00C04fB984F9}" + "\Machine\Microsoft\Windows NT\SecEdit\GptTmpl.inf" Select-String -Path $p -Pattern "SeDenyNetworkLogonRight|SeNetworkLogonRight" Resolve the SIDs and see whether your user is in any of the denied groups. Also worth checking your grant side: if Access this computer from the network doesn't list Authenticated Users directly, ordinary users are probably getting it transitively via Pre-Windows 2000 Compatible Access. Worth confirming before you assume a plain non-privileged account will work. What I am NOT doing Removing those groups from the deny right. It's doing exactly what I rebuilt it to do. Restoring an app login by handing admin groups network access to the DCs for SMB/RPC/LDAP is a bad trade, and I'd just be undoing my own remediation. Fix Interim: local XO accounts for the admins who need them. No AD objects created, nothing to unwind later, per-user attribution preserved in the audit log. Long term: federate XO through Keycloak (OIDC) instead of LDAP. Kerberos ticket issuance is a KDC service operation and is not gated by SeNetworkLogonRight — which is exactly why these accounts log into workstations all day while failing an LDAP bind. ️ Important if you go the Keycloak route: Keycloak's LDAP user federation validates passwords by doing an LDAP bind. Configure it that way and you'll hit data 569 inside Keycloak instead of inside XO and gain nothing. Password validation has to be delegated to Kerberos/GSSAPI. This will bite you on anything else you point at LDAP too — Bitwarden, NPM, TrueNAS, the lot. Worth solving once at the IdP. One request for Vates @olivierlambert @julien-f — you were right that it was environmental, and I owe you both thanks for the time you put in back in 2023. That said, there's a real (small) improvement available here. xo-server collapses every auth provider exception into a generic invalid credentials, and the plugin only emits the actual AD error at DEBUG. The DC told us precisely what was wrong on the very first attempt — it just never reached anywhere a user would look. Surfacing the LDAP result code and the AD data sub-code at INFO on failure, and in the plugin test output in the UI, would turn this class of problem from a multi-year hunt into a single-session diagnosis. Happy to open an issue on GitHub with the full reproduction if that's useful. Hope this saves someone else three years. If you found this thread by searching data 569, *ERROR_LOGON_TYPE_NOT_GRANTED**, or "LDAP invalid credentials but password is correct" — check your Deny access to this computer from the network user right first. That's almost certainly it.

@john.c said: nd need to be cleared, before re-attempting the update. Checking the logs will help Just for completeness... it kept doing this after all the typical diagnosis steps. So never found a solution Definitely wasn't cache or disc space related. In the end I built the XOA from source on the same VM and that worked fine. Although subsequent to that I've moved XOA to the containerised version thus saving an extra VM!.

@irtaza9 i would look a the usual suspect, /var/log/SMlog /var/log/xensource.log see https://docs.xcp-ng.org/troubleshooting/

@acebmxer Thank you for your quick feedback.

@acebmxer all good and thank you again for sharing your discovery on the forums

Perfect, thanks for your feedback!