XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    • Profile
    • Following 0
    • Followers 0
    • Topics 14
    • Posts 69
    • Groups 0
    K Offline
    1. Home
    2. kagbasi-wgsdac
    3. Best

    Posts

    Recent Best Controversial
    • RE: Existing AD Users Cannot Login to XOCE but New Users Can

      RESOLVED — root cause found, three years later. Leaving a full write-up for anyone who lands here from a search.

      Short version: this was never an XO bug, and it was never intermittent. The answer was sitting in the very first test-cli.js output I posted back in May 2023, and I misread it — as did everyone else in this thread, myself very much included.

      The line that mattered

      failed to bind as CN=Agbasi\, Kismet,...:
      80090308: LdapErr: DSID-0C090434, comment: AcceptSecurityContext error, data 569, v4f7c
      

      We all pattern-matched AcceptSecurityContext error to "bad credentials" and moved on. But the meaning is entirely carried by the data field, which is the underlying Win32 status in hex:

      • data 52e = 0x52E = 1326 = ERROR_LOGON_FAILURE — this is the "wrong password" one
      • data 525 = 1317 = ERROR_NO_SUCH_USER
      • data 532 = 1330 = password expired
      • data 775 = 1909 = account locked out
      • data 569 = 0x569 = 1385 = ERROR_LOGON_TYPE_NOT_GRANTED

      I was getting 569, not 52e. My password was correct all along. AD validated it, then refused the logon type.

      Why that happens

      xo-server-auth-ldap verifies a password the only way LDAP allows — it re-binds to the directory as the user. Against Active Directory, an LDAP simple bind to a DC is processed as a Type 3 (network) logon on that DC.

      So if an account is caught by "Deny access to this computer from the network" (SeDenyNetworkLogonRight) in the Default Domain Controllers Policy (or any other WINNING GPO, for that matter), it cannot complete an LDAP bind — no matter how correct the password is, and no matter which LDAP client is asking.

      My environment uses a tiered admin model. Non-domain-admin admin groups are explicitly denied network logon to the DCs. My admin account is in those groups. Hence 569, every single time, by design.

      Why it looked intermittent

      It wasn't. I sampled it either side of a config change.

      I could prove a bind had succeeded recently, because my LDAP-only XO account (no local password on the record at all) minted an API token on 28 July. Then on 31 July I restored RBAC settings on the Default Domain Controllers Policy that had drifted at some
      point — I found that during unrelated PKI work. GptTmpl.inf last-write confirms it. The token's last successful use is about eleven hours before that edit.

      Two deterministic states, one config change in the middle. That's the whole "intermittency."

      My 2023 "seven security groups" theory was wrong

      For the record, since it's still up there and someone will find it: I removed group memberships one at a time until auth worked, and concluded there was a membership count limit. There isn't. My own control test disproved it at the time — adding fifteen groups
      never reproduced the failure — and I should have taken that seriously instead of filing it under "weird." The variable was never the count. It was which group. One of the removals happened to drop the account out of a denied group.

      My other closing theory in this thread — special-character handling in the username or password — was also wrong. Getting 569 back proves AD parsed the escaped DN (CN=Agbasi\, Kismet), found the object, and got as far as evaluating the password. A mangled DN gives you 525 or a DN syntax error, not a logon-rights rejection. ldapts and passport were behaving correctly throughout.

      How to check this in 60 seconds

      1. Run the plugin test CLI and note the data value. Convert hex → decimal, look it up in Microsoft's System Error Codes list.

      2. On the DC, look for Security event 4625 with Sub Status 0xC000015B (STATUS_LOGON_TYPE_NOT_GRANTED).

      3. Fastest test of all — from a workstation, as the affected account: net use \\dc01\sysvol. If network logon to the DC is denied, this fails too, and you've confirmed it without touching XO at all.

      4. Check the policy directly:

        $p = "\\mydomain.net\SYSVOL\mydomain.net\Policies\{6AC1786C-016F-11D2-945F-00C04fB984F9}" + "\Machine\Microsoft\Windows NT\SecEdit\GptTmpl.inf"
        Select-String -Path $p -Pattern "SeDenyNetworkLogonRight|SeNetworkLogonRight"
        
      5. Resolve the SIDs and see whether your user is in any of the denied groups.

      Also worth checking your grant side: if Access this computer from the network doesn't list Authenticated Users directly, ordinary users are probably getting it transitively via Pre-Windows 2000 Compatible Access. Worth confirming before you assume a plain
      non-privileged account will work.

      What I am NOT doing

      Removing those groups from the deny right. It's doing exactly what I rebuilt it to do. Restoring an app login by handing admin groups network access to the DCs for SMB/RPC/LDAP is a bad trade, and I'd just be undoing my own remediation.

      Fix

      • Interim: local XO accounts for the admins who need them. No AD objects created, nothing to unwind later, per-user attribution preserved in the audit log.
      • Long term: federate XO through Keycloak (OIDC) instead of LDAP. Kerberos ticket issuance is a KDC service operation and is not gated by SeNetworkLogonRight — which is exactly why these accounts log into workstations all day while failing an LDAP bind.

      ⚠️ Important if you go the Keycloak route: Keycloak's LDAP user federation validates passwords by doing an LDAP bind. Configure it that way and you'll hit data 569 inside Keycloak instead of inside XO and gain nothing. Password validation has to be delegated to
      Kerberos/GSSAPI.

      This will bite you on anything else you point at LDAP too — Bitwarden, NPM, TrueNAS, the lot. Worth solving once at the IdP.

      One request for Vates

      @olivierlambert @julien-f — you were right that it was environmental, and I owe you both thanks for the time you put in back in 2023.

      That said, there's a real (small) improvement available here. xo-server collapses every auth provider exception into a generic invalid credentials, and the plugin only emits the actual AD error at DEBUG. The DC told us precisely what was wrong on the very first
      attempt — it just never reached anywhere a user would look.

      Surfacing the LDAP result code and the AD data sub-code at INFO on failure, and in the plugin test output in the UI, would turn this class of problem from a multi-year hunt into a single-session diagnosis. Happy to open an issue on GitHub with the full reproduction if that's useful.

      Hope this saves someone else three years. If you found this thread by searching data 569, *ERROR_LOGON_TYPE_NOT_GRANTED**, or "LDAP invalid credentials but password is correct" — check your Deny access to this computer from the network user right first. That's almost certainly it.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue Bug report filed as requested — https://github.com/xcp-ng/xcp/issues/825 — and tagging @Team-Storage per your suggestion.

      Full evidence bundle is attached to the issue (versions, sweep output, vhd-util vs xe comparison, SMlog). Summary of what I found:

      One correction to the mechanism, and I think it matters. The recap describes is-a-snapshot being flipped to true. On my system that isn't what's happening — is-a-snapshot is false on every affected VDI. The field being wrongly written is snapshot-of, which is getting populated on base disks that aren't snapshots at all. XO's disappearing-disks symptom is consistent with either (it filters on a non-empty snapshot-of), but if the storage team is hunting for a bad is-a-snapshot write, that may be the wrong field. Every affected VDI here looks like:

      is-a-snapshot: false      <-- correct
      snapshot-of:   <populated with an unrelated VDI's UUID>   <-- wrong
      

      A VDI that is a snapshot of itself. The clearest single artifact:

      uuid:          806f7f42-083f-4a40-b3f1-0700d00bab5a
      name-label:    WinSrv2022SHB_Disk1_Data
      is-a-snapshot: false
      snapshot-of:   806f7f42-083f-4a40-b3f1-0700d00bab5a   <-- itself
      snapshot-time: 20260709T11:19:15Z
      sm-config:     vhd-parent: c86e3247-...   <-- bears no relation to the snapshot-of value
      

      No valid code path produces snapshot-of = self. Whatever writes this field isn't validating the target.

      It's still actively corrupting new VDIs — this is not just legacy damage. That self-referential VDI was created 2026-07-09, a week after my patch + reboot. Sweeps 9 days apart went from ~180 → 191 affected VDIs on one SR, and a fourth anchor UUID appeared that didn't exist in the first sweep. Newly created VHDs keep landing in the affected set. So "stop it happening again" is the urgent half of the two-part fix, at least in my case.

      The bogus targets cluster onto a tiny anchor set, and the anchors point at each other:

      Count Anchor
      97 937c3945 (→ a893fdb4)
      50 a893fdb4 (→ ea150883)
      37 ea150883
      7 806f7f42 (→ itself, new since Jul 9)

      That looks less like corrupted lineage and more like the field being filled from an incorrect/uninitialised source.

      On-disk VHDs are completely healthy. vhd-util check says valid, parent locators are consistent, GC reports no work. The two VDIs the DB calls parent/child are, on disk, siblings under a common parent. The corruption is purely in the XAPI database — which is good news for recoverability.

      The VDI_IN_USE is not a real lock. current-operations is empty, xe task-list is empty, no tapdisk holds it. VM.start fails because it's walking a snapshot relationship that doesn't exist on disk. Reproduces from xe on the pool master with XO entirely out of the path — which is why I filed against xcp-ng/xcp rather than the XO tracker.

      Versions: XCP-ng 8.3.0, xapi 26.1.11 (xapi-core-26.1.11-1.2), sm-3.2.12-17.9, sm-fairlock-3.2.12-17.9, blktap 3.55.5-9.1, build 20260618.

      I have not attempted to bulk-clear the fields — on-disk data is intact and I'd rather not do a mass write against the XAPI DB on a live SR without guidance. Backing store snapshotted as a safety net.

      Happy to run whatever diagnostics would help. And +1 to the hand-grenade feeling — the affected set growing on its own is the part that worries me.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XCP-ng v8.3 Host Crashing Upon Console Login and Performing Any Action

      Good evening all,

      Just a quick update. It's been a couple of days now after the rebuild and everything seems to be humming along fine, so I believe this topic can be marked as resolved. I can confidently conclude that this wasn't an XCP-ng issue, although the error message seems a bit misleading.

      posted in XCP-ng
      K
      kagbasi-wgsdac
    • RE: Need Help Understanding the VM Suspend Process

      @olivierlambert Yes sir, it is and I'm glad I confirmed this for myself. Thanks also for helping me understand how the VM Suspend process works. Hopefully this post helps other newbies with the same understanding in the future.

      posted in Management
      K
      kagbasi-wgsdac
    • RE: Cannot Download Exported VM After Export Task Completes Successfully

      @olivierlambert I didn't time it. I'm leaving home to drop the kids off at school, then off to work afterwards. I'll run it again and time it and report back.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: ISO Import to Local Storage via XO Not Working

      @olivierlambert Oh nice, hope it's not too cold over there. It's 9AM over here in Maryland and a bit nippy but not too cold.

      Nope, no host or pool logs available (both are empty). Then again, I cleared all the alerts from the dashboard earlier this morning, so not sure if that's what wiped all the logs. I'm not too worried, as this is a testing environment so I'm using this host to do a lot of learning.

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue You're most welcome, sir. 👍

      posted in Xen Orchestra
      K
      kagbasi-wgsdac
    • RE: XenOrchestra not showing VM Disks on Pool (on single Server working) - XCP-ng Center is showing them

      @poddingue Thanks for the response. I got side tracked with another issue and didn't get that bug report filed, I'll do that as soon as possible.

      It feels like I'm playing with a hand grenade...lol, quite scary!

      posted in Xen Orchestra
      K
      kagbasi-wgsdac