@kagbasi-ngc I figured out what the problem was, so wanted to post it for the benefit of the community.

The root cause for me ended up being a firewall issue on the Windows Server where I was running the NFS server role. While I hadn't made any changes to the firewall, the profile of the network interface connected to the management network had changed from Domain to Public/Private, thus, Windows Defender Firewall applied the most restrictive settings since it saw that the NIC wasn't connected to the Domain profile (even though it physically was).

This post on Serverfault (https://serverfault.com/a/647201/189248) clued me in and I looked at the Network Location Awareness service (NLA). It was set automatic and was running, so I restarted it and this reverted the profile of the NIC back to Domain and caused Windows Defender Firewall to apply the previously configured rules.

It is important to note that this situation won't be common, as most of you won't be running your NAS on a dual-homed domain controller that is also running the DHCP server role. Hope this helps someone.

@Prilly Looks like @lawrencesystems has a good explainer video on YouTube at https://youtu.be/weVoKm8kDb4?si=QH93rOKaXLglrIbS

Check it out when you get a chance, thanks.

@stormi Sorry for the delayed reply. I did as you requested, and below are the resulting log files. I started the VM at timestamp Dec 17 14:43:34 and stopped it about 2 minutes later.

• daemon.log
• xensource.log

If I can be of assistance, don't hesitate to ask please. Thank you.

@olivierlambert No, it's the other way around. Once the subject user is a memberof 3 or more groups, the problem occurs.

@manilx Yeah, not doing it in production, just in a test environment. Plus I only did it because I actually found a write-up by the Vates team on how to get it done, which confirmed to me that while they don't recommend it for production, they understand that it has some merit in a testing environment and want to see it working.

That said, I definitely hear your admonition and would never do it in a production setting.

@olivierlambert Certainly strange, then, that the same settings and same plugin works in XOA but not in XOCE. We could argue perhaps some configuration variances within the Domain Controllers, but given that I'm the same person that configured both environments - I'm highly confident that I stood up the DCs in the same way. Only things that would obviously be different would be the Domain Name and OU Structure.

I'll confirm the Forest and Domain functional levels later this evening and perhaps run an XOA trial and see if it works. I'll report back.

@olivierlambert So XOA is running on local storage of host #1 (VMH01) and all the VMs run off an NFS datastore which I've checked and is not in read-only mode or full (has about 60% free space).

What I haven't checked is whether the partition being used as the Local SR on VMH01 is perhaps the culprit. I ran xe check yesterday and the only three things it complained about were NTP, Updates, and not having Internet connectivity.

I should be in the office in about an hour, so I'll inspect and report back.

@stormi Yes it does. I just deleted the VTPM and tried starting it, but no success. Still the same issue - can't initialize the display.

I also disabled Secure Boot, and got the same results.

@stormi No worries at all, I understand.

@stormi Sorry for the delayed reply. I did as you requested, and below are the resulting log files. I started the VM at timestamp Dec 17 14:43:34 and stopped it about 2 minutes later.

• daemon.log
• xensource.log

If I can be of assistance, don't hesitate to ask please. Thank you.

@pdonias As requested, I added the log filter to the XO config file, restarted the xo-server service, and then ran the LDAP tester from the UI twice. Afterwards, I then ran journalctl -u xo-server, and this is all I see in the logs:

ON XOA:

Dec 16 17:15:59 xoa xo-server[418777]: 2024-12-16T22:15:59.962Z xo:api WARN user | plugin.test(...) [165ms] =!> Error: could not authenticate user
Dec 16 17:17:46 xoa xo-server[418777]: 2024-12-16T22:17:46.904Z xo:api WARN user | plugin.test(...) [13ms] =!> Error: could not authenticate user

ON XOCE:

Dec 16 17:22:34 XO1 xo-server[85750]: 2024-12-16T22:22:34.618Z xo:api WARN user | plugin.test(...) [62ms] =!> Error: could not authenticate user
Dec 16 17:23:04 XO1 xo-server[85750]: 2024-12-16T22:23:04.792Z xo:api WARN user | plugin.test(...) [11ms] =!> Error: could not authenticate user

I am still able to login, if I reduce the security groups I'm member of to 2 or less. 3 or more, results in failure.

@pdonias Will do as requested and report back. I have a meeting to get to now, but should be able to do this afterwards.

@olivierlambert As promised, I checked my user account at work and it is a member of 5 security groups, and the primary group is not Domain Users.

@olivierlambert When I get to work tomorrow, I'll check my AD account in my lab to see how many groups I'm a member of. If you'll recall, I didn't have any issue at all getting the plugin to work on the first try within the air-gapped test lab.

@olivierlambert Yes sir.

Would you like me to create a video of it happening? I'm home all day today, so I can run any test you need me to. Just let me know.

@olivierlambert No, it's the other way around. Once the subject user is a memberof 3 or more groups, the problem occurs.

@Danp Good catch!

I fixed it and I was now getting the previous error I was getting:

plugin.test
{
  "id": "auth-ldap",
  "data": {
    "username": "yykagbasi",
    "password": "* obfuscated *"
  }
}
{
  "message": "could not authenticate user",
  "name": "Error",
  "stack": "Error: could not authenticate user
    at /opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server-auth-ldap/src/index.js:246:15
    at default.testPlugin (file:///opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server/src/xo-mixins/plugins.mjs:285:5)
    at Xo.test (file:///opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server/src/api/plugin.mjs:109:3)
    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202412100608/@vates/task/index.js:172:22)
    at Task.run (/opt/xo/xo-builds/xen-orchestra-202412100608/@vates/task/index.js:156:20)
    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server/src/xo-mixins/api.mjs:469:18)"
}

So that got me thinking that perhaps there was a special character in my password that was causing the query to fail. It was interest to me that the LDAP bind was successful, but the query was returning no results, even though the response packet (as seen in Wireshark) contain results that I thought were valid.

So I performed several tests, with various variations of my username and password combinations. I even created a new user and test - the results were mixed. Sometimes succeeding and sometimes failing. However, I noticed that all the successful tests were with the new test account I'd created, not my personal account. So I compared the two, and realized that my account was part of 9 Security Groups whereas the test account was a member of only 2 Security Groups. So to confirm, I removed myself from all but 2 groups and tested, and it was successful. To confirm, I added myself to a 3rd group and tested - FAILURE.

So, at this juncture, it seems as though when a user is a member of more than 2 groups in AD, the LDAP query is failing (or at least the plugin test if failing - haven't attempted to login to XO to confirm).

Has anybody seen this behavior in their environments? By the way, I noticed this behavior in both XOA and XOCE.

UPDATE:

• I confirmed that as long as I kept my AD Group Membership to less than 3, I was able to login using my domain credentials. The moment I added a 3rd group, login failed.

• Noticed also that if my primary group is anything other than Domain Users, login fails (even if my group count is under 3).

@olivierlambert Thanks for extending the trial, much appreciated. I was able to activate the auth-ldap plugin (v0.10.10) with the same settings I used in my XOCE instance and it failed the test, even though the Wireshark capture seems to suggest the bind was successful.

ERROR MESSAGE IN XOA:

plugin.test
{
  "id": "auth-ldap",
  "data": {
    "username": "yykagbasi",
    "password": "* obfuscated *"
  }
}
{
  "message": "Illegal unescaped character: ( in value: ({name}",
  "name": "Error",
  "stack": "Error: Illegal unescaped character: ( in value: ({name}
    at Function._unescapeHexValues (/opt/xo/xo-builds/xen-orchestra-202412100608/node_modules/ldapts/FilterParser.ts:334:17)
    at Function._parseExpressionFilterFromString (/opt/xo/xo-builds/xen-orchestra-202412100608/node_modules/ldapts/FilterParser.ts:247:29)
    at Function._parseString (/opt/xo/xo-builds/xen-orchestra-202412100608/node_modules/ldapts/FilterParser.ts:198:31)
    at Function._parseString (/opt/xo/xo-builds/xen-orchestra-202412100608/node_modules/ldapts/FilterParser.ts:154:44)
    at Function.parseString (/opt/xo/xo-builds/xen-orchestra-202412100608/node_modules/ldapts/FilterParser.ts:31:38)
    at Client.search (/opt/xo/xo-builds/xen-orchestra-202412100608/node_modules/ldapts/Client.ts:584:31)
    at AuthLdap._authenticate (/opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server-auth-ldap/src/index.js:277:55)
    at default.testPlugin (file:///opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server/src/xo-mixins/plugins.mjs:285:5)
    at Xo.test (file:///opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server/src/api/plugin.mjs:109:3)
    at Task.runInside (/opt/xo/xo-builds/xen-orchestra-202412100608/@vates/task/index.js:172:22)
    at Task.run (/opt/xo/xo-builds/xen-orchestra-202412100608/@vates/task/index.js:156:20)
    at Api.#callApiMethod (file:///opt/xo/xo-builds/xen-orchestra-202412100608/packages/xo-server/src/xo-mixins/api.mjs:469:18)"
}

WIRESHARK CAPTURE: