No problem, this is always good to have feedback and cases showing an error and the explanation

RESOLVED — root cause found, three years later. Leaving a full write-up for anyone who lands here from a search. Short version: this was never an XO bug, and it was never intermittent. The answer was sitting in the very first test-cli.js output I posted back in May 2023, and I misread it — as did everyone else in this thread, myself very much included. The line that mattered failed to bind as CN=Agbasi\, Kismet,...: 80090308: LdapErr: DSID-0C090434, comment: AcceptSecurityContext error, data 569, v4f7c We all pattern-matched AcceptSecurityContext error to "bad credentials" and moved on. But the meaning is entirely carried by the data field, which is the underlying Win32 status in hex: data 52e = 0x52E = 1326 = ERROR_LOGON_FAILURE — this is the "wrong password" one data 525 = 1317 = ERROR_NO_SUCH_USER data 532 = 1330 = password expired data 775 = 1909 = account locked out data 569 = 0x569 = 1385 = ERROR_LOGON_TYPE_NOT_GRANTED I was getting 569, not 52e. My password was correct all along. AD validated it, then refused the logon type. Why that happens xo-server-auth-ldap verifies a password the only way LDAP allows — it re-binds to the directory as the user. Against Active Directory, an LDAP simple bind to a DC is processed as a Type 3 (network) logon on that DC. So if an account is caught by "Deny access to this computer from the network" (SeDenyNetworkLogonRight) in the Default Domain Controllers Policy (or any other WINNING GPO, for that matter), it cannot complete an LDAP bind — no matter how correct the password is, and no matter which LDAP client is asking. My environment uses a tiered admin model. Non-domain-admin admin groups are explicitly denied network logon to the DCs. My admin account is in those groups. Hence 569, every single time, by design. Why it looked intermittent It wasn't. I sampled it either side of a config change. I could prove a bind had succeeded recently, because my LDAP-only XO account (no local password on the record at all) minted an API token on 28 July. Then on 31 July I restored RBAC settings on the Default Domain Controllers Policy that had drifted at some point — I found that during unrelated PKI work. GptTmpl.inf last-write confirms it. The token's last successful use is about eleven hours before that edit. Two deterministic states, one config change in the middle. That's the whole "intermittency." My 2023 "seven security groups" theory was wrong For the record, since it's still up there and someone will find it: I removed group memberships one at a time until auth worked, and concluded there was a membership count limit. There isn't. My own control test disproved it at the time — adding fifteen groups never reproduced the failure — and I should have taken that seriously instead of filing it under "weird." The variable was never the count. It was which group. One of the removals happened to drop the account out of a denied group. My other closing theory in this thread — special-character handling in the username or password — was also wrong. Getting 569 back proves AD parsed the escaped DN (CN=Agbasi\, Kismet), found the object, and got as far as evaluating the password. A mangled DN gives you 525 or a DN syntax error, not a logon-rights rejection. ldapts and passport were behaving correctly throughout. How to check this in 60 seconds Run the plugin test CLI and note the data value. Convert hex → decimal, look it up in Microsoft's System Error Codes list. On the DC, look for Security event 4625 with Sub Status 0xC000015B (STATUS_LOGON_TYPE_NOT_GRANTED). Fastest test of all — from a workstation, as the affected account: net use \\dc01\sysvol. If network logon to the DC is denied, this fails too, and you've confirmed it without touching XO at all. Check the policy directly: $p = "\\mydomain.net\SYSVOL\mydomain.net\Policies\{6AC1786C-016F-11D2-945F-00C04fB984F9}" + "\Machine\Microsoft\Windows NT\SecEdit\GptTmpl.inf" Select-String -Path $p -Pattern "SeDenyNetworkLogonRight|SeNetworkLogonRight" Resolve the SIDs and see whether your user is in any of the denied groups. Also worth checking your grant side: if Access this computer from the network doesn't list Authenticated Users directly, ordinary users are probably getting it transitively via Pre-Windows 2000 Compatible Access. Worth confirming before you assume a plain non-privileged account will work. What I am NOT doing Removing those groups from the deny right. It's doing exactly what I rebuilt it to do. Restoring an app login by handing admin groups network access to the DCs for SMB/RPC/LDAP is a bad trade, and I'd just be undoing my own remediation. Fix Interim: local XO accounts for the admins who need them. No AD objects created, nothing to unwind later, per-user attribution preserved in the audit log. Long term: federate XO through Keycloak (OIDC) instead of LDAP. Kerberos ticket issuance is a KDC service operation and is not gated by SeNetworkLogonRight — which is exactly why these accounts log into workstations all day while failing an LDAP bind. ️ Important if you go the Keycloak route: Keycloak's LDAP user federation validates passwords by doing an LDAP bind. Configure it that way and you'll hit data 569 inside Keycloak instead of inside XO and gain nothing. Password validation has to be delegated to Kerberos/GSSAPI. This will bite you on anything else you point at LDAP too — Bitwarden, NPM, TrueNAS, the lot. Worth solving once at the IdP. One request for Vates @olivierlambert @julien-f — you were right that it was environmental, and I owe you both thanks for the time you put in back in 2023. That said, there's a real (small) improvement available here. xo-server collapses every auth provider exception into a generic invalid credentials, and the plugin only emits the actual AD error at DEBUG. The DC told us precisely what was wrong on the very first attempt — it just never reached anywhere a user would look. Surfacing the LDAP result code and the AD data sub-code at INFO on failure, and in the plugin test output in the UI, would turn this class of problem from a multi-year hunt into a single-session diagnosis. Happy to open an issue on GitHub with the full reproduction if that's useful. Hope this saves someone else three years. If you found this thread by searching data 569, *ERROR_LOGON_TYPE_NOT_GRANTED**, or "LDAP invalid credentials but password is correct" — check your Deny access to this computer from the network user right first. That's almost certainly it.

Hard reboot sorted itself out

@john.c said: nd need to be cleared, before re-attempting the update. Checking the logs will help Just for completeness... it kept doing this after all the typical diagnosis steps. So never found a solution Definitely wasn't cache or disc space related. In the end I built the XOA from source on the same VM and that worked fine. Although subsequent to that I've moved XOA to the containerised version thus saving an extra VM!.

@irtaza9 i would look a the usual suspect, /var/log/SMlog /var/log/xensource.log see https://docs.xcp-ng.org/troubleshooting/

@yllar Hello, We are still validating the ISOs, but we are getting closer to releasing them. We hope to make them available soon.

I have also opened a support ticket with Veeam. Well trying to... Issue with my account preventing me to and working to fix that issue. Veeam forum did make this statment about CBT and snapshots - There is one known snapshot issue that I'm guessing could possibly have this side-effect (noted in the release notes BTW). We take standard Xen snapshots for CBT but then immediately rename them with our own naming convention. We've noticed that sometimes this rename operation returns success status even if it fails. In this case I can see how this might throw this sort of error but again best to confirm w/support Veeam support ticket - 08188639 Update from Veeam - Thanks for the update I'm currently checking with my team and I will update as soon as possible, what I noticed in the debug logs is the following API failing \Backup\Plugins\XEN\Backup\Backup Job 1\Delilah ArcDC02 with debug enable 2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]: <== Request localhost:19000, IsTaskFinished, body: {"taskId":"491371d9-4699-4087-b549-9862d2188177"} 2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]: ==> Response localhost:19000, IsTaskFinished, success, duration: 0.1404 msec, body: false 2026-08-07 20:05:04.843 00019 INFO | [XenRpcClient]: Start ListChangedBlocks 2026-08-07 20:05:04.843 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/Async.VDI.list_changed_blocks, body: ["OpaqueRef:b77ca409-d438-d97e-d8c3-f212647ad469","OpaqueRef:8c83f698-8836-3b4c-1506-8e97315aae1f"] 2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/Async.VDI.list_changed_blocks code: "OK", duration: 2 msec, body:{"opaque_ref":"OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"} 2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"] 2026-08-07 20:05:04.847 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 2 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":["cancel"],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"1970-01-01T00:00:00Z","status":"pending","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":0.0,"type":"<none/>","result":"","error_info":[],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"()","opaque_ref":null} 2026-08-07 20:05:04.848 00019 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "pending" Progress: 0 2026-08-07 20:05:06.848 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"] 2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 1 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":[],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"2026-08-08T00:05:05Z","status":"failure","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":1.0,"type":"<none/>","result":"","error_info":["SR_BACKEND_FAILURE_460","","Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]",""],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 150))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))","opaque_ref":null} 2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "failure" Progress: 1 2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.destroy, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"] 2026-08-07 20:05:06.851 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.destroy code: "OK", duration: 0.735 msec, body:"" 2026-08-07 20:05:06.851 00018 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. ---> Failed to calculate changed blocks for given VDIs. --- End of inner exception stack trace --- at Veeam.XenBackup.RestClient.XenRpcClient.GetTaskResult(XenRef`1 taskRef, CancellationToken cancellationToken) at Veeam.XenBackup.RestClient.XenRpcClient.<>c__DisplayClass94_0.<<ListChangedBlocksAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Veeam.Vbf.Common.Helper.Retry.RetryHelper.ExecuteActionAsync[T](Func`2 asyncAction, String description, ILogger logger, LogLevel logLevel, CancellationToken cancellationToken) 2026-08-07 20:05:06.851 00018 INFO | [XenRpcClient]: Retry after 10 sec. Retry 6/10 for ListChangedBlocks There is a XEN forum and guide from him, but I need to check internally as first https://docs.xenserver.com/en-us/xenserver/developer/changed-block-tracking-guide/troubleshoot.html#you-cant-list-changed-blocks-between-two-vdi-snapshots CBT: the thread to centralize your feedback | XCP-ng and XO forum https://xcp-ng.org/forum/topic/9268/cbt-the-thread-to-centralize-your-feedback/364 Regards Update - Veeam came back and suggested I power off the vms with the warrning and power back on. This cleared the error for 2 out of the 4 vms. 1 vm still showed the error the other vm i was not able to power off at that time. @olivierlambert - From veeam... After checking with my team, is it possible if you can engage XEN support on this, I checked some of the XEN forum including the KB article about how to troubleshoot CBT errors, seems to be a clean metadata that can be done from their side Troubleshoot Changed Block Tracking | Develop for XenServer CBT: the thread to centralize your feedback | XCP-ng and XO forum Regards Looks like our issue may be different at the end even with involving CBT... Let me know if i should continue in a new post. Support ticket created - Ticket#7762393

So I ended up going with the Windows B&R as i could net setup the windows mount server when setting up the repo for the backups. It would setup the linux one but not the windows one. This was with using the veeam console from a windows client. So i ended up with the Windows one. Now after a few backups i am see this warning / error.... 8/5/2026 5:34:50 PM Warning : Failed to use CBT: [Task e75c4247-2ee1-7b51-088c-b970ace60f3d (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated] First i thought it was because i maped the new job to older backups from Beta v1. So i purged all old backups and started fresh. The full backups were successful. Now on first delta 4 out of 7 vms have that warrning. This a Veeam issue or xcp-ng?

What kind of storage are you using? Is it a 4K block native storage? You should be able to see if you can create a virtual disk easily with xe vdi-create name-label=test virtual-size=1G sr-uuid=356475ac-…

Hi, I found your original request. I will make sure someone from the Sales team responds ASAP. Regards, Dan

If i recall from my migration I belive thats all there is to it. I know the docs mention about doing a test migration but mine just migrated and it just worked as you stated after making the needed adjustments. Let other chime in but i think you are good to go. And you could delete the snapshot from the vm.

Very cool, thanks for the feedback @maximsachs !

This might have worked, at that time i've allready created new VMs from the information inside the Backup xml file. It worked out allright but building VMs this way, is quite tidious. Maybe there could be a way to find the correct installer and patches, to get a new host in a state to restore a pool backup, would be good.

Hi! No, but check XO logs/output to see what's going on

Filed both XO related issues: Memory visibility at VM creation: https://github.com/vatesfr/xen-orchestra/issues/10225 domain_crash invisible in XO: https://github.com/vatesfr/xen-orchestra/issues/10226 Linking to this thread as promised earlier, thanks all. Hope this helps someone out.

@poddingue, that is true, my first post here. I'll try to play with these BIOS settings again. and keep you posted. Cheers

Drivers are actually in @Team-Hypervisor-Kernel scope nowadays

@poddingue Borrow away - "smoke alarm" is a better name for it than anything we had, so we might borrow it right back. Since you mentioned reading the matrix - here is the classification you would be reading, straight from the file: grep access: dadl/xen-orchestra.dadl | sort | uniq -c 49 access: admin 21 access: dangerous 122 access: read 75 access: write 122 of the 267 tools are plain read - that is the entire surface a review-capped agent gets. The other 145 exist in the same file, but for that agent they might as well not. The whole security taxonomy is greppable plaintext - which is rather the point of a declarative format. And if anything in the matrix looks wrong or missing, this thread is exactly the right place - real-world corrections are how it improves.

@yannsionneau , may this link helps if you not anyway know it https://medium.com/@timemaster5/getting-amd-phoenix-igpu-passthrough-working-on-xen-xcp-ng-8856593f9c0d I will give it a try this weekend

@oliv77 said: Hi, XOA version: xo-server 5.193.1 xo-web-premium 5.191.0 We've noticed on our production XOA server that when we put an XCPNG host in a pool into maintenance mode, we reboot it and then it reconnects to its pool but is no longer in maintenance mode. Is it possible to configure the XCPNG hosts so that they automatically return to maintenance mode after a reboot? Regards, Oliv77 Hi @oliv77, XO 6.7 - latest - adds this option in REST API and next month we should have the available action (Disable host and evacuate VM) in XO6 !