RESOLVED — root cause found, three years later. Leaving a full write-up for anyone who lands here from a search.

Short version: this was never an XO bug, and it was never intermittent. The answer was sitting in the very first test-cli.js output I posted back in May 2023, and I misread it — as did everyone else in this thread, myself very much included.

The line that mattered

failed to bind as CN=Agbasi\, Kismet,...:
80090308: LdapErr: DSID-0C090434, comment: AcceptSecurityContext error, data 569, v4f7c

We all pattern-matched AcceptSecurityContext error to "bad credentials" and moved on. But the meaning is entirely carried by the data field, which is the underlying Win32 status in hex:

• data 52e = 0x52E = 1326 = ERROR_LOGON_FAILURE — this is the "wrong password" one
• data 525 = 1317 = ERROR_NO_SUCH_USER
• data 532 = 1330 = password expired
• data 775 = 1909 = account locked out
• data 569 = 0x569 = 1385 = ERROR_LOGON_TYPE_NOT_GRANTED

I was getting 569, not 52e. My password was correct all along. AD validated it, then refused the logon type.

Why that happens

xo-server-auth-ldap verifies a password the only way LDAP allows — it re-binds to the directory as the user. Against Active Directory, an LDAP simple bind to a DC is processed as a Type 3 (network) logon on that DC.

So if an account is caught by "Deny access to this computer from the network" (SeDenyNetworkLogonRight) in the Default Domain Controllers Policy (or any other WINNING GPO, for that matter), it cannot complete an LDAP bind — no matter how correct the password is, and no matter which LDAP client is asking.

My environment uses a tiered admin model. Non-domain-admin admin groups are explicitly denied network logon to the DCs. My admin account is in those groups. Hence 569, every single time, by design.

Why it looked intermittent

It wasn't. I sampled it either side of a config change.

I could prove a bind had succeeded recently, because my LDAP-only XO account (no local password on the record at all) minted an API token on 28 July. Then on 31 July I restored RBAC settings on the Default Domain Controllers Policy that had drifted at some
point — I found that during unrelated PKI work. GptTmpl.inf last-write confirms it. The token's last successful use is about eleven hours before that edit.

Two deterministic states, one config change in the middle. That's the whole "intermittency."

My 2023 "seven security groups" theory was wrong

For the record, since it's still up there and someone will find it: I removed group memberships one at a time until auth worked, and concluded there was a membership count limit. There isn't. My own control test disproved it at the time — adding fifteen groups
never reproduced the failure — and I should have taken that seriously instead of filing it under "weird." The variable was never the count. It was which group. One of the removals happened to drop the account out of a denied group.

My other closing theory in this thread — special-character handling in the username or password — was also wrong. Getting 569 back proves AD parsed the escaped DN (CN=Agbasi\, Kismet), found the object, and got as far as evaluating the password. A mangled DN gives you 525 or a DN syntax error, not a logon-rights rejection. ldapts and passport were behaving correctly throughout.

How to check this in 60 seconds

1. Run the plugin test CLI and note the data value. Convert hex → decimal, look it up in Microsoft's System Error Codes list.

2. On the DC, look for Security event 4625 with Sub Status 0xC000015B (STATUS_LOGON_TYPE_NOT_GRANTED).

3. Fastest test of all — from a workstation, as the affected account: net use \\dc01\sysvol. If network logon to the DC is denied, this fails too, and you've confirmed it without touching XO at all.

4. Check the policy directly:

   $p = "\\mydomain.net\SYSVOL\mydomain.net\Policies\{6AC1786C-016F-11D2-945F-00C04fB984F9}" + "\Machine\Microsoft\Windows NT\SecEdit\GptTmpl.inf"
   Select-String -Path $p -Pattern "SeDenyNetworkLogonRight|SeNetworkLogonRight"
   

5. Resolve the SIDs and see whether your user is in any of the denied groups.

Also worth checking your grant side: if Access this computer from the network doesn't list Authenticated Users directly, ordinary users are probably getting it transitively via Pre-Windows 2000 Compatible Access. Worth confirming before you assume a plain
non-privileged account will work.

What I am NOT doing

Removing those groups from the deny right. It's doing exactly what I rebuilt it to do. Restoring an app login by handing admin groups network access to the DCs for SMB/RPC/LDAP is a bad trade, and I'd just be undoing my own remediation.

Fix

• Interim: local XO accounts for the admins who need them. No AD objects created, nothing to unwind later, per-user attribution preserved in the audit log.
• Long term: federate XO through Keycloak (OIDC) instead of LDAP. Kerberos ticket issuance is a KDC service operation and is not gated by SeNetworkLogonRight — which is exactly why these accounts log into workstations all day while failing an LDAP bind.

️ Important if you go the Keycloak route: Keycloak's LDAP user federation validates passwords by doing an LDAP bind. Configure it that way and you'll hit data 569 inside Keycloak instead of inside XO and gain nothing. Password validation has to be delegated to
Kerberos/GSSAPI.

This will bite you on anything else you point at LDAP too — Bitwarden, NPM, TrueNAS, the lot. Worth solving once at the IdP.

One request for Vates

@olivierlambert @julien-f — you were right that it was environmental, and I owe you both thanks for the time you put in back in 2023.

That said, there's a real (small) improvement available here. xo-server collapses every auth provider exception into a generic invalid credentials, and the plugin only emits the actual AD error at DEBUG. The DC told us precisely what was wrong on the very first
attempt — it just never reached anywhere a user would look.

Surfacing the LDAP result code and the AD data sub-code at INFO on failure, and in the plugin test output in the UI, would turn this class of problem from a multi-year hunt into a single-session diagnosis. Happy to open an issue on GitHub with the full reproduction if that's useful.

Hope this saves someone else three years. If you found this thread by searching data 569, *ERROR_LOGON_TYPE_NOT_GRANTED**, or "LDAP invalid credentials but password is correct" — check your Deny access to this computer from the network user right first. That's almost certainly it.

Filed both XO related issues:

• Memory visibility at VM creation: https://github.com/vatesfr/xen-orchestra/issues/10225
• domain_crash invisible in XO: https://github.com/vatesfr/xen-orchestra/issues/10226

Linking to this thread as promised earlier, thanks all. Hope this helps someone out.

@dinhngtu That explains it, and it's the answer I was missing. Thank you.

Ran the command on both arms. The Xenbus versions are the part worth having in the thread:

Arm A — Citrix XenServer VM Tools 9.4.0, the one that was destroyed:

major: 9; minor: 4; micro: 0; build: 146;
xennet:   XenServer 9.1.7.65
xeniface: XenServer 9.1.10.87
xenvif:   XenServer 9.1.12.101
xenvbd:   XenServer 9.1.8.79
xenbus:   XenServer 9.1.9.105

Arm B — XCP-ng Windows Guest Tools 9.1.200.0, the one that held:

major: 9; minor: 1; micro: 200; build: 0;
xennet, xencons, xeniface, xenvif, xenvbd, xenbus: all XCP_ng 9.1.200.0

So arm A was on xenbus 9.1.9.105, against the 9.1.13.23 you'd expect from 9.6.0. Four minor versions behind on the driver that actually matters here.

That reframes the finding, and I'd rather state the narrower version. This is not "Citrix tools do not reclaim under PoD" — it is "a Xenbus balloon driver old enough to predate the upstream fix does not reclaim under PoD, and 9.4.0 ships one." Which also fits @poddingue's counter-example neatly: different toolset versions, different behaviour, and no reason to expect a categorical
answer either way.

I'm not going to retest with 9.6.0 — I've moved to the XCP-ng tools and have no reason to go back, particularly with a new release coming. But I'd suggest the thread is still useful as a signpost for two audiences:

Anyone still on Citrix tools older than 9.6.0 can check in one command rather than inferring from a tools version, which is what I could not do before you posted this:

xe vm-param-get uuid=<VM UUID> param-name=PV-drivers-version

If xenbus reads below 9.1.13.23 and the VM has a wide static-max to dynamic-max gap, that's the combination — and the symptom is a guest that reports only an unclean restart with no bugcheck, so nothing in the guest points at the driver.

And anyone on XCP-ng wondering whether to switch: this is a concrete reason rather than a general recommendation. The XCP-ng tools were current, the Citrix ones I happened to have were not, and nothing in either the guest or XO indicated the difference until the domain started dying.

Which leaves the two XO asks standing on their own, and I think more strongly than before. If the tools half is a stale-driver problem with a known fix, then the thing that actually cost the evening was that a supported configuration can destroy a guest with no warning at creation and no report at failure — regardless of which driver happens to be underneath. A domain_crash surfaced in XO
would have pointed me at the host log on day one, whatever the root cause turned out to be.

Still filing both, and I'll link them here.

@poddingue Thanks — that's a useful correction and two useful references.

On the narrower claim: you're right, and I'll take the correction. What I tested is Windows Server 2022, Citrix tools 9.4.0, under PoD pressure, with no migration involved. "Citrix tools do not reclaim" overstates it. The supported statement is: under PoD pressure on a Windows guest, the Citrix 9.4.0 balloon driver did not return pages fast enough to prevent domain destruction, where
the XCP-ng 9.1.200.0 driver did. One OS, one workload shape, one version of each toolset.

I've read topic/11955 properly and I think it points away from a shared cause rather than towards one, though it's a fair thing to have raised:

	topic/11955	Mine
Guest	Linux (Debian 12/13)	Windows Server 2022
Agent	Rust xen-guest-agent	Citrix XenServer VM Tools 9.4.0
Trigger	live migration / resume	none — idle, and under allocation
Balloon at idle	correct before migration	correct throughout, memory-actual tracks memory-target
Failure	stuck at dynamic-min, undersized	domain destroyed by domain_crash
Fixed by	reverting to Citrix utilities	replacing Citrix with XCP-ng tools

The squeezed log in that thread is explicit — "domid 53 just started a guest agent (but has no balloon driver)" — so the driver was absent after resume. In my case the driver is present and ballooning correctly at idle on both arms; the divergence only appears when the guest touches a page beyond what's backed. Different mechanism, and the direction of the fix is opposite.

Worth saying that thread also makes the general point better than my post did: whether ballooning works is guest-agent-specific and version-specific in both directions, and neither toolset is categorically the safe one. That's a stronger reason to warn on the memory configuration than anything about a particular driver, since the configuration is what makes any reclaim failure fatal rather than merely untidy.

On the PoD half being known — thanks for topic/10179 and the Xen Project ballooning article. I should have led with "PoD exhaustion under a wide static/dynamic gap is documented; what I think is new is the guest tools variable and the reporting gap." Closing the gap is the right advice and it is what I've done: the template now ships all four values coherent, and the range is only used
where the XCP-ng tools are installed.

On the XO asks — filing both, and I'll link the issues back here:

1. Warn (or surface all four values) when a clone's memory configuration leaves static-max materially above dynamic-max. Note that the creation form writes dynamic-max only — dynamic-min, static-max and static-min all come from the template, so an operator setting one number gets three they never saw.
2. Surface domain_crash in XO. With on_crash=Start the domain is back before anyone notices, and the only trace is xl dmesg on the host.

Agreed they stand on their own regardless of where the tools question lands — arguably more so given how differently these two threads' guest agents behaved.

On XO 6 — appreciated, and I'd rather have "I don't know" than a guess. I'll test it myself when I next rebuild and report back either way.

Happy to run further tests if useful. Two that would sharpen this and that I have the rig for:

• Same Windows test with Citrix 8.4 rather than 9.4.0, since 8.4 is the version that behaved correctly for Linux in topic/11955. If 8.4 holds on Windows under PoD too, the finding narrows to a specific version rather than the toolset.
• Whether XCP-ng Windows tools survive live migration with DMC, which is the trigger from that thread and the one thing my test never exercised.

Full watcher log, xl dmesg extracts and the allocation script are still available if anyone wants to reproduce.

Hello Folks,

Posting this because it cost me an evening and was misattributed to four different subsystems before I found it, and because I think the fix and the reporting gap are separable questions.

Summary

Windows Server 2022 guests on XCP-ng 8.3 were being destroyed and restarted every 12–15 minutes. No bugcheck, no dump, nothing in the Windows event log except event 41 / 6008 — "rebooted without cleanly shutting down."

The cause was populate-on-demand exhaustion:

(XEN) p2m_pod_demand_populate: Dom124 out of PoD memory! (tot=1049115 ents=3145696 dom0)
(XEN) domain_crash called from p2m_pod_demand_populate+0x4e2/0x8c0

The VMs had memory-static-max at 16 GiB with memory-dynamic-max at 4 GiB. Xen backs the dynamic amount and populates the rest on demand, relying on the balloon driver to return pages before the pool runs dry.

The balloon driver in the Citrix XenServer VM Tools 9.4.0 does not reclaim under that pressure.
Replacing them with XCP-ng Windows Guest Tools 9.1.200.0 fixed it completely.

The test

Two VMs from the same sysprepped Server 2022 template, same host, identical memory configuration, differing only in guest tools.

	Arm A	Arm B
Guest tools	Citrix XenServer VM Tools 9.4.0	XCP-ng Windows Guest Tools 9.1.200.0
memory-static-max	17179869184 (16 GiB)	17179869184 (16 GiB)
memory-dynamic-max	4294967296 (4 GiB)	4294967296 (4 GiB)
memory-dynamic-min	4294967296 (4 GiB)	4294967296 (4 GiB)
memory-static-min	1073741824 (1 GiB)	1073741824 (1 GiB)

At idle both ballooned correctly — memory-actual 4297175040 and 4297027584 against a memory-target of 4294967296. Ballooning at idle is not the differentiator. Both drivers do it. The difference only appears when the guest touches a page beyond what is backed.

Results:

Test	Arm A (Citrix)	Arm B (XCP-ng)
Unattended, idle, 22 hours	4 recreations	0 recreations
Interactive login	crashed shortly after	survived
Allocation loop to 14 GiB	domain destroyed in 13 seconds	completed
Domain IDs over 22 h	137, 139, 140, 141, 151	138, unchanged

The allocation loop was a PowerShell script holding 256 MiB chunks and touching every page. Arm A's log file contains its start line and nothing else — not one chunk completed. The host watcher:

07:47:04  PoD=170  TEST-DMC-B=138 TEST-DMC-A=140
07:47:19  PoD=173  TEST-DMC-B=138                  <- arm A gone
07:47:34  PoD=173  TEST-DMC-B=138 TEST-DMC-A=141   <- recreated

Arm B ran the identical script to completion. Windows threw OutOfMemoryException per chunk once it hit its own limit, which is the correct failure — the guest refusing an allocation is right, the hypervisor destroying the domain is not.

I then left both running and kept sampling every 15 seconds. Over the following 22 hours, 5,255 samples:

• Arm B held a single domain ID (138) throughout. Not one recreation.
• Arm A was destroyed and rebuilt four times — domain IDs 137, 139, 140, 141, 151 — with the host's PoD crash count going 167 to 176.

Same host, byte-identical memory limits, both idle at a logon prompt. (Domain IDs increment globally, so the gaps in arm A's sequence are other VMs starting, not extra crashes.)

What I think is worth discussing

I do not think Xen's behaviour is the bug. Xen cannot swap, PoD exhaustion leaves it no option, and an operator who sets static-max at four times dynamic-max has accepted that risk. Fair enough.

What made this take an evening is that nothing anywhere says so.

• XO's VM creation form writes memory-dynamic-max and nothing else. I verified this separately: cloning a template that carried 4-to-16 and setting the form's RAM field to 8 GiB produced dynamic-min 4 GiB, dynamic-max 8 GiB, static-max 16 GiB. Three of the four values came from the template. The operator types one number and sees one number; the Advanced fields that would reveal the rest are collapsed by default.

  This was the XO 5 UI, not XO 6 — see the note below, since it may already be different there.

• XO reports nothing when a domain is killed this way. No alert, no task, no message on the VM.

• With on_crash=Start the domain restarts immediately, so the guest records only an unclean restart with no bugcheck code — which points the operator at Windows, which is innocent.

• The cause appears only in xl dmesg on the host, which is not visible through XO at all.

So a supported path through the UI produces a configuration that will destroy the guest, and nothing in the management layer indicates it either at creation or at failure.

A caveat on all of the above: I was using the XO 5 interface throughout. I have not tried this in XO 6, so if the new VM creation form surfaces all four memory values, or warns when a clone's dynamic-max lands well below the template's static-max, then part of this is already solved and I am reporting a problem you have fixed. Worth someone confirming either way, because the XAPI-level behaviour is the same underneath and the difference would be entirely in what the UI shows.

The domain_crash reporting gap is more likely to be common to both, since it is about surfacing a host-level event rather than about a form.

Two things that would have saved the evening, in rough order of value:

1. A warning in XO when a VM's static-max materially exceeds its dynamic-max — or simply surfacing all four values where RAM is set, rather than one.
2. Something visible in XO when a domain is destroyed by domain_crash. Even a task or a VM message would have pointed me at the host log on day one.

And if the Citrix tools genuinely do not reclaim under PoD on XCP-ng, that seems worth stating plainly in the guest tools documentation. The current guidance recommends XCP-ng tools without naming this as a consequence, and the failure mode looks nothing like a driver problem.

Environment

XCP-ng 8.3.0, platform 3.4.0, xapi 26.1.11, build 20260707
Xen 4.17.6-9, changeset 8c80ec836310, pq 9f872f8199c0
Xen Orchestra: XO 5 UI (Community Edition, from sources)
Host: 768 GiB RAM, no memory pressure at any point (710 GiB free during testing)
Guests: Windows Server 2022 Standard, 10.0.20348, UEFI, Secure Boot

Happy to provide the full watcher log, xl dmesg extracts, or the allocation script if useful.

Is this known? Is the Citrix-tools reclaim failure expected on XCP-ng, or worth reporting upstream? And does the XO 6 creation form already handle the memory side of this differently?

@john.c See this post - https://xcp-ng.org/forum/post/105564

@john.c Unfortunately, I cannot answer your question in the affirmative. I only use XCP-ng with TrueNAS.

@Team-Storage have y'all had a chance to take a look at the bug report I filed? This issue is quite serious.

@poddingue You're most welcome, sir.

@poddingue Bug report filed as requested — https://github.com/xcp-ng/xcp/issues/825 — and tagging @Team-Storage per your suggestion.

Full evidence bundle is attached to the issue (versions, sweep output, vhd-util vs xe comparison, SMlog). Summary of what I found:

One correction to the mechanism, and I think it matters. The recap describes is-a-snapshot being flipped to true. On my system that isn't what's happening — is-a-snapshot is false on every affected VDI. The field being wrongly written is snapshot-of, which is getting populated on base disks that aren't snapshots at all. XO's disappearing-disks symptom is consistent with either (it filters on a non-empty snapshot-of), but if the storage team is hunting for a bad is-a-snapshot write, that may be the wrong field. Every affected VDI here looks like:

is-a-snapshot: false      <-- correct
snapshot-of:   <populated with an unrelated VDI's UUID>   <-- wrong

A VDI that is a snapshot of itself. The clearest single artifact:

uuid:          806f7f42-083f-4a40-b3f1-0700d00bab5a
name-label:    WinSrv2022SHB_Disk1_Data
is-a-snapshot: false
snapshot-of:   806f7f42-083f-4a40-b3f1-0700d00bab5a   <-- itself
snapshot-time: 20260709T11:19:15Z
sm-config:     vhd-parent: c86e3247-...   <-- bears no relation to the snapshot-of value

No valid code path produces snapshot-of = self. Whatever writes this field isn't validating the target.

It's still actively corrupting new VDIs — this is not just legacy damage. That self-referential VDI was created 2026-07-09, a week after my patch + reboot. Sweeps 9 days apart went from ~180 → 191 affected VDIs on one SR, and a fourth anchor UUID appeared that didn't exist in the first sweep. Newly created VHDs keep landing in the affected set. So "stop it happening again" is the urgent half of the two-part fix, at least in my case.

The bogus targets cluster onto a tiny anchor set, and the anchors point at each other:

Count	Anchor
97	937c3945 (→ a893fdb4)
50	a893fdb4 (→ ea150883)
37	ea150883
7	806f7f42 (→ itself, new since Jul 9)

That looks less like corrupted lineage and more like the field being filled from an incorrect/uninitialised source.

On-disk VHDs are completely healthy. vhd-util check says valid, parent locators are consistent, GC reports no work. The two VDIs the DB calls parent/child are, on disk, siblings under a common parent. The corruption is purely in the XAPI database — which is good news for recoverability.

The VDI_IN_USE is not a real lock. current-operations is empty, xe task-list is empty, no tapdisk holds it. VM.start fails because it's walking a snapshot relationship that doesn't exist on disk. Reproduces from xe on the pool master with XO entirely out of the path — which is why I filed against xcp-ng/xcp rather than the XO tracker.

Versions: XCP-ng 8.3.0, xapi 26.1.11 (xapi-core-26.1.11-1.2), sm-3.2.12-17.9, sm-fairlock-3.2.12-17.9, blktap 3.55.5-9.1, build 20260618.

I have not attempted to bulk-clear the fields — on-disk data is intact and I'd rather not do a mass write against the XAPI DB on a live SR without guidance. Backing store snapshotted as a safety net.

Happy to run whatever diagnostics would help. And +1 to the hand-grenade feeling — the affected set growing on its own is the part that worries me.

@poddingue Thanks for the response. I got side tracked with another issue and didn't get that bug report filed, I'll do that as soon as possible.

It feels like I'm playing with a hand grenade...lol, quite scary!

@olivierlambert Thanks for the response, much appreciated.

I'm getting ready to file a bug report, as I noticed this morning that this issue is now causing a VDI-IN-USE error; preventing me from starting a VM. Fortunately, that VM isn't critical, so I want to report it and help with the troubleshooting that will lead to a fix before it spreads.

@olivierlambert Any chance you can have someone please take a look at this thread? The issue persists and is creating problems for me. If someone out there has figured out the solution, kindly share, thanks.

Another confirmed data point, with package delta and the specific malformed field.

Host: XCP-ng 8.3.0, xapi 26.1 (build 26.1.4), Xen 4.17.6-9.

Setup: XO from sources (community). All VDIs vanished from the per-VM Disks tab (XO 5 and XO 6); xe and the SR Disks tab show them fine; VMs run normally. Trigger was the 8.3 host update + reboot this morning — XO build unchanged since May 28, disks visible yesterday.

Host update delta (today): all 26.1.3-1.10 → 26.1.4-3.1 (xapi-core, xenopsd, sm-cli, sm-fairlock, xapi-storage-script, vhd-tool, message-switch, etc.), plus sm 3.2.12-17.8 → 17.9 as an independent bump.

The malformed field. An affected live OS disk (VM running):

is-a-snapshot: false
snapshot-of:   <populated, points to another VDI>
snapshot-time: <populated>

A normal base VDI should have an empty snapshot-of. After the update, snapshot-of/snapshot-time are populated on real, non-snapshot base VDIs, and XO filters anything with a non-empty snapshot-of out of the per-VM Disks view — which is the disappearance.

The VDI that snapshot-of points to is a legitimate base image in my environment (a heavily-reused Win2022 build template with a large genuine snapshot/clone lineage), so I can't tell from the host side whether the parentage links themselves changed or only the snapshot-of on live VDI labeling did. Either way, the consumer-visible effect is the same.

REST confirms: /rest/v0/vms/<uuid>/vdis → []; /rest/v0/vdis/<uuid> → "no such VDI" for the VBD's referenced UUID, while xe vdi-list shows it.

Caution for others: since live disks now carry snapshot-like metadata, be careful with Health-dashboard "orphan" cleanup and snapshot deletion on affected VMs until this is understood.

Workaround that restored the per-VM Disks view: snapshot → revert → delete-snapshot (tested on a powered-off VM, immediate).

Happy to provide more diagnostics.

---

Quick Follow-up:

Additional symptom, same root cause: ISO-SR VDIs are also affected. Pre-existing ISOs disappeared from the XO ISO picker (only ISOs uploaded after the patch still show). An affected ISO's vdi-param-list shows:

is-a-snapshot: false
snapshot-of:   937c3945-...   (same anchor UUID as an affected VM disk on a different SR)
snapshot-time: 19700101T00:00:00Z   (Unix epoch — clearly synthetic)

Notably the spurious snapshot-of on both an ISO VDI and an unrelated VM OS disk points to the same anchor UUID, with an epoch timestamp — so this looks like the update is stamping pre-existing VDIs with a bogus snapshot-of rather than any real lineage. VHD chains/GC are clean (GC reports no work).

Tagging a related GitHub Issue for easy correlation - https://github.com/vatesfr/xen-orchestra/issues/9578

@julienXOvates Got it.

I didn't get a chance to work on this over the weekend, so sometime this week, I'll just bounce the host and see if it comes back. Right now, while I can SSH into it, the admin console appears to be frozen and unresponsive (over iLO).

Hello Folks,

I encountered a failure while attempting a Rolling Pool Update (RPU) on my two-node pool on Wednesday and wanted to report the logs to see if this is a known mirror issue or an orchestration edge case.

Environment:

Hypervisor: XCP-ng 8.3 (Two-node pool)
Management: Xen Orchestra (Community Edition / From Sources | Commit 5af3268 )
Update Method: Rolling Pool Update (RPU) feature via XO

Issue Description:
I initiated the RPU task via XOCE. The process failed during the host update phase. The RPU task in the XO UI appears hung (running since 2026-04-29 00:39), likely because the underlying yum process exited with an error that wasn't gracefully handled by the RPU state monitoring mechanism.

The Error:
The logs show a yum failure due to an HTTPS 500 Internal Server Error specifically when trying to fetch the xo-lite package from the mirrors. I can confirm there were no network outages at the time (since I was working remotely and my VPN connection to the site network dropped).

Relevant Log Snippet:

stderr: 'http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/Packages/xo-lite-0.20.0-1.xcpng8.3.noarch.rpm: [Errno 14] HTTPS Error 500 - Internal Server Error
Trying other mirror.

Error downloading packages:
  xo-lite-0.20.0-1.xcpng8.3.noarch: [Errno 256] No more mirrors to try.'

Full JSON Log:

forum won't allow the full log, so I put it on pastebin.  Here's the link:  https://pastebin.com/raw/ueL86MiL

Notes:

It seems like a transient mirror issue, but I wanted to report it as it left the RPU task in a hung state in the UI for over 24 hours. I intend to clear the task and attempt a manual yum update on the hosts as a workaround, however, I'm going to hold off until the end of day today. Perhaps this will provide a live production environment, should the XCP-ng team wish to leverage it, for testing a fix.

Has anyone else seen 500 errors on the xo-lite package recently, or is there a specific mirror I should check?

As it stands now - 2026-05-01 04:30 - the RPU task is still hung (as shown in screenshot #1) and the first host (VMH01 - the pool master) is still in a disabled state (screenshot #2).

Thanks for all the work on 8.3!

SCREENSHOT #1 - Showing hung RPU task

SCREENSHOT #2 - Showing master host in disabled state

@dinhngtu Thanks for the fast response, as always.

Good-day Folks,

Running the XenClean.exe utility that ships with xcpng-winpv-9.1.146.0-Release-x64, on a Windows Server 2022 VM (Version 21H2 - OS Build 20348.5020) with Citrix Guest Tools v9.4.2 installed, results in the following error and a subsequent BSOD with stopcode = INACCESSIBLE_BOOT_DEVICE:

• [Alert] Cleanup FAILED: -2147467259 Msiexec failed with code 1605
• [Interactive] Cleanup task status is Error

Also, contrary to what's stated in the release notes, the VM does not automatically reboot. I suspect that's a consequence of the utility not completing successfully to trigger the reboot.

ULTIMATE SOLUTION FOR ME (might be different for you) :

• Mount any Windows ISO to the VM (I used the Windows Server 2022 ISO)
• Navigate to the Advanced tab of the VM and change the boot order so DVD-Drive is first (don't forget to click Save)
• Press any key, as prompted, to trigger a boot into the ISO
• Press next and select "Repair your computer"
• On the next screen select "Troubleshoot" and then "Command Prompt"
• You will now have a command prompt open and at the X:\ drive
• Type C:\ and hit enter to drop to the system drive for the installed OS
• Navigate to the folder holding the XenBootFix utility. For me it was C:\Users<MyUser>\Downloads\xcpng-winpv-9.1.146.0-Release-x64\package\XenBootFix
• Type XenBootFixe.exe C:\Windows and wait for the utility to finish, then reboot the VM.
• Install the XCP-ng Windows Guest Tools and reboot when prompted.
• After the VM boots back into Windows, confirm on the VM's General Tab that you see the message Management agent 9.1.145-77 detected.
• Don't forget to go back into the VM's Advanced tab and disable the DVD-Drive under Boot Order.

Sharing this, in hopes that the Vates team will see and address this issue, and it might help someone else out of a jam.

@olivierlambert & @mathieura thanks for the speedy response. Duly noted, very much appreciated.

Hi Vates team,

I'm building an automated deployment tool that provisions VMs via the XO REST API (/rest/v0/). The tool needs to work on both Xen Orchestra from Sources (XOCE) and XO Appliance (XOA). I ran into a few API limitations and wanted to share what I found, along with the workarounds I implemented. Hopefully this feedback is useful.

1. cpus field in create_vm — XOCE vs XOA inconsistency

When calling POST /rest/v0/pools/{id}/actions/create_vm, XOCE accepts a cpus field in the request body to set the vCPU count on the new VM. XOA rejects the same payload with a 400 error citing "excess property".

Workaround: I send the payload with cpus included, and if the response contains "excess property", I retry without it. The VM inherits the template's vCPU count in that case, which is acceptable but not ideal — it means XOA users can't set vCPUs at creation time via the REST API.

Request: Could the cpus field be added to XOA's CreateVmBody schema to match XOCE behavior? Or if there's a different field name XOA expects, documentation would be appreciated.

2. No way to rename VDIs via REST API

When cloning a VM from a template, the new VM's VDIs (virtual disks) inherit their names from the template. In a Storage Repository with many VMs, this creates confusing duplicate names. I wanted to rename VDIs after creation to follow a {vm_name}_Disk0_OS convention, but the REST API doesn't appear to support PATCH or PUT on /rest/v0/vdis/{id}.

I also attempted using the JSON-RPC API (/api/) with vdi.set, but discovered that the JSON-RPC endpoint only supports WebSocket connections — HTTP POST to /api/ returns an HTML redirect rather than a JSON-RPC response.

Workaround: I document that users should name the template's VDIs descriptively before converting to a template, since those names propagate to all clones.

Request: Would it be possible to add PATCH support for VDI properties (at minimum name_label) to the REST API? Alternatively, if there's an existing method I'm missing, I'd appreciate a pointer. I see that this may have been mentioned here as well: https://xcp-ng.org/forum/topic/11970/request-add-patch-vms-id-for-updating-vm-properties-name_description-name_label

3. Boot order altered when cloning without vdis array (XO #4980)

When creating a VM via create_vm with clone: true and no vdis array in the payload, the resulting VM has network boot prepended to its boot order (e.g., "ncn" instead of "cn"). This causes the VM to PXE boot instead of booting from disk.

Including a vdis array in the payload — even if the VM doesn't need a new disk — preserves the correct boot order from the template.

Workaround: When no data disk is needed, I inject a temporary 1 GB dummy VDI in the vdis array to force the correct boot order, then delete it immediately after VM creation (before the first boot). This is obviously a hack, but it works reliably.

I believe this is related to XO issue #4980. Any update on whether this will be addressed in the REST API?

---

Environment

• XO versions tested: XOCE 5.x (built from sources, commit d1736) and XOA (v6.1.2)
• XCP-ng: 8.3
• API version: REST v0 (/rest/v0/)
• Automation context: Bash-based installer using curl for all API calls

Happy to provide payload examples or logs if any of the above would benefit from more detail. Thanks for the great platform — these are relatively minor friction points in an otherwise excellent API.

@dinhngtu Yeah, I suspected that as well. So I inspected the SR and it showed connected to both hosts (at least in the XO UI - I didn't drop to the CLI to really confirm).

By altering my workflow a bit and slowing down, I seem to have found the right "sweet spot" of delay and the issue hasn't resurfaced. Here's what I'm doing now, when I need to revert the snapshots of all three VMs:

• In the XO VM list, I select the three VMs and power them off at the same time.
• I then start with VM1 and eject the ISO, VM2 and eject the ISO, then VM3 and eject the ISO. By the time I circle back to VM1 for the next step, about 10-15 secs have elapsed.
• I then start with VM1 and revert the snapshot, and repeat the same on VM2 and VM3. By the time I circle back to VM1 for the next step, another 10-15 secs have elapsed.
• I re-attach the ISO to all three VMs in sequence. Another 10-15 secs elapse.
• I then start with VM1 and power all three VMs sequentially.

The entire workflow takes about 30-45 secs, and I'm finding that by doing this, the issue hasn't resurfaced.