Windows guests destroyed by PoD exhaustion with Citrix tools — XCP-ng tools fix it, but XO gives no warning either way

kagbasi-wgsdac

Hello Folks,

Posting this because it cost me an evening and was misattributed to four different subsystems before I found it, and because I think the fix and the reporting gap are separable questions.

Summary

Windows Server 2022 guests on XCP-ng 8.3 were being destroyed and restarted every 12–15 minutes. No bugcheck, no dump, nothing in the Windows event log except event 41 / 6008 — "rebooted without cleanly shutting down."

The cause was populate-on-demand exhaustion:

(XEN) p2m_pod_demand_populate: Dom124 out of PoD memory! (tot=1049115 ents=3145696 dom0)
(XEN) domain_crash called from p2m_pod_demand_populate+0x4e2/0x8c0

The VMs had memory-static-max at 16 GiB with memory-dynamic-max at 4 GiB. Xen backs the dynamic amount and populates the rest on demand, relying on the balloon driver to return pages before the pool runs dry.

The balloon driver in the Citrix XenServer VM Tools 9.4.0 does not reclaim under that pressure.
Replacing them with XCP-ng Windows Guest Tools 9.1.200.0 fixed it completely.

The test

Two VMs from the same sysprepped Server 2022 template, same host, identical memory configuration, differing only in guest tools.

	Arm A	Arm B
Guest tools	Citrix XenServer VM Tools 9.4.0	XCP-ng Windows Guest Tools 9.1.200.0
`memory-static-max`	17179869184 (16 GiB)	17179869184 (16 GiB)
`memory-dynamic-max`	4294967296 (4 GiB)	4294967296 (4 GiB)
`memory-dynamic-min`	4294967296 (4 GiB)	4294967296 (4 GiB)
`memory-static-min`	1073741824 (1 GiB)	1073741824 (1 GiB)

At idle both ballooned correctly — memory-actual 4297175040 and 4297027584 against a memory-target of 4294967296. Ballooning at idle is not the differentiator. Both drivers do it. The difference only appears when the guest touches a page beyond what is backed.

Results:

Test	Arm A (Citrix)	Arm B (XCP-ng)
Unattended, idle, 22 hours	4 recreations	0 recreations
Interactive login	crashed shortly after	survived
Allocation loop to 14 GiB	domain destroyed in 13 seconds	completed
Domain IDs over 22 h	137, 139, 140, 141, 151	138, unchanged

The allocation loop was a PowerShell script holding 256 MiB chunks and touching every page. Arm A's log file contains its start line and nothing else — not one chunk completed. The host watcher:

07:47:04  PoD=170  TEST-DMC-B=138 TEST-DMC-A=140
07:47:19  PoD=173  TEST-DMC-B=138                  <- arm A gone
07:47:34  PoD=173  TEST-DMC-B=138 TEST-DMC-A=141   <- recreated

Arm B ran the identical script to completion. Windows threw OutOfMemoryException per chunk once it hit its own limit, which is the correct failure — the guest refusing an allocation is right, the hypervisor destroying the domain is not.

I then left both running and kept sampling every 15 seconds. Over the following 22 hours, 5,255 samples:

Arm B held a single domain ID (138) throughout. Not one recreation.
Arm A was destroyed and rebuilt four times — domain IDs 137, 139, 140, 141, 151 — with the host's PoD crash count going 167 to 176.

Same host, byte-identical memory limits, both idle at a logon prompt. (Domain IDs increment globally, so the gaps in arm A's sequence are other VMs starting, not extra crashes.)

What I think is worth discussing

I do not think Xen's behaviour is the bug. Xen cannot swap, PoD exhaustion leaves it no option, and an operator who sets static-max at four times dynamic-max has accepted that risk. Fair enough.

What made this take an evening is that nothing anywhere says so.

XO's VM creation form writes memory-dynamic-max and nothing else. I verified this separately: cloning a template that carried 4-to-16 and setting the form's RAM field to 8 GiB produced dynamic-min 4 GiB, dynamic-max 8 GiB, static-max 16 GiB. Three of the four values came from the template. The operator types one number and sees one number; the Advanced fields that would reveal the rest are collapsed by default.

This was the XO 5 UI, not XO 6 — see the note below, since it may already be different there.
XO reports nothing when a domain is killed this way. No alert, no task, no message on the VM.
With on_crash=Start the domain restarts immediately, so the guest records only an unclean restart with no bugcheck code — which points the operator at Windows, which is innocent.
The cause appears only in xl dmesg on the host, which is not visible through XO at all.

So a supported path through the UI produces a configuration that will destroy the guest, and nothing in the management layer indicates it either at creation or at failure.

A caveat on all of the above: I was using the XO 5 interface throughout. I have not tried this in XO 6, so if the new VM creation form surfaces all four memory values, or warns when a clone's dynamic-max lands well below the template's static-max, then part of this is already solved and I am reporting a problem you have fixed. Worth someone confirming either way, because the XAPI-level behaviour is the same underneath and the difference would be entirely in what the UI shows.

The domain_crash reporting gap is more likely to be common to both, since it is about surfacing a host-level event rather than about a form.

Two things that would have saved the evening, in rough order of value:

A warning in XO when a VM's static-max materially exceeds its dynamic-max — or simply surfacing all four values where RAM is set, rather than one.
Something visible in XO when a domain is destroyed by domain_crash. Even a task or a VM message would have pointed me at the host log on day one.

And if the Citrix tools genuinely do not reclaim under PoD on XCP-ng, that seems worth stating plainly in the guest tools documentation. The current guidance recommends XCP-ng tools without naming this as a consequence, and the failure mode looks nothing like a driver problem.

Environment

XCP-ng 8.3.0, platform 3.4.0, xapi 26.1.11, build 20260707
Xen 4.17.6-9, changeset 8c80ec836310, pq 9f872f8199c0
Xen Orchestra: XO 5 UI (Community Edition, from sources)
Host: 768 GiB RAM, no memory pressure at any point (710 GiB free during testing)
Guests: Windows Server 2022 Standard, 10.0.20348, UEFI, Secure Boot

Happy to provide the full watcher log, xl dmesg extracts, or the allocation script if useful.

Is this known? Is the Citrix-tools reclaim failure expected on XCP-ng, or worth reporting upstream? And does the XO 6 creation form already handle the memory side of this differently?

poddingue

The PoD crash half is known: https://xcp-ng.org/forum/topic/10179/rockylinux-vm-s-random-reboots hit the same p2m_pod_demand_populate line, and the answer there was closing the gap between static-max and dynamic-max, with the background at https://xenproject.org/blog/ballooning-rebooting-and-the-feature-youve-never-heard-of/ .
The guest tools angle I have not seen reported before, and the 13-second repro makes it hard to wave off. What gives me pause is https://xcp-ng.org/forum/topic/11955/memory-ballooning-dmc-broken-since-xcp-ng-8.3-january-2026-patches, where the Vates Rust guest agent stopped re-arming ballooning after live migration and the Citrix utilities were the ones behaving.
Different guest OS and probably a different mechanism, so it may well be unrelated, but it makes me wary of reading your result as "Citrix tools do not reclaim" rather than the narrower "on Windows under PoD pressure".
I could not find an existing issue for either of your XO asks, so both look worth filing at vatesfr/xen-orchestra, and they stand on their own regardless of where the tools question lands.

Whether the XO 6 creation form already treats the memory side differently, I do not know, and I would rather say that than guess.

kagbasi-wgsdac

@poddingue Thanks — that's a useful correction and two useful references.

On the narrower claim: you're right, and I'll take the correction. What I tested is Windows Server 2022, Citrix tools 9.4.0, under PoD pressure, with no migration involved. "Citrix tools do not reclaim" overstates it. The supported statement is: under PoD pressure on a Windows guest, the Citrix 9.4.0 balloon driver did not return pages fast enough to prevent domain destruction, where
the XCP-ng 9.1.200.0 driver did. One OS, one workload shape, one version of each toolset.

I've read topic/11955 properly and I think it points away from a shared cause rather than towards one, though it's a fair thing to have raised:

	topic/11955	Mine
Guest	Linux (Debian 12/13)	Windows Server 2022
Agent	Rust `xen-guest-agent`	Citrix XenServer VM Tools 9.4.0
Trigger	live migration / resume	none — idle, and under allocation
Balloon at idle	correct before migration	correct throughout, `memory-actual` tracks `memory-target`
Failure	stuck at `dynamic-min`, undersized	domain destroyed by `domain_crash`
Fixed by	reverting to Citrix utilities	replacing Citrix with XCP-ng tools

The squeezed log in that thread is explicit — "domid 53 just started a guest agent (but has no balloon driver)" — so the driver was absent after resume. In my case the driver is present and ballooning correctly at idle on both arms; the divergence only appears when the guest touches a page beyond what's backed. Different mechanism, and the direction of the fix is opposite.

Worth saying that thread also makes the general point better than my post did: whether ballooning works is guest-agent-specific and version-specific in both directions, and neither toolset is categorically the safe one. That's a stronger reason to warn on the memory configuration than anything about a particular driver, since the configuration is what makes any reclaim failure fatal rather than merely untidy.

On the PoD half being known — thanks for topic/10179 and the Xen Project ballooning article. I should have led with "PoD exhaustion under a wide static/dynamic gap is documented; what I think is new is the guest tools variable and the reporting gap." Closing the gap is the right advice and it is what I've done: the template now ships all four values coherent, and the range is only used
where the XCP-ng tools are installed.

On the XO asks — filing both, and I'll link the issues back here:

Warn (or surface all four values) when a clone's memory configuration leaves static-max materially above dynamic-max. Note that the creation form writes dynamic-max only — dynamic-min, static-max and static-min all come from the template, so an operator setting one number gets three they never saw.
Surface domain_crash in XO. With on_crash=Start the domain is back before anyone notices, and the only trace is xl dmesg on the host.

Agreed they stand on their own regardless of where the tools question lands — arguably more so given how differently these two threads' guest agents behaved.

On XO 6 — appreciated, and I'd rather have "I don't know" than a guess. I'll test it myself when I next rebuild and report back either way.

Happy to run further tests if useful. Two that would sharpen this and that I have the rig for:

Same Windows test with Citrix 8.4 rather than 9.4.0, since 8.4 is the version that behaved correctly for Linux in topic/11955. If 8.4 holds on Windows under PoD too, the finding narrows to a specific version rather than the toolset.
Whether XCP-ng Windows tools survive live migration with DMC, which is the trigger from that thread and the one thing my test never exercised.

Full watcher log, xl dmesg extracts and the allocation script are still available if anyone wants to reproduce.