Windows guests destroyed by PoD exhaustion with Citrix tools — XCP-ng tools fix it, but XO gives no warning either way

kagbasi-wgsdac

Hello Folks,

Posting this because it cost me an evening and was misattributed to four different subsystems before I found it, and because I think the fix and the reporting gap are separable questions.

Summary

Windows Server 2022 guests on XCP-ng 8.3 were being destroyed and restarted every 12–15 minutes. No bugcheck, no dump, nothing in the Windows event log except event 41 / 6008 — "rebooted without cleanly shutting down."

The cause was populate-on-demand exhaustion:

(XEN) p2m_pod_demand_populate: Dom124 out of PoD memory! (tot=1049115 ents=3145696 dom0)
(XEN) domain_crash called from p2m_pod_demand_populate+0x4e2/0x8c0

The VMs had memory-static-max at 16 GiB with memory-dynamic-max at 4 GiB. Xen backs the dynamic amount and populates the rest on demand, relying on the balloon driver to return pages before the pool runs dry.

The balloon driver in the Citrix XenServer VM Tools 9.4.0 does not reclaim under that pressure.
Replacing them with XCP-ng Windows Guest Tools 9.1.200.0 fixed it completely.

The test

Two VMs from the same sysprepped Server 2022 template, same host, identical memory configuration, differing only in guest tools.

	Arm A	Arm B
Guest tools	Citrix XenServer VM Tools 9.4.0	XCP-ng Windows Guest Tools 9.1.200.0
`memory-static-max`	17179869184 (16 GiB)	17179869184 (16 GiB)
`memory-dynamic-max`	4294967296 (4 GiB)	4294967296 (4 GiB)
`memory-dynamic-min`	4294967296 (4 GiB)	4294967296 (4 GiB)
`memory-static-min`	1073741824 (1 GiB)	1073741824 (1 GiB)

At idle both ballooned correctly — memory-actual 4297175040 and 4297027584 against a memory-target of 4294967296. Ballooning at idle is not the differentiator. Both drivers do it. The difference only appears when the guest touches a page beyond what is backed.

Results:

Test	Arm A (Citrix)	Arm B (XCP-ng)
Unattended, idle, 22 hours	4 recreations	0 recreations
Interactive login	crashed shortly after	survived
Allocation loop to 14 GiB	domain destroyed in 13 seconds	completed
Domain IDs over 22 h	137, 139, 140, 141, 151	138, unchanged

The allocation loop was a PowerShell script holding 256 MiB chunks and touching every page. Arm A's log file contains its start line and nothing else — not one chunk completed. The host watcher:

07:47:04  PoD=170  TEST-DMC-B=138 TEST-DMC-A=140
07:47:19  PoD=173  TEST-DMC-B=138                  <- arm A gone
07:47:34  PoD=173  TEST-DMC-B=138 TEST-DMC-A=141   <- recreated

Arm B ran the identical script to completion. Windows threw OutOfMemoryException per chunk once it hit its own limit, which is the correct failure — the guest refusing an allocation is right, the hypervisor destroying the domain is not.

I then left both running and kept sampling every 15 seconds. Over the following 22 hours, 5,255 samples:

Arm B held a single domain ID (138) throughout. Not one recreation.
Arm A was destroyed and rebuilt four times — domain IDs 137, 139, 140, 141, 151 — with the host's PoD crash count going 167 to 176.

Same host, byte-identical memory limits, both idle at a logon prompt. (Domain IDs increment globally, so the gaps in arm A's sequence are other VMs starting, not extra crashes.)

What I think is worth discussing

I do not think Xen's behaviour is the bug. Xen cannot swap, PoD exhaustion leaves it no option, and an operator who sets static-max at four times dynamic-max has accepted that risk. Fair enough.

What made this take an evening is that nothing anywhere says so.

XO's VM creation form writes memory-dynamic-max and nothing else. I verified this separately: cloning a template that carried 4-to-16 and setting the form's RAM field to 8 GiB produced dynamic-min 4 GiB, dynamic-max 8 GiB, static-max 16 GiB. Three of the four values came from the template. The operator types one number and sees one number; the Advanced fields that would reveal the rest are collapsed by default.

This was the XO 5 UI, not XO 6 — see the note below, since it may already be different there.
XO reports nothing when a domain is killed this way. No alert, no task, no message on the VM.
With on_crash=Start the domain restarts immediately, so the guest records only an unclean restart with no bugcheck code — which points the operator at Windows, which is innocent.
The cause appears only in xl dmesg on the host, which is not visible through XO at all.

So a supported path through the UI produces a configuration that will destroy the guest, and nothing in the management layer indicates it either at creation or at failure.

A caveat on all of the above: I was using the XO 5 interface throughout. I have not tried this in XO 6, so if the new VM creation form surfaces all four memory values, or warns when a clone's dynamic-max lands well below the template's static-max, then part of this is already solved and I am reporting a problem you have fixed. Worth someone confirming either way, because the XAPI-level behaviour is the same underneath and the difference would be entirely in what the UI shows.

The domain_crash reporting gap is more likely to be common to both, since it is about surfacing a host-level event rather than about a form.

Two things that would have saved the evening, in rough order of value:

A warning in XO when a VM's static-max materially exceeds its dynamic-max — or simply surfacing all four values where RAM is set, rather than one.
Something visible in XO when a domain is destroyed by domain_crash. Even a task or a VM message would have pointed me at the host log on day one.

And if the Citrix tools genuinely do not reclaim under PoD on XCP-ng, that seems worth stating plainly in the guest tools documentation. The current guidance recommends XCP-ng tools without naming this as a consequence, and the failure mode looks nothing like a driver problem.

Environment

XCP-ng 8.3.0, platform 3.4.0, xapi 26.1.11, build 20260707
Xen 4.17.6-9, changeset 8c80ec836310, pq 9f872f8199c0
Xen Orchestra: XO 5 UI (Community Edition, from sources)
Host: 768 GiB RAM, no memory pressure at any point (710 GiB free during testing)
Guests: Windows Server 2022 Standard, 10.0.20348, UEFI, Secure Boot

Happy to provide the full watcher log, xl dmesg extracts, or the allocation script if useful.

Is this known? Is the Citrix-tools reclaim failure expected on XCP-ng, or worth reporting upstream? And does the XO 6 creation form already handle the memory side of this differently?