<![CDATA[Can't boot a VM with 1TB memory / 128 CPUs]]>I'm using XCP-NG 8.3 on a DELL R7625 AMD EPYC Host. It has 1.5 TB memory, 128 CPUs (256 threads) and I try to boot a domU with 128 CPUs and 1TB of memory.

The DomU is created and started, I can see a 100% CPU usage on CPU0 but the console displays "Guest has not initialized the display (yet)."

Virtualization mode : Hardware virtualization (HVM)
CPU limits 128/128
Topology: 2 sockets with 64 cores per socket
Memory limits (min/max) Static: 1 GiB/1 TiB Dynamic: 1 TiB/1 TiB

]]>https://xcp-ng.org/forum/topic/11121/can-t-boot-a-vm-with-1tb-memory-128-cpusRSS for NodeWed, 15 Jul 2026 08:06:42 GMTThu, 31 Jul 2025 09:52:33 GMT60<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 13:53:28 GMT]]>@olivierlambert said in Can't boot a VM with 1TB memory / 128 CPUs:

We are testing some machines with 6TiB RAM 😆

My god

]]>https://xcp-ng.org/forum/post/95849https://xcp-ng.org/forum/post/95849Thu, 31 Jul 2025 13:53:28 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 13:49:28 GMT]]>We are testing some machines with 6TiB RAM 😆

]]>https://xcp-ng.org/forum/post/95847https://xcp-ng.org/forum/post/95847Thu, 31 Jul 2025 13:49:28 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 13:48:56 GMT]]>I can't even imagine how long a memtest would take on a box with this much ram. Ha

]]>https://xcp-ng.org/forum/post/95846https://xcp-ng.org/forum/post/95846Thu, 31 Jul 2025 13:48:56 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 13:45:38 GMT]]>@olivierlambert it's for a rare usecase and we can wait for a proper fix

]]>https://xcp-ng.org/forum/post/95845https://xcp-ng.org/forum/post/95845Thu, 31 Jul 2025 13:45:38 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 12:45:14 GMT]]>Thanks, feel free to reach us if you need more details on the point 2 and our progress, we can prioritize if your org is needing it.

]]>https://xcp-ng.org/forum/post/95842https://xcp-ng.org/forum/post/95842Thu, 31 Jul 2025 12:45:14 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 12:32:19 GMT]]>@olivierlambert Thanks a lot, the limitations of 128 cores and BIOS are fine for my usage.

]]>https://xcp-ng.org/forum/post/95840https://xcp-ng.org/forum/post/95840Thu, 31 Jul 2025 12:32:19 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 12:18:58 GMT]]>Okay so there's 2 things:

  1. UEFI 96 cores limitation is hard coded here: https://github.com/xcp-ng-rpms/edk2/blob/5af940036ace555e5315a78a301912d294277ec0/SPECS/edk2.spec#L138 (to be invetigated)
  2. BIOS hasn't such limitation, but then you are maxing the next barrier, 128 cores, due to something else (up to 255 vCPUs)
  3. Then the "final" boss is beyond that
]]>https://xcp-ng.org/forum/post/95839https://xcp-ng.org/forum/post/95839Thu, 31 Jul 2025 12:18:58 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 12:15:25 GMT]]>@olivierlambert it works in BIOS mode. The system is Dell Poweredge R7625 with AMD EPYC

I can't go beyond 128 cores : xenopsd internal error: Xenctrl.Error("22: Invalid argument")

]]>https://xcp-ng.org/forum/post/95838https://xcp-ng.org/forum/post/95838Thu, 31 Jul 2025 12:15:25 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 11:52:57 GMT]]>Can you try the guest in BIOS mode in case that could change anything?

]]>https://xcp-ng.org/forum/post/95836https://xcp-ng.org/forum/post/95836Thu, 31 Jul 2025 11:52:57 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 11:49:58 GMT]]>I remember we tested 128vCPU and it worked (Ubuntu VM, but Debian should be similar).

htop18vcpus.png

It was on a 2CRSi machine (1 socket however).

]]>https://xcp-ng.org/forum/post/95835https://xcp-ng.org/forum/post/95835Thu, 31 Jul 2025 11:49:58 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 11:42:00 GMT]]>Yeah that makes sense, beyond 64 there's potential topology issues. But I remember we tested internally up to 128 and it worked, probably on a single socket though.

The official limit for strict security support is 32 though: https://docs.xcp-ng.org/installation/requirements/#xcp-ng-83-lts-1

I'm pretty sure we managed to get 128 working. But still, there's a lot of topology work to be done in Xen upstream to go beyond. The real complex limit should be 256. So fixing <256 is probably more bug fixing than huge rework.

]]>https://xcp-ng.org/forum/post/95834https://xcp-ng.org/forum/post/95834Thu, 31 Jul 2025 11:42:00 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 11:36:24 GMT]]>@olivierlambert Debian 12 template with PXE boot, local storage
Host has 2 x AMD EPYC 9534 64-Core Processor / 1.5 TiB of memory (24 x 64 GiB)

The issue comes from the number of CPUs, not the memory. I've made more testing. I've created some VMs from XOA 5.107.1 with template Debian Bookworm 12, PXE boot, local storage (EXT4)

no boot means console with "Guest has not initialized the display"
boot means : Tianocore logo and PXE boot

64x / 1 TiB => boot => upgrade to 128x => no boot
128x / 1TiB => no boot => downgrade to 64x => no boot
96x / 256 GiB => boot => upgrade to 102x => no boot => downgrade to 96x => no boot
97x / 256 GiB => no boot
99x / 512 GiB => no boot

There is a limitation of 96 cores, I've tried different CPU topology without any success

]]>https://xcp-ng.org/forum/post/95832https://xcp-ng.org/forum/post/95832Thu, 31 Jul 2025 11:36:24 GMT<![CDATA[Reply to Can't boot a VM with 1TB memory / 128 CPUs on Thu, 31 Jul 2025 10:43:22 GMT]]>Hi,

What kind of guest do you use? Linux or Windows?

]]>https://xcp-ng.org/forum/post/95828https://xcp-ng.org/forum/post/95828Thu, 31 Jul 2025 10:43:22 GMT