Can't boot a VM with 1TB memory / 128 CPUs
-
Yeah that makes sense, beyond 64 there's potential topology issues. But I remember we tested internally up to 128 and it worked, probably on a single socket though.
The official limit for strict security support is 32 though: https://docs.xcp-ng.org/installation/requirements/#xcp-ng-83-lts-1
I'm pretty sure we managed to get 128 working. But still, there's a lot of topology work to be done in Xen upstream to go beyond. The real complex limit should be 256. So fixing <256 is probably more bug fixing than huge rework.
-
I remember we tested 128vCPU and it worked (Ubuntu VM, but Debian should be similar).
It was on a 2CRSi machine (1 socket however).
-
Can you try the guest in BIOS mode in case that could change anything?
-
@olivierlambert it works in BIOS mode. The system is Dell Poweredge R7625 with AMD EPYC
I can't go beyond 128 cores : xenopsd internal error: Xenctrl.Error("22: Invalid argument")
-
Okay so there's 2 things:
- UEFI 96 cores limitation is hard coded here: https://github.com/xcp-ng-rpms/edk2/blob/5af940036ace555e5315a78a301912d294277ec0/SPECS/edk2.spec#L138 (to be invetigated)
- BIOS hasn't such limitation, but then you are maxing the next barrier, 128 cores, due to something else (up to 255 vCPUs)
- Then the "final" boss is beyond that
-
@olivierlambert Thanks a lot, the limitations of 128 cores and BIOS are fine for my usage.
-
Thanks, feel free to reach us if you need more details on the point 2 and our progress, we can prioritize if your org is needing it.
-
@olivierlambert it's for a rare usecase and we can wait for a proper fix
-
I can't even imagine how long a memtest would take on a box with this much ram. Ha
-
We are testing some machines with 6TiB RAM
-
@olivierlambert said in Can't boot a VM with 1TB memory / 128 CPUs:
We are testing some machines with 6TiB RAM
My god
