Can't boot a VM with 1TB memory / 128 CPUs

[olivierlambert]

I remember we tested 128vCPU and it worked (Ubuntu VM, but Debian should be similar).

It was on a 2CRSi machine (1 socket however).

[olivierlambert]

Can you try the guest in BIOS mode in case that could change anything?

[fva-anssi]

@olivierlambert it works in BIOS mode. The system is Dell Poweredge R7625 with AMD EPYC

I can't go beyond 128 cores : xenopsd internal error: Xenctrl.Error("22: Invalid argument")

[olivierlambert]

Okay so there's 2 things:

1. UEFI 96 cores limitation is hard coded here: https://github.com/xcp-ng-rpms/edk2/blob/5af940036ace555e5315a78a301912d294277ec0/SPECS/edk2.spec#L138 (to be invetigated)
2. BIOS hasn't such limitation, but then you are maxing the next barrier, 128 cores, due to something else (up to 255 vCPUs)
3. Then the "final" boss is beyond that

[fva-anssi]

@olivierlambert Thanks a lot, the limitations of 128 cores and BIOS are fine for my usage.

[olivierlambert]

Thanks, feel free to reach us if you need more details on the point 2 and our progress, we can prioritize if your org is needing it.

[fva-anssi]

@olivierlambert it's for a rare usecase and we can wait for a proper fix

[DustinB]

I can't even imagine how long a memtest would take on a box with this much ram. Ha

[olivierlambert]

We are testing some machines with 6TiB RAM

[DustinB]

@olivierlambert said in Can't boot a VM with 1TB memory / 128 CPUs:

We are testing some machines with 6TiB RAM

My god