More than 64 vCPU on Debian11 VM and AMD EPYC

TodorPetkov

Hello,

I came across a similar case as described in https://xcp-ng.org/forum/topic/4689/xcp-8-2-vcpus-max-settings
I have server with 2x AMD EPYC 7532. I give 128 vCPUs to Debian11 VM, but it sees only 64 of them. xl vcpu-list shows the upper half of CPUs in paused state. Server is running XCP 8.2, kernel 4.19.19-7.0.14.1 (updated around 4th of December last year).
I managed to fix this by installing "generic" kernel in the VM (linux-image-5.10.0-21-amd64), instead of the usual -cloud (VM comes with 5.10.0-20-cloud-amd64).

When I tried to turn off ACPI as suggested in the post I mentioned, cloud-image sees only one CPU. However, acpi=off in grub is needed for the generic image in order to see all. I haven't played with params on the VM.

I saw few posts here about the vCPU issue, but people are running Intel. Anyone with AMD and similar experience?

Thanks in advance.

olivierlambert

I think that might be a Xen limitation

olivierlambert

Yes, indeed, for now, you can't have more 64vCPU per guest. It's a complex problem (between Xen and the toolstack) regarding the CPU topology.

Some deep rework is needed and this is only possible on future Xen versions. I wouldn't expect a solution before next year.

TodorPetkov

Thanks for the fast reply, is there a documentation/statement about it?

I managed to have more than 64 vCPU with the regular kernel (not the cloud one). I was just curious if anyone has similar experience with the combo AMD/Debian11/cloud kernel.

olivierlambert

https://github.com/xenserver/xen.pg/blob/XS-8.3.x/patches/0001-Partially-revert-08754333892-hvmloader-limit-CPUs-ex.patch

My source is one of the main Xen dev If you have working setups with more than 64 vCPUs, I'm curious!

TodorPetkov

@olivierlambert I have a working VM (booted) that shows 2 sockets, 64 cpu each. I am running sysbench with --max-threads=128 and it shows load 128.
I played a bit, with normal acpi in the VM OS and disabled acpi in xe xe vm-param-set platform:acpi=0 results are the same. I am attaching lscpu and dmesg from the second case (acpi is untouched in the VM, but disabled with xe command)

Let me know if anything else is needed.

dmesg1.txt lscpu1.txt

TodorPetkov

On second thought, I was not clear in the beginning. I don't expect to see 1 socket with 128 vCPU in the VM, but maybe 2 sockets with splitted vCPUs between them if I assign more than 64 to the VM. Initially I had 1 socket with 64 CPU and after turning ACPI off (either in grub or in the VM config itself), secondary socket appeared in the VM with the rest of the CPUs. More funny, turning ACPI off while running the cloud kernel of Debian makes the VM see one CPU only.