More than 64 vCPU on Debian11 VM and AMD EPYC
-
Hello,
I came across a similar case as described in https://xcp-ng.org/forum/topic/4689/xcp-8-2-vcpus-max-settings
I have server with 2x AMD EPYC 7532. I give 128 vCPUs to Debian11 VM, but it sees only 64 of them.xl vcpu-list
shows the upper half of CPUs in paused state. Server is running XCP 8.2, kernel 4.19.19-7.0.14.1 (updated around 4th of December last year).
I managed to fix this by installing "generic" kernel in the VM (linux-image-5.10.0-21-amd64), instead of the usual -cloud (VM comes with 5.10.0-20-cloud-amd64).When I tried to turn off ACPI as suggested in the post I mentioned, cloud-image sees only one CPU. However,
acpi=off
in grub is needed for the generic image in order to see all. I haven't played withparams
on the VM.I saw few posts here about the vCPU issue, but people are running Intel. Anyone with AMD and similar experience?
Thanks in advance.
-
I think that might be a Xen limitation
-
Yes, indeed, for now, you can't have more 64vCPU per guest. It's a complex problem (between Xen and the toolstack) regarding the CPU topology.
Some deep rework is needed and this is only possible on future Xen versions. I wouldn't expect a solution before next year.
-
Thanks for the fast reply, is there a documentation/statement about it?
I managed to have more than 64 vCPU with the regular kernel (not the cloud one). I was just curious if anyone has similar experience with the combo AMD/Debian11/cloud kernel.
-
My source is one of the main Xen dev
If you have working setups with more than 64 vCPUs, I'm curious!
-
@olivierlambert I have a working VM (booted) that shows 2 sockets, 64 cpu each. I am running sysbench with --max-threads=128 and it shows load 128.
I played a bit, with normal acpi in the VM OS and disabled acpi in xexe vm-param-set platform:acpi=0
results are the same. I am attaching lscpu and dmesg from the second case (acpi is untouched in the VM, but disabled withxe
command)Let me know if anything else is needed.
-
On second thought, I was not clear in the beginning. I don't expect to see 1 socket with 128 vCPU in the VM, but maybe 2 sockets with splitted vCPUs between them if I assign more than 64 to the VM. Initially I had 1 socket with 64 CPU and after turning ACPI off (either in grub or in the VM config itself), secondary socket appeared in the VM with the rest of the CPUs. More funny, turning ACPI off while running the cloud kernel of Debian makes the VM see one CPU only.