Epyc VM to VM networking slow
-
The different is really weird between Windows and Linux
In theory, you should have a similar result on your Linux VMs vs Windows for the network speed.What is the template you are using for your Debian VMs?
-
I cited this performance problem on Epyc early this year when I was building a cluster for a company. I couldn't get any VM to VM performance, so SR-IOV to the rescue which getting that to work was a mess.
I've got Supermicro H12SSL boards with 7302p processors. I don't have all the memory slots populated so memory throughput isn't as good as it could be, but it sounds like other people are having a similar experience to me.
Currently I'm on the latest 8.3 patched. NICs are Mellanox ConnextX4-LX (not that they matter in guest to guest traffic).
That said...My nearly 12 year-old Xeon E3-1230v2 server crushes the Epyc in guest to guest traffic. Quick test:
Xeon E3-1230v2 iperf3 with 4 threads (Debian guests):
[SUM] 0.00-10.00 sec 15.2 GBytes 13.1 Gbits/sec 0 sender
[SUM] 0.00-10.04 sec 15.2 GBytes 13.0 Gbits/sec receiverEpyc 7302p iperf3 with 4 threads (Debian guests):
[SUM] 0.00-10.00 sec 7.66 GBytes 6.58 Gbits/sec 870 sender
[SUM] 0.00-10.03 sec 7.65 GBytes 6.55 Gbits/sec receiverInteresting to note the retries on the Epyc transfer.
These two CPU's have similar single-thread performance and similar clock-rate. All the C and P states on the Epyc systems have been tweaked.
Guest to Guest traffic is seemingly really impaired compared to the old Xeons.
-
@JamesG can you test between 2x Windows guest? (and/or between 2x BSD guests)
It's maybe a weird Linux thing
What's the kernel version used in your Debian? -
@JamesG said in Epyc VM to VM networking slow:
I've got Supermicro H12SSL boards with 7302p processors. I don't have all the memory slots populated so memory throughput isn't as good as it could be, but it sounds like other people are having a similar experience to me.
That said...My nearly 12 year-old Xeon E3-1230v2 server crushes the Epyc in guest to guest traffic. Quick test:
Guest to Guest traffic is seemingly really impaired compared to the old Xeons.
Finally someone with similar experience, i was starting to think i am a bit crazy
There was update of xen hypervisor today, i will do another test....
-
Here's my thread from earlier this year:
https://xcp-ng.org/forum/topic/6916/tracking-down-poor-network-performance/11
Here's a referenced thread in that thread:
I'd be curious how this works in VMWare.
-
I cloned a Win10 VM...Win10 to Win10, same performance as Debian.
This is most definitely something within the networking infrastructure of Xen.
-
This is already different result than @nicols
-
@olivierlambert Not really...
In Nicols first post, a single threaded iperf got 3.38Gb/s.
Single threaded for me was 3.32Gb/s.With two threads I get 5.19Gb/s
With 20 threads I cap off at 7.02Gb/sThis performance is about the same with Windows VM's and Debian VM's. So it's not a guest OS issue. It's something in the hypervisor.
-
A note...
I'm running a single 16 core, 32 thread second-gen Epyc.
Nicols is running a dual proc, 24 core, 48 thread third-gen Epyc.My base clock rate is 3.0Ghz. His is 2.9Ghz.
The improved caching and memory handling in the third-gen Epyc should be behaving better than my second gen CPU's, but generally speaking, our performance seems to be the same.
-
He said he could reach 18gbits between 2x Windows VM, if I remember correctly.
I wonder about the guest kernel too (Debian 11 vs 12)
-
@olivierlambert With a billion threads.
Anyway...
I'm most definitely a willing subject to help get this resolved. Heck..I'll even give you guys access to the environment to do whatever you want to do. I would just like to see this get fixed.
With that...You guys tell me. What tests do you want run and do you want access to the environment to do your own thing with it?
-
Can you reproduce the same speed he got on your side? Because first we need a consistent result between multiple people to be sure it's not platform related (ie Supermicro or something).
The only reason we could think it's not normal is the difference between Intel and AMD, that shouldn't be that huge. Or maybe AMD CPU are a lot slower with
memcpy()? -
@JamesG said in Epyc VM to VM networking slow:
@olivierlambert With a billion threads.
nope, Win 10 VM with 4 or 8 VCPU and 8GB RAM
But, with billion threads in Linux VM, speed increases up to 8 threats, then it drops again. -
@olivierlambert For single threaded iperf....Yes. Our speeds match 100%. Which is half the transfer rate of a single threaded iperf on 12 year-old Xeon E3 hardware.
I understand that we've had lots of security issues in the past decade and several steps have been taken to protect and isolate the memory inside all virtualization platforms. When I first built my E3-1230 Xeon system for homelab, VM to VM iperfs were like 20Gb/s. Nowadays that's significantly slowed down.
Anyway...I just find it hard to believe that with as superior a computing platform as Epyc is, that the single-threaded iperf is so much slwoer than 12-year-old entry level Intel CPUs.
Maybe I should load VMWare on this system and see how it does and report back. Same hardware, but different hypervisor, and compare notes.
-
@nicols said in Epyc VM to VM networking slow:
@JamesG said in Epyc VM to VM networking slow:
@olivierlambert With a billion threads.
nope, Win 10 VM with 4 or 8 VCPU and 8GB RAM
But, with billion threads in Linux VM, speed increases up to 8 threats, then it drops again.This is with 1 and 16 threads:
-
@nicols said in Epyc VM to VM networking slow:
@nicols said in Epyc VM to VM networking slow:
@JamesG said in Epyc VM to VM networking slow:
@olivierlambert With a billion threads.
nope, Win 10 VM with 4 or 8 VCPU and 8GB RAM
But, with billion threads in Linux VM, speed increases up to 8 threats, then it drops again.This is with 1 and 16 threads:
also, this:
https://nextcloud.openit.hr/s/CptZpTt4jbWcRPX
is cpu load on host during 2 VM linux doing 16 thread iperf (with cumulative speed of pathetic 4 Gbit/sec).
It seems way to high for this kind of job? -
@olivierlambert said in Epyc VM to VM networking slow:
Or maybe AMD CPU are a lot slower with memcpy()?
Has anyone reviewed this issue? Is there a way to test with a newer version of
glibc? -
@nicols give me your VM specs and I'll run the exact same tests. vCPU, RAM, anything else relevant.
-
@Danp That's interesting...
-
@JamesG said in Epyc VM to VM networking slow:
@nicols give me your VM specs and I'll run the exact same tests. vCPU, RAM, anything else relevant.
Debian 12: 16 VCPU, 2GB RAM
Windows 10 pro: 16 VCPU, 8GB RAM, citrix vmtols 9.3.1On Linux Debian there is no much difference between 8 and 16 VCPU
On Windows 10, 8 VCPU: 16 Gbit/sec, 16 VCPU: 21 Gbit/sec
