Epyc VM to VM networking slow
-
In any case, what's going on: since you are sending packets that are not leaving the host, it's not using any NIC offload. And since you are using Xen to isolate between guest, it's up to basically
memcopy()to deal with all the packets you send to the other guest.So you are limited by the capacity of your Dom0 to make many
memcopy()per seconds. There's no fastpath in this situation.What could make it at only few GBits/s? Could be no turbo, or high latency with the RAM or many other aspects. Benchmarks I have on my host isn't that bad than yours though (especially it scales with more iperf3 processes in parallel).
I would check if the CPU is able to turbo (thermal limitations? RAM speed? latency?). It's hard to get a precise answer since there's many parameters.
-
@olivierlambert said in Epyc VM to VM networking slow:
In any case, what's going on: since you are sending packets that are not leaving the host, it's not using any NIC offload. And since you are using Xen to isolate between guest, it's up to basically
memcopy()to deal with all the packets you send to the other guest.So you are limited by the capacity of your Dom0 to make many
memcopy()per seconds. There's no fastpath in this situation.What could make it at only few GBits/s? Could be no turbo, or high latency with the RAM or many other aspects. Benchmarks I have on my host isn't that bad than yours though (especially it scales with more iperf3 processes in parallel).
I would check if the CPU is able to turbo (thermal limitations? RAM speed? latency?). It's hard to get a precise answer since there's many parameters.
On dom0, i have normal speed of 9.5 / 9.5 Gbits/sec to another physical server, via 10G switch.
-
Again, you need to understand: sending packets to another host is very different (since it's going to the NIC). So the Dom0 speed has nothing to do with what's happening behind the scene between regular guests on the same physical host.
-
@olivierlambert said in Epyc VM to VM networking slow:
I have a pretty large diff when having far more process iperf3 process, it's weird you don't
(eg I can reach 16G+ vs 8G on the same host with 14 iperf threads vs 1)is this 16gbit/sec between two VMs on same physical host?
if yes, which HW is it ? -
Yes, between 2x VMs on the same host. Ryzen 5 7600.
-
@olivierlambert said in Epyc VM to VM networking slow:
Yes, between 2x VMs on the same host. Ryzen 5 7600.
We did some more tests and BIOS tweaking.
We are getting max. 5 Gbps VM to VM trafic on Linux.
But, we also did tests with Windows OS, with multiple iperf threads we are able to achieve up to 18 Gbps VM to VM on same physical host.So, where is the catch?
Anybody here with EPYC server willing to do some tests?
Best Regards!
-
You mean 2x Windows VMs (which versions?) on the same EPYC host running XCP-ng, you got 18G with multiple threads (how many?) vs Linux in the same configuration (which kernel/distro? same VM cores & memory?), right?
-
@olivierlambert said in Epyc VM to VM networking slow:
You mean 2x Windows VMs (which versions?) on the same EPYC host running XCP-ng, you got 18G with multiple threads (how many?) vs Linux in the same configuration (which kernel/distro? same VM cores & memory?), right?
yes, 2 x win10 pro, on same host, 8-12 multiple threads vs debian 12, same cpu/memory configurations, also 8-12 multiple threads.
-
Do you have tools and PV drivers installed in Windows? Which version of the tools?
-
@olivierlambert said in Epyc VM to VM networking slow:
Do you have tools and PV drivers installed in Windows? Which version of the tools?
citrix vm tools 9.3.1 are instaleld
-
The different is really weird between Windows and Linux
In theory, you should have a similar result on your Linux VMs vs Windows for the network speed.What is the template you are using for your Debian VMs?
-
I cited this performance problem on Epyc early this year when I was building a cluster for a company. I couldn't get any VM to VM performance, so SR-IOV to the rescue which getting that to work was a mess.
I've got Supermicro H12SSL boards with 7302p processors. I don't have all the memory slots populated so memory throughput isn't as good as it could be, but it sounds like other people are having a similar experience to me.
Currently I'm on the latest 8.3 patched. NICs are Mellanox ConnextX4-LX (not that they matter in guest to guest traffic).
That said...My nearly 12 year-old Xeon E3-1230v2 server crushes the Epyc in guest to guest traffic. Quick test:
Xeon E3-1230v2 iperf3 with 4 threads (Debian guests):
[SUM] 0.00-10.00 sec 15.2 GBytes 13.1 Gbits/sec 0 sender
[SUM] 0.00-10.04 sec 15.2 GBytes 13.0 Gbits/sec receiverEpyc 7302p iperf3 with 4 threads (Debian guests):
[SUM] 0.00-10.00 sec 7.66 GBytes 6.58 Gbits/sec 870 sender
[SUM] 0.00-10.03 sec 7.65 GBytes 6.55 Gbits/sec receiverInteresting to note the retries on the Epyc transfer.
These two CPU's have similar single-thread performance and similar clock-rate. All the C and P states on the Epyc systems have been tweaked.
Guest to Guest traffic is seemingly really impaired compared to the old Xeons.
-
@JamesG can you test between 2x Windows guest? (and/or between 2x BSD guests)
It's maybe a weird Linux thing
What's the kernel version used in your Debian? -
@JamesG said in Epyc VM to VM networking slow:
I've got Supermicro H12SSL boards with 7302p processors. I don't have all the memory slots populated so memory throughput isn't as good as it could be, but it sounds like other people are having a similar experience to me.
That said...My nearly 12 year-old Xeon E3-1230v2 server crushes the Epyc in guest to guest traffic. Quick test:
Guest to Guest traffic is seemingly really impaired compared to the old Xeons.
Finally someone with similar experience, i was starting to think i am a bit crazy
There was update of xen hypervisor today, i will do another test....
-
Here's my thread from earlier this year:
https://xcp-ng.org/forum/topic/6916/tracking-down-poor-network-performance/11
Here's a referenced thread in that thread:
I'd be curious how this works in VMWare.
-
I cloned a Win10 VM...Win10 to Win10, same performance as Debian.
This is most definitely something within the networking infrastructure of Xen.
-
This is already different result than @nicols
-
@olivierlambert Not really...
In Nicols first post, a single threaded iperf got 3.38Gb/s.
Single threaded for me was 3.32Gb/s.With two threads I get 5.19Gb/s
With 20 threads I cap off at 7.02Gb/sThis performance is about the same with Windows VM's and Debian VM's. So it's not a guest OS issue. It's something in the hypervisor.
-
A note...
I'm running a single 16 core, 32 thread second-gen Epyc.
Nicols is running a dual proc, 24 core, 48 thread third-gen Epyc.My base clock rate is 3.0Ghz. His is 2.9Ghz.
The improved caching and memory handling in the third-gen Epyc should be behaving better than my second gen CPU's, but generally speaking, our performance seems to be the same.
-
He said he could reach 18gbits between 2x Windows VM, if I remember correctly.
I wonder about the guest kernel too (Debian 11 vs 12)
