Epyc VM to VM networking slow

nicols

I have Dual socket Supermicro server with 2 x EPYC 7443.
Fresh installation of XCP-ng 8,2, all updates applied.

Inter-server networking is slow. I am getting max 4 Gbits/s:

root@debilan-12-1:~# iperf3 -c 10.10.10.202 
Connecting to host 10.10.10.202, port 5201
[  5] local 10.10.10.201 port 54208 connected to 10.10.10.202 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   430 MBytes  3.61 Gbits/sec   45   1021 KBytes       
[  5]   1.00-2.00   sec   344 MBytes  2.88 Gbits/sec   45    899 KBytes       
[  5]   2.00-3.00   sec   436 MBytes  3.66 Gbits/sec   90    837 KBytes       
[  5]   3.00-4.00   sec   444 MBytes  3.72 Gbits/sec   13    868 KBytes       
[  5]   4.00-5.00   sec   405 MBytes  3.40 Gbits/sec   45    848 KBytes       
[  5]   5.00-6.00   sec   459 MBytes  3.85 Gbits/sec   68    725 KBytes       
[  5]   6.00-7.00   sec   401 MBytes  3.37 Gbits/sec    0   1011 KBytes       
[  5]   7.00-8.00   sec   325 MBytes  2.73 Gbits/sec    0   1.12 MBytes       
[  5]   8.00-9.00   sec   360 MBytes  3.02 Gbits/sec   45    994 KBytes       
[  5]   9.00-10.00  sec   420 MBytes  3.52 Gbits/sec  119    588 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.93 GBytes  3.38 Gbits/sec  470             sender
[  5]   0.00-10.00  sec  3.93 GBytes  3.38 Gbits/sec                  receiver

iperf Done.

I have tried this (didn't help):

Advanced -> CPU Configuration -> Global C-state Control: Disabled
Advanced -> North Bridge -> Determinism Slider: Performance

Also, tried various bios stuff, nothing.
I tried using real network or single server private network, results are the same.
VM-s are freshly installed debian 12, i tried changing VCPU / RAM, nothing helps.
Increased Dom0 memory, doesn't help

Also, VM to outside server: i am getting asymmetric network speed, incoming is full 10Gbps, outgoing about 2 Gbits/sec.
If i try iperf directly from Dom0 to outside, i am getting normal speed 9.5 / 9.5 Gbits/sec.

What am I missing?
New Intel based servers doesn't have that problem.

olivierlambert

Hi,

There's many topics opened on this, you should check Also, please use Iperf with multiple process at once, it's more relevant

Are those VMs in the same physical host?

nicols

@olivierlambert said in Epyc VM to VM networking slow:

Hi,

There's many topics opened on this, you should check Also, please use Iperf with multiple process at once, it's more relevant

Are those VMs in the same physical host?

Yes, all VMs on same physical host.
There are many topics on EPYC, but nothing specific on netwel performance.
Multiple iperf3 processes don't give more speed.

This is on identical server with proxmox 7.4 with default debian 12 VM installation (qemu/virtio):

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-600.00 sec  1.48 TBytes  21.7 Gbits/sec   12             sender
[  5]   0.00-600.01 sec  1.48 TBytes  21.7 Gbits/sec                  receiver

olivierlambert

There's many topics on network perf, especially in the last 2 or 3 months:

In the last one, I made a bench on some test Ryzen machines, reaching 20GBit+ between VMs on 2 different hosts I can probably do something similar on my 2nd gen EPYC in prod (I did not test on the same host IIRC, I'll do that later).

Also, please compare apples to apples, KVM+virtio is using DMA without any decent security, so I would expect it to be like no virtualization in the middle (ie closer to bare metal).

nicols

@olivierlambert said in Epyc VM to VM networking slow:

There's many topics on network perf, especially in the last 2 or 3 months:

https://xcp-ng.org/forum/topic/7775/poor-network-speeds-with-40g-56g-100g-network-cards

https://xcp-ng.org/forum/topic/7714/expected-10gbit-performance

https://xcp-ng.org/forum/topic/7723/vms-network-speed-limited-to-10gbit-s

https://xcp-ng.org/forum/topic/7708/problems-with-xcp-ng-bond-20gb-less-performance-between-vm-s

In the last one, I made a bench on some test Ryzen machines, reaching 20GBit+ between VMs on 2 different hosts I can probably do something similar on my 2nd gen EPYC in prod (I did not test on the same host IIRC, I'll do that later).

Also, please compare apples to apples, KVM+virtio is using DMA without any decent security, so I would expect it to be like no virtualization in the middle (ie closer to bare metal).

Thanks for the links, some of this is somehow relevant to me, but not all the way. Most of this test are cross-host network.

Ok, please ignore my comparison with proxmox/kvm, i know it is not "apples to apples"

I Just did a test on dual socket Supermicro server with Intel E5-2630L v2 CPU, two VMs on single host are getting 14 Gbit/sec. (2 Debian 11 VM, 2 VCPU, 2GB RAM).

So, where is the catch? This new EPYC is far more powerfull, but results are 3-4 times slower?

Anybody have tests with EPYC servers VM-to-VM network speed on single server?
Is this some kind of EPYC/xen incompatibility?
Maybe we should ask Supermicro about that?

olivierlambert

That's indeed weird. If it's between 2x VMs on the same host, that shouldn't touch any physical NIC in the path. Let me run some tests when I'm at home to see if I can reproduce.

olivierlambert

I wonder if you have a CPU that is boosting correctly, because even in the worst case scenario (no NIC offload), you should have something closer to 10G than 2G

Also, can you check if you are all firmware up to date?

Finally, can you try with many iperf3 threads, like 14 or 16 and check the result?

nicols

@olivierlambert said in Epyc VM to VM networking slow:

I wonder if you have a CPU that is boosting correctly, because even in the worst case scenario (no NIC offload), you should have something closer to 10G than 2G

Also, can you check if you are all firmware up to date?

Finally, can you try with many iperf3 threads, like 14 or 16 and check the result?

All firmwares are up to date.

I tried many iperf3 threads combination, including 14 and 16, no any speed gain. I Have tried running VMs with 1,2,4,8 and 16 VCPUs,
and various amounts of RAM.

There is no NIC involved, i am testing with single-server private network.
Nevertheless, result is same on "real" network or single-server private network.
In both cases, like you said earlier, there is no actual NIC (drivers, offload ...) involved, because this two VMs are on same physical host.

I am a bit lost, because i tried all usual and common sense stuff in BIOS... I think i didn't try disabling SMT, so i will try it now.

If you (or anybody reading this) have some EPYC server (single or dual socket), please test network speed between two VMs running on single physical host. I would like to see how my test compare with MBOs from another manufacturers.

olivierlambert

I have a pretty large diff when having far more process iperf3 process, it's weird you don't (eg I can reach 16G+ vs 8G on the same host with 14 iperf threads vs 1)

nicols

@olivierlambert said in Epyc VM to VM networking slow:

I have a pretty large diff when having far more process iperf3 process, it's weird you don't (eg I can reach 16G+ vs 8G on the same host with 14 iperf threads vs 1)

iperf2:

root@debilan-12-1:~# iperf -c 10.33.66.135
...
[  1] 0.0000-10.0191 sec  3.25 GBytes  2.78 Gbits/sec

root@debilan-12-1:~# iperf -c 10.33.66.135 -P16
...
[SUM] 0.0000-10.0422 sec  5.19 GBytes  4.44 Gbits/sec

iperf3:

root@debilan-12-1:~# iperf3 -c 10.33.66.135
...
[  5]   0.00-10.00  sec  3.91 GBytes  3.36 Gbits/sec  180             sender
[  5]   0.00-10.00  sec  3.91 GBytes  3.36 Gbits/sec                  receiver

root@debilan-12-1:~# iperf3 -c 10.33.66.135 -P16
...
[SUM]   0.00-10.01  sec  3.83 GBytes  3.29 Gbits/sec    0             sender
[SUM]   0.00-10.01  sec  3.83 GBytes  3.29 Gbits/sec                  receiver

olivierlambert

Try iperf3 with 2 threads

nicols

@olivierlambert said in Epyc VM to VM networking slow:

Try iperf3 with 2 threads

[SUM]   0.00-10.00  sec  4.78 GBytes  4.10 Gbits/sec  318             sender
[SUM]   0.00-10.00  sec  4.78 GBytes  4.10 Gbits/sec                  receiver

olivierlambert

In any case, what's going on: since you are sending packets that are not leaving the host, it's not using any NIC offload. And since you are using Xen to isolate between guest, it's up to basically memcopy() to deal with all the packets you send to the other guest.

So you are limited by the capacity of your Dom0 to make many memcopy() per seconds. There's no fastpath in this situation.

What could make it at only few GBits/s? Could be no turbo, or high latency with the RAM or many other aspects. Benchmarks I have on my host isn't that bad than yours though (especially it scales with more iperf3 processes in parallel).

I would check if the CPU is able to turbo (thermal limitations? RAM speed? latency?). It's hard to get a precise answer since there's many parameters.

nicols

@olivierlambert said in Epyc VM to VM networking slow:

In any case, what's going on: since you are sending packets that are not leaving the host, it's not using any NIC offload. And since you are using Xen to isolate between guest, it's up to basically memcopy() to deal with all the packets you send to the other guest.

So you are limited by the capacity of your Dom0 to make many memcopy() per seconds. There's no fastpath in this situation.

What could make it at only few GBits/s? Could be no turbo, or high latency with the RAM or many other aspects. Benchmarks I have on my host isn't that bad than yours though (especially it scales with more iperf3 processes in parallel).

I would check if the CPU is able to turbo (thermal limitations? RAM speed? latency?). It's hard to get a precise answer since there's many parameters.

On dom0, i have normal speed of 9.5 / 9.5 Gbits/sec to another physical server, via 10G switch.

olivierlambert

Again, you need to understand: sending packets to another host is very different (since it's going to the NIC). So the Dom0 speed has nothing to do with what's happening behind the scene between regular guests on the same physical host.

nicols

@olivierlambert said in Epyc VM to VM networking slow:

I have a pretty large diff when having far more process iperf3 process, it's weird you don't (eg I can reach 16G+ vs 8G on the same host with 14 iperf threads vs 1)

is this 16gbit/sec between two VMs on same physical host?
if yes, which HW is it ?

olivierlambert

Yes, between 2x VMs on the same host. Ryzen 5 7600.

nicols

@olivierlambert said in Epyc VM to VM networking slow:

Yes, between 2x VMs on the same host. Ryzen 5 7600.

We did some more tests and BIOS tweaking.

We are getting max. 5 Gbps VM to VM trafic on Linux.
But, we also did tests with Windows OS, with multiple iperf threads we are able to achieve up to 18 Gbps VM to VM on same physical host.

So, where is the catch?

Anybody here with EPYC server willing to do some tests?

Best Regards!

olivierlambert

You mean 2x Windows VMs (which versions?) on the same EPYC host running XCP-ng, you got 18G with multiple threads (how many?) vs Linux in the same configuration (which kernel/distro? same VM cores & memory?), right?

nicols

@olivierlambert said in Epyc VM to VM networking slow:

You mean 2x Windows VMs (which versions?) on the same EPYC host running XCP-ng, you got 18G with multiple threads (how many?) vs Linux in the same configuration (which kernel/distro? same VM cores & memory?), right?

yes, 2 x win10 pro, on same host, 8-12 multiple threads vs debian 12, same cpu/memory configurations, also 8-12 multiple threads.