Epyc VM to VM networking slow

nicols

@olivierlambert said in Epyc VM to VM networking slow:

Hi,

There's many topics opened on this, you should check Also, please use Iperf with multiple process at once, it's more relevant

Are those VMs in the same physical host?

Yes, all VMs on same physical host.
There are many topics on EPYC, but nothing specific on netwel performance.
Multiple iperf3 processes don't give more speed.

This is on identical server with proxmox 7.4 with default debian 12 VM installation (qemu/virtio):

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-600.00 sec  1.48 TBytes  21.7 Gbits/sec   12             sender
[  5]   0.00-600.01 sec  1.48 TBytes  21.7 Gbits/sec                  receiver

olivierlambert

There's many topics on network perf, especially in the last 2 or 3 months:

In the last one, I made a bench on some test Ryzen machines, reaching 20GBit+ between VMs on 2 different hosts I can probably do something similar on my 2nd gen EPYC in prod (I did not test on the same host IIRC, I'll do that later).

Also, please compare apples to apples, KVM+virtio is using DMA without any decent security, so I would expect it to be like no virtualization in the middle (ie closer to bare metal).

nicols

@olivierlambert said in Epyc VM to VM networking slow:

There's many topics on network perf, especially in the last 2 or 3 months:

https://xcp-ng.org/forum/topic/7775/poor-network-speeds-with-40g-56g-100g-network-cards

https://xcp-ng.org/forum/topic/7714/expected-10gbit-performance

https://xcp-ng.org/forum/topic/7723/vms-network-speed-limited-to-10gbit-s

https://xcp-ng.org/forum/topic/7708/problems-with-xcp-ng-bond-20gb-less-performance-between-vm-s

In the last one, I made a bench on some test Ryzen machines, reaching 20GBit+ between VMs on 2 different hosts I can probably do something similar on my 2nd gen EPYC in prod (I did not test on the same host IIRC, I'll do that later).

Also, please compare apples to apples, KVM+virtio is using DMA without any decent security, so I would expect it to be like no virtualization in the middle (ie closer to bare metal).

Thanks for the links, some of this is somehow relevant to me, but not all the way. Most of this test are cross-host network.

Ok, please ignore my comparison with proxmox/kvm, i know it is not "apples to apples"

I Just did a test on dual socket Supermicro server with Intel E5-2630L v2 CPU, two VMs on single host are getting 14 Gbit/sec. (2 Debian 11 VM, 2 VCPU, 2GB RAM).

So, where is the catch? This new EPYC is far more powerfull, but results are 3-4 times slower?

Anybody have tests with EPYC servers VM-to-VM network speed on single server?
Is this some kind of EPYC/xen incompatibility?
Maybe we should ask Supermicro about that?

olivierlambert

That's indeed weird. If it's between 2x VMs on the same host, that shouldn't touch any physical NIC in the path. Let me run some tests when I'm at home to see if I can reproduce.

olivierlambert

I wonder if you have a CPU that is boosting correctly, because even in the worst case scenario (no NIC offload), you should have something closer to 10G than 2G

Also, can you check if you are all firmware up to date?

Finally, can you try with many iperf3 threads, like 14 or 16 and check the result?

nicols

@olivierlambert said in Epyc VM to VM networking slow:

I wonder if you have a CPU that is boosting correctly, because even in the worst case scenario (no NIC offload), you should have something closer to 10G than 2G

Also, can you check if you are all firmware up to date?

Finally, can you try with many iperf3 threads, like 14 or 16 and check the result?

All firmwares are up to date.

I tried many iperf3 threads combination, including 14 and 16, no any speed gain. I Have tried running VMs with 1,2,4,8 and 16 VCPUs,
and various amounts of RAM.

There is no NIC involved, i am testing with single-server private network.
Nevertheless, result is same on "real" network or single-server private network.
In both cases, like you said earlier, there is no actual NIC (drivers, offload ...) involved, because this two VMs are on same physical host.

I am a bit lost, because i tried all usual and common sense stuff in BIOS... I think i didn't try disabling SMT, so i will try it now.

If you (or anybody reading this) have some EPYC server (single or dual socket), please test network speed between two VMs running on single physical host. I would like to see how my test compare with MBOs from another manufacturers.

olivierlambert

I have a pretty large diff when having far more process iperf3 process, it's weird you don't (eg I can reach 16G+ vs 8G on the same host with 14 iperf threads vs 1)

nicols

@olivierlambert said in Epyc VM to VM networking slow:

I have a pretty large diff when having far more process iperf3 process, it's weird you don't (eg I can reach 16G+ vs 8G on the same host with 14 iperf threads vs 1)

iperf2:

root@debilan-12-1:~# iperf -c 10.33.66.135
...
[  1] 0.0000-10.0191 sec  3.25 GBytes  2.78 Gbits/sec

root@debilan-12-1:~# iperf -c 10.33.66.135 -P16
...
[SUM] 0.0000-10.0422 sec  5.19 GBytes  4.44 Gbits/sec

iperf3:

root@debilan-12-1:~# iperf3 -c 10.33.66.135
...
[  5]   0.00-10.00  sec  3.91 GBytes  3.36 Gbits/sec  180             sender
[  5]   0.00-10.00  sec  3.91 GBytes  3.36 Gbits/sec                  receiver

root@debilan-12-1:~# iperf3 -c 10.33.66.135 -P16
...
[SUM]   0.00-10.01  sec  3.83 GBytes  3.29 Gbits/sec    0             sender
[SUM]   0.00-10.01  sec  3.83 GBytes  3.29 Gbits/sec                  receiver

olivierlambert

Try iperf3 with 2 threads

nicols

@olivierlambert said in Epyc VM to VM networking slow:

Try iperf3 with 2 threads

[SUM]   0.00-10.00  sec  4.78 GBytes  4.10 Gbits/sec  318             sender
[SUM]   0.00-10.00  sec  4.78 GBytes  4.10 Gbits/sec                  receiver

olivierlambert

In any case, what's going on: since you are sending packets that are not leaving the host, it's not using any NIC offload. And since you are using Xen to isolate between guest, it's up to basically memcopy() to deal with all the packets you send to the other guest.

So you are limited by the capacity of your Dom0 to make many memcopy() per seconds. There's no fastpath in this situation.

What could make it at only few GBits/s? Could be no turbo, or high latency with the RAM or many other aspects. Benchmarks I have on my host isn't that bad than yours though (especially it scales with more iperf3 processes in parallel).

I would check if the CPU is able to turbo (thermal limitations? RAM speed? latency?). It's hard to get a precise answer since there's many parameters.

nicols

@olivierlambert said in Epyc VM to VM networking slow:

In any case, what's going on: since you are sending packets that are not leaving the host, it's not using any NIC offload. And since you are using Xen to isolate between guest, it's up to basically memcopy() to deal with all the packets you send to the other guest.

So you are limited by the capacity of your Dom0 to make many memcopy() per seconds. There's no fastpath in this situation.

What could make it at only few GBits/s? Could be no turbo, or high latency with the RAM or many other aspects. Benchmarks I have on my host isn't that bad than yours though (especially it scales with more iperf3 processes in parallel).

I would check if the CPU is able to turbo (thermal limitations? RAM speed? latency?). It's hard to get a precise answer since there's many parameters.

On dom0, i have normal speed of 9.5 / 9.5 Gbits/sec to another physical server, via 10G switch.

olivierlambert

Again, you need to understand: sending packets to another host is very different (since it's going to the NIC). So the Dom0 speed has nothing to do with what's happening behind the scene between regular guests on the same physical host.

nicols

@olivierlambert said in Epyc VM to VM networking slow:

I have a pretty large diff when having far more process iperf3 process, it's weird you don't (eg I can reach 16G+ vs 8G on the same host with 14 iperf threads vs 1)

is this 16gbit/sec between two VMs on same physical host?
if yes, which HW is it ?

olivierlambert

Yes, between 2x VMs on the same host. Ryzen 5 7600.

nicols

@olivierlambert said in Epyc VM to VM networking slow:

Yes, between 2x VMs on the same host. Ryzen 5 7600.

We did some more tests and BIOS tweaking.

We are getting max. 5 Gbps VM to VM trafic on Linux.
But, we also did tests with Windows OS, with multiple iperf threads we are able to achieve up to 18 Gbps VM to VM on same physical host.

So, where is the catch?

Anybody here with EPYC server willing to do some tests?

Best Regards!

olivierlambert

You mean 2x Windows VMs (which versions?) on the same EPYC host running XCP-ng, you got 18G with multiple threads (how many?) vs Linux in the same configuration (which kernel/distro? same VM cores & memory?), right?

nicols

@olivierlambert said in Epyc VM to VM networking slow:

You mean 2x Windows VMs (which versions?) on the same EPYC host running XCP-ng, you got 18G with multiple threads (how many?) vs Linux in the same configuration (which kernel/distro? same VM cores & memory?), right?

yes, 2 x win10 pro, on same host, 8-12 multiple threads vs debian 12, same cpu/memory configurations, also 8-12 multiple threads.

olivierlambert

Do you have tools and PV drivers installed in Windows? Which version of the tools?

nicols

@olivierlambert said in Epyc VM to VM networking slow:

Do you have tools and PV drivers installed in Windows? Which version of the tools?

citrix vm tools 9.3.1 are instaleld