Epyc VM to VM networking slow

bullerwins

@probain said in Epyc VM to VM networking slow:

I ran these tests now that newer updates have been released for 8.3-beta.
Results are as below:

iperf-sender -> iperf-receiver: 5.06Gbit/s
iperf-sender -> iperf-receiver -P4: 7.53Gbit/s
host -> iperf-receiver: 7.83Gbit/s
host -> iperf-receiver -P4: 13.0Gbit/s

Host (dom0):

CPU: AMD EPYC 7302P
Sockets: 1
RAM: 6.59GB (dom0) / 112GB for VMs
MotherBoard: H12SSL-i
NIC: X540-AT2 (rev 01)

xl info -n

host                   : xcp
release                : 4.19.0+1
version                : #1 SMP Mon Jun 24 17:20:04 CEST 2024
machine                : x86_64
nr_cpus                : 32
max_cpu_id             : 31
nr_nodes               : 1
cores_per_socket       : 16
threads_per_core       : 2
cpu_mhz                : 2999.997
hw_caps                : 178bf3ff:7ed8320b:2e500800:244037ff:0000000f:219c91a9:00400004:00000780
virt_caps              : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2
total_memory           : 114549
free_memory            : 62685
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
cpu_topology           :
cpu:    core    socket     node
  0:       0        0        0
  1:       0        0        0
  2:       1        0        0
  3:       1        0        0
  4:       4        0        0
  5:       4        0        0
  6:       5        0        0
  7:       5        0        0
  8:       8        0        0
  9:       8        0        0
 10:       9        0        0
 11:       9        0        0
 12:      12        0        0
 13:      12        0        0
 14:      13        0        0
 15:      13        0        0
 16:      16        0        0
 17:      16        0        0
 18:      17        0        0
 19:      17        0        0
 20:      20        0        0
 21:      20        0        0
 22:      21        0        0
 23:      21        0        0
 24:      24        0        0
 25:      24        0        0
 26:      25        0        0
 27:      25        0        0
 28:      28        0        0
 29:      28        0        0
 30:      29        0        0
 31:      29        0        0
device topology        :
device           node
No device topology data available
numa_info              :
node:    memsize    memfree    distances
   0:    115955      62685      10
xen_major              : 4
xen_minor              : 17
xen_extra              : .4-3
xen_version            : 4.17.4-3
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : d530627aaa9b, pq 7587628e7d91
xen_commandline        : dom0_mem=6752M,max:6752M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
cc_compiler            : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
cc_compile_by          : mockbuild
cc_compile_domain      : [unknown]
cc_compile_date        : Thu Jun 20 18:17:10 CEST 2024
build_id               : 9497a1ec7ec99f5075421732b0ec37781ba739a9
xend_config_format     : 4

VMs - Sender and Receiver

Distro: Ubuntu 24.04
Kernel: 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC
vCPUs: 32
RAM: 4GB

have you tested without the 8.3 updates? The results seem still low. Any improvement?

probain

@bullerwins Unfortunately I didn't. In hindsight I wish I did.

JamesG

These latest 8.3 update speeds are still slower than a 13 year-old Xeon E3 1230.

Seneram

I can unfortunately share that from ongoing ticket investigations in this, It is far more deeply rooted than something that a patch of going from one major kernel to another will "just fix" There are multiple leads being investigated and multiple vendors involved.

olivierlambert

I'd like to check something to see if it's coherent with our tests, by using 2x similar VMs (4vCPUs/4G RAM):

iperf monothread speed on a "fresh" Debian 10 install (4.19 kernel)
the same bench with 5.10.0 kernel from backports (add deb http://deb.debian.org/debian buster-backports main contrib non-free in your source list and then apt install linux-image-5.10, don't forget to reboot to be on that kernel)

Do you see a performance diff between those?

john.c

@olivierlambert said in Epyc VM to VM networking slow:

I'd like to check something to see if it's coherent with our tests, by using 2x similar VMs (4vCPUs/4G RAM):

iperf monothread speed on a "fresh" Debian 10 install (4.19 kernel)

the same bench with 5.10.0 kernel from backports (add deb http://deb.debian.org/debian buster-backports main contrib non-free in your source list and then apt install linux-image-5.10, don't forget to reboot to be on that kernel)

Do you see a performance diff between those?

FYI, getting a Debian 10 backports or non-backports packages are going to now be extremely difficult. The Debian Linux 10 LTS has reached EOL. Now currently in ELTS from the beginning of this month until 30/06/2029, though covering only a subset of the packages.

https://www.debian.org/News/2024/20240615

olivierlambert

I had no issue to test it quickly. The thing is for the sake of testing and try to identify a potential regression, not for production usage or whatnot.

olivierlambert

I identified a specific regression in a Debian kernel build since 5.10, we are investigating the "why" (starting from this exact build: https://snapshot.debian.org/package/linux/5.10.92-1/)

probain

@olivierlambert
Would it be possible for you to either offer a ISO to download? Or maybe seed one? I really want to help test this. But I'm getting lost with how Debian provides their legacy images and this jig-boo (intentionally misspelled)

G-Ork

May someone could graph their vm.
Comparing a slow vm with a full speed could bring light into darknes.

https://www.brendangregg.com/Articles/Linux_Kernel_Performance_Flame_Graphs.pdf

olivierlambert

@probain Debian 10 is available in the XOA Hub.

probain

@olivierlambert
I wasn't aware. Thanks! Downloading for doing a test, right away

Test done:

				Run1	Run2	Run3
Sender:   Debian10 kernel 4.19	4.81Gb	4.81Gb	4.83Gb
Reveiver: Debian10 kernel 4.19

Sender:   Debian10 kernel 5.10	5.13Gb	5.02Gb	5.12Gb
Reveiver: Debian10 kernel 4.19

Sender:   Debian10 kernel 5.10	4.98Gb	5.02Gb	4.97Gb
Reveiver: Debian10 kernel 5.10

sender runs 'iperf -c <IP-to-receiver> -t 60'

Kernel 4.19 = 4.19.0-6-amd64
Kernel 5.10 = 5.10.0-0.deb10.24-amd64

CPU 4 cores (AMD EPYC 7302P)
RAM 4GB

Created from XOA-hub

olivierlambert

Thanks @probain , now can you try iperf -s in the Dom0 and iperf -c <IP dom0> in the Debian guest?

probain

@olivierlambert
vm -> dom0 results in "no route to host": firewall?

Results will be shown for dom0 -> vm. Listed by each kernel installed on vm.

Just as earlier. VM is installed via XOA Hub, with 4 CPU and 4GB RAM. Host CPU running on AMD EPYC 7302P.

VM kernel ver.	Run1	Run2	Run3
kernel 4.19.0	8.47Gb	8.82Gb	8.43Gb
kernel 5.10.0	7.12Gb	7.07Gb	7.11Gb

olivierlambert

yes disable the fw first (only in a testing lab obviously) with iptables -F

probain

@olivierlambert how do I restore the iptables again afterwards? Other than reboot ofc

Update: Tests done

vm -> dom0

		Run1	Run2	Run3
kernel 4.19.0	5.84Gb	5.77Gb	5.85Gb
kernel 5.10.0	1.25Gb	1.26Gb	1.28

Specs are just as previous post.

olivierlambert

Thanks so at least it confirms something we are also spotting in here. We found the exact commit.

G-Ork

Here are the opterons with dropped firewall:

source	destination	OS	Kernel	Speed Average
vm	dom	debian 10	4.19.0-6-amd64	6.57 Gbits/sec
dom	vm	debian 10	4.19.0-6-amd64	1.79 Gbits/sec
vm	dom	truenas	6.6.20	2.01 Gbits/sec
dom	vm	truenas	6.6.20	1.82 Gbits/sec
host	vm	debian 10	4.19.0-6-amd64	5.32 Gbits/sec
host	vm	truenas	6.6.20	1.92 Gbits/sec
host	dom	debian	4.19.0+1	8.97 Gbits/sec

G-Ork

@probain said in Epyc VM to VM networking slow:

I restore the iptables again afterwards? Other than reboot

this worked for me

action	command
save	iptables-save > firewall.conf
flush	iptables -F
restore	cat firewall.conf \| iptables-restore

sluflyer06

Here's a little test I just ran between VM's over SMB on my Threadripper 7960x build on a Supermicro H13SRA-TF motherboard, def not too bad, these VM's are on different SR's.