Epyc VM to VM networking slow

[probain]

I ran these tests now that newer updates have been released for 8.3-beta.
Results are as below:

• iperf-sender -> iperf-receiver: 5.06Gbit/s
• iperf-sender -> iperf-receiver -P4: 7.53Gbit/s
• host -> iperf-receiver: 7.83Gbit/s
• host -> iperf-receiver -P4: 13.0Gbit/s

Host (dom0):

• CPU: AMD EPYC 7302P
• Sockets: 1
• RAM: 6.59GB (dom0) / 112GB for VMs
• MotherBoard: H12SSL-i
• NIC: X540-AT2 (rev 01)

xl info -n

host                   : xcp
release                : 4.19.0+1
version                : #1 SMP Mon Jun 24 17:20:04 CEST 2024
machine                : x86_64
nr_cpus                : 32
max_cpu_id             : 31
nr_nodes               : 1
cores_per_socket       : 16
threads_per_core       : 2
cpu_mhz                : 2999.997
hw_caps                : 178bf3ff:7ed8320b:2e500800:244037ff:0000000f:219c91a9:00400004:00000780
virt_caps              : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2
total_memory           : 114549
free_memory            : 62685
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
cpu_topology           :
cpu:    core    socket     node
  0:       0        0        0
  1:       0        0        0
  2:       1        0        0
  3:       1        0        0
  4:       4        0        0
  5:       4        0        0
  6:       5        0        0
  7:       5        0        0
  8:       8        0        0
  9:       8        0        0
 10:       9        0        0
 11:       9        0        0
 12:      12        0        0
 13:      12        0        0
 14:      13        0        0
 15:      13        0        0
 16:      16        0        0
 17:      16        0        0
 18:      17        0        0
 19:      17        0        0
 20:      20        0        0
 21:      20        0        0
 22:      21        0        0
 23:      21        0        0
 24:      24        0        0
 25:      24        0        0
 26:      25        0        0
 27:      25        0        0
 28:      28        0        0
 29:      28        0        0
 30:      29        0        0
 31:      29        0        0
device topology        :
device           node
No device topology data available
numa_info              :
node:    memsize    memfree    distances
   0:    115955      62685      10
xen_major              : 4
xen_minor              : 17
xen_extra              : .4-3
xen_version            : 4.17.4-3
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : d530627aaa9b, pq 7587628e7d91
xen_commandline        : dom0_mem=6752M,max:6752M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
cc_compiler            : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
cc_compile_by          : mockbuild
cc_compile_domain      : [unknown]
cc_compile_date        : Thu Jun 20 18:17:10 CEST 2024
build_id               : 9497a1ec7ec99f5075421732b0ec37781ba739a9
xend_config_format     : 4

VMs - Sender and Receiver

• Distro: Ubuntu 24.04
• Kernel: 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC
• vCPUs: 32
• RAM: 4GB

[bullerwins]

@probain said in Epyc VM to VM networking slow:

I ran these tests now that newer updates have been released for 8.3-beta.
Results are as below:

• iperf-sender -> iperf-receiver: 5.06Gbit/s
• iperf-sender -> iperf-receiver -P4: 7.53Gbit/s
• host -> iperf-receiver: 7.83Gbit/s
• host -> iperf-receiver -P4: 13.0Gbit/s

Host (dom0):

• CPU: AMD EPYC 7302P
• Sockets: 1
• RAM: 6.59GB (dom0) / 112GB for VMs
• MotherBoard: H12SSL-i
• NIC: X540-AT2 (rev 01)

xl info -n

host                   : xcp
release                : 4.19.0+1
version                : #1 SMP Mon Jun 24 17:20:04 CEST 2024
machine                : x86_64
nr_cpus                : 32
max_cpu_id             : 31
nr_nodes               : 1
cores_per_socket       : 16
threads_per_core       : 2
cpu_mhz                : 2999.997
hw_caps                : 178bf3ff:7ed8320b:2e500800:244037ff:0000000f:219c91a9:00400004:00000780
virt_caps              : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2
total_memory           : 114549
free_memory            : 62685
sharing_freed_memory   : 0
sharing_used_memory    : 0
outstanding_claims     : 0
free_cpus              : 0
cpu_topology           :
cpu:    core    socket     node
  0:       0        0        0
  1:       0        0        0
  2:       1        0        0
  3:       1        0        0
  4:       4        0        0
  5:       4        0        0
  6:       5        0        0
  7:       5        0        0
  8:       8        0        0
  9:       8        0        0
 10:       9        0        0
 11:       9        0        0
 12:      12        0        0
 13:      12        0        0
 14:      13        0        0
 15:      13        0        0
 16:      16        0        0
 17:      16        0        0
 18:      17        0        0
 19:      17        0        0
 20:      20        0        0
 21:      20        0        0
 22:      21        0        0
 23:      21        0        0
 24:      24        0        0
 25:      24        0        0
 26:      25        0        0
 27:      25        0        0
 28:      28        0        0
 29:      28        0        0
 30:      29        0        0
 31:      29        0        0
device topology        :
device           node
No device topology data available
numa_info              :
node:    memsize    memfree    distances
   0:    115955      62685      10
xen_major              : 4
xen_minor              : 17
xen_extra              : .4-3
xen_version            : 4.17.4-3
xen_caps               : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : d530627aaa9b, pq 7587628e7d91
xen_commandline        : dom0_mem=6752M,max:6752M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
cc_compiler            : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
cc_compile_by          : mockbuild
cc_compile_domain      : [unknown]
cc_compile_date        : Thu Jun 20 18:17:10 CEST 2024
build_id               : 9497a1ec7ec99f5075421732b0ec37781ba739a9
xend_config_format     : 4

VMs - Sender and Receiver

• Distro: Ubuntu 24.04
• Kernel: 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC
• vCPUs: 32
• RAM: 4GB

have you tested without the 8.3 updates? The results seem still low. Any improvement?

[probain]

@bullerwins Unfortunately I didn't. In hindsight I wish I did.

[JamesG]

These latest 8.3 update speeds are still slower than a 13 year-old Xeon E3 1230.

[Seneram]

I can unfortunately share that from ongoing ticket investigations in this, It is far more deeply rooted than something that a patch of going from one major kernel to another will "just fix" There are multiple leads being investigated and multiple vendors involved.

[olivierlambert]

I'd like to check something to see if it's coherent with our tests, by using 2x similar VMs (4vCPUs/4G RAM):

• iperf monothread speed on a "fresh" Debian 10 install (4.19 kernel)
• the same bench with 5.10.0 kernel from backports (add deb http://deb.debian.org/debian buster-backports main contrib non-free in your source list and then apt install linux-image-5.10, don't forget to reboot to be on that kernel)

Do you see a performance diff between those?

[john.c]

@olivierlambert said in Epyc VM to VM networking slow:

I'd like to check something to see if it's coherent with our tests, by using 2x similar VMs (4vCPUs/4G RAM):

• iperf monothread speed on a "fresh" Debian 10 install (4.19 kernel)
• the same bench with 5.10.0 kernel from backports (add deb http://deb.debian.org/debian buster-backports main contrib non-free in your source list and then apt install linux-image-5.10, don't forget to reboot to be on that kernel)

Do you see a performance diff between those?

FYI, getting a Debian 10 backports or non-backports packages are going to now be extremely difficult. The Debian Linux 10 LTS has reached EOL. Now currently in ELTS from the beginning of this month until 30/06/2029, though covering only a subset of the packages.

https://www.debian.org/News/2024/20240615

[olivierlambert]

I had no issue to test it quickly. The thing is for the sake of testing and try to identify a potential regression, not for production usage or whatnot.

[olivierlambert]

I identified a specific regression in a Debian kernel build since 5.10, we are investigating the "why" (starting from this exact build: https://snapshot.debian.org/package/linux/5.10.92-1/)

[probain]

@olivierlambert
Would it be possible for you to either offer a ISO to download? Or maybe seed one? I really want to help test this. But I'm getting lost with how Debian provides their legacy images and this jig-boo (intentionally misspelled)

[G-Ork]

May someone could graph their vm.
Comparing a slow vm with a full speed could bring light into darknes.

https://www.brendangregg.com/Articles/Linux_Kernel_Performance_Flame_Graphs.pdf

[olivierlambert]

@probain Debian 10 is available in the XOA Hub.

[probain]

@olivierlambert
I wasn't aware. Thanks! Downloading for doing a test, right away

Test done:

				Run1	Run2	Run3
Sender:   Debian10 kernel 4.19	4.81Gb	4.81Gb	4.83Gb
Reveiver: Debian10 kernel 4.19

Sender:   Debian10 kernel 5.10	5.13Gb	5.02Gb	5.12Gb
Reveiver: Debian10 kernel 4.19

Sender:   Debian10 kernel 5.10	4.98Gb	5.02Gb	4.97Gb
Reveiver: Debian10 kernel 5.10

sender runs 'iperf -c <IP-to-receiver> -t 60'

Kernel 4.19 = 4.19.0-6-amd64
Kernel 5.10 = 5.10.0-0.deb10.24-amd64

CPU 4 cores (AMD EPYC 7302P)
RAM 4GB

Created from XOA-hub

[olivierlambert]

Thanks @probain , now can you try iperf -s in the Dom0 and iperf -c <IP dom0> in the Debian guest?

[probain]

@olivierlambert
vm -> dom0 results in "no route to host": firewall?

Results will be shown for dom0 -> vm. Listed by each kernel installed on vm.

Just as earlier. VM is installed via XOA Hub, with 4 CPU and 4GB RAM. Host CPU running on AMD EPYC 7302P.

VM kernel ver.	Run1	Run2	Run3
kernel 4.19.0	8.47Gb	8.82Gb	8.43Gb
kernel 5.10.0	7.12Gb	7.07Gb	7.11Gb

[olivierlambert]

yes disable the fw first (only in a testing lab obviously) with iptables -F

[probain]

@olivierlambert how do I restore the iptables again afterwards? Other than reboot ofc

Update: Tests done

vm -> dom0

		Run1	Run2	Run3
kernel 4.19.0	5.84Gb	5.77Gb	5.85Gb
kernel 5.10.0	1.25Gb	1.26Gb	1.28

Specs are just as previous post.

[olivierlambert]

Thanks so at least it confirms something we are also spotting in here. We found the exact commit.

[G-Ork]

Here are the opterons with dropped firewall:

source	destination	OS	Kernel	Speed Average
vm	dom	debian 10	4.19.0-6-amd64	6.57 Gbits/sec
dom	vm	debian 10	4.19.0-6-amd64	1.79 Gbits/sec
vm	dom	truenas	6.6.20	2.01 Gbits/sec
dom	vm	truenas	6.6.20	1.82 Gbits/sec
host	vm	debian 10	4.19.0-6-amd64	5.32 Gbits/sec
host	vm	truenas	6.6.20	1.92 Gbits/sec
host	dom	debian	4.19.0+1	8.97 Gbits/sec

[G-Ork]

@probain said in Epyc VM to VM networking slow:

I restore the iptables again afterwards? Other than reboot

this worked for me

action	command
save	iptables-save > firewall.conf
flush	iptables -F
restore	cat firewall.conf | iptables-restore