Epyc VM to VM networking slow
-
Just for completeness. I have the same issue with older AMD 6380 Opterons. The same hardware had full speed on esxi hypervisor. Also have full speed using harvester / rancher. I'm using xcp-ng for more than two years and have that issue since day one.
----------------------------------------------------------- Server listening on 5201 (test #1 - dom0 to workstation) ----------------------------------------------------------- [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 10.8 GBytes 9.29 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #2 - vm to workstation) ----------------------------------------------------------- [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 3.18 GBytes 2.73 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #3 dom0 to vm) ----------------------------------------------------------- [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 2.23 GBytes 1.91 Gbits/sec receiver ----------------------------------------------------------- Server listening on 5201 (test #4 vom to vm) ----------------------------------------------------------- [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.97 GBytes 1.69 Gbits/sec receiver ----------------------------------------------------------- -
I ran these tests now that newer updates have been released for 8.3-beta.
Results are as below:- iperf-sender -> iperf-receiver: 5.06Gbit/s
- iperf-sender -> iperf-receiver -P4: 7.53Gbit/s
- host -> iperf-receiver: 7.83Gbit/s
- host -> iperf-receiver -P4: 13.0Gbit/s
Host (dom0):
- CPU: AMD EPYC 7302P
- Sockets: 1
- RAM: 6.59GB (dom0) / 112GB for VMs
- MotherBoard: H12SSL-i
- NIC: X540-AT2 (rev 01)
xl info -n
host : xcp release : 4.19.0+1 version : #1 SMP Mon Jun 24 17:20:04 CEST 2024 machine : x86_64 nr_cpus : 32 max_cpu_id : 31 nr_nodes : 1 cores_per_socket : 16 threads_per_core : 2 cpu_mhz : 2999.997 hw_caps : 178bf3ff:7ed8320b:2e500800:244037ff:0000000f:219c91a9:00400004:00000780 virt_caps : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2 total_memory : 114549 free_memory : 62685 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 cpu_topology : cpu: core socket node 0: 0 0 0 1: 0 0 0 2: 1 0 0 3: 1 0 0 4: 4 0 0 5: 4 0 0 6: 5 0 0 7: 5 0 0 8: 8 0 0 9: 8 0 0 10: 9 0 0 11: 9 0 0 12: 12 0 0 13: 12 0 0 14: 13 0 0 15: 13 0 0 16: 16 0 0 17: 16 0 0 18: 17 0 0 19: 17 0 0 20: 20 0 0 21: 20 0 0 22: 21 0 0 23: 21 0 0 24: 24 0 0 25: 24 0 0 26: 25 0 0 27: 25 0 0 28: 28 0 0 29: 28 0 0 30: 29 0 0 31: 29 0 0 device topology : device node No device topology data available numa_info : node: memsize memfree distances 0: 115955 62685 10 xen_major : 4 xen_minor : 17 xen_extra : .4-3 xen_version : 4.17.4-3 xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : d530627aaa9b, pq 7587628e7d91 xen_commandline : dom0_mem=6752M,max:6752M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 cc_compiler : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1) cc_compile_by : mockbuild cc_compile_domain : [unknown] cc_compile_date : Thu Jun 20 18:17:10 CEST 2024 build_id : 9497a1ec7ec99f5075421732b0ec37781ba739a9 xend_config_format : 4VMs - Sender and Receiver
- Distro: Ubuntu 24.04
- Kernel: 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC
- vCPUs: 32
- RAM: 4GB
-
@probain said in Epyc VM to VM networking slow:
I ran these tests now that newer updates have been released for 8.3-beta.
Results are as below:- iperf-sender -> iperf-receiver: 5.06Gbit/s
- iperf-sender -> iperf-receiver -P4: 7.53Gbit/s
- host -> iperf-receiver: 7.83Gbit/s
- host -> iperf-receiver -P4: 13.0Gbit/s
Host (dom0):
- CPU: AMD EPYC 7302P
- Sockets: 1
- RAM: 6.59GB (dom0) / 112GB for VMs
- MotherBoard: H12SSL-i
- NIC: X540-AT2 (rev 01)
xl info -n
host : xcp release : 4.19.0+1 version : #1 SMP Mon Jun 24 17:20:04 CEST 2024 machine : x86_64 nr_cpus : 32 max_cpu_id : 31 nr_nodes : 1 cores_per_socket : 16 threads_per_core : 2 cpu_mhz : 2999.997 hw_caps : 178bf3ff:7ed8320b:2e500800:244037ff:0000000f:219c91a9:00400004:00000780 virt_caps : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2 total_memory : 114549 free_memory : 62685 sharing_freed_memory : 0 sharing_used_memory : 0 outstanding_claims : 0 free_cpus : 0 cpu_topology : cpu: core socket node 0: 0 0 0 1: 0 0 0 2: 1 0 0 3: 1 0 0 4: 4 0 0 5: 4 0 0 6: 5 0 0 7: 5 0 0 8: 8 0 0 9: 8 0 0 10: 9 0 0 11: 9 0 0 12: 12 0 0 13: 12 0 0 14: 13 0 0 15: 13 0 0 16: 16 0 0 17: 16 0 0 18: 17 0 0 19: 17 0 0 20: 20 0 0 21: 20 0 0 22: 21 0 0 23: 21 0 0 24: 24 0 0 25: 24 0 0 26: 25 0 0 27: 25 0 0 28: 28 0 0 29: 28 0 0 30: 29 0 0 31: 29 0 0 device topology : device node No device topology data available numa_info : node: memsize memfree distances 0: 115955 62685 10 xen_major : 4 xen_minor : 17 xen_extra : .4-3 xen_version : 4.17.4-3 xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : d530627aaa9b, pq 7587628e7d91 xen_commandline : dom0_mem=6752M,max:6752M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311 cc_compiler : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1) cc_compile_by : mockbuild cc_compile_domain : [unknown] cc_compile_date : Thu Jun 20 18:17:10 CEST 2024 build_id : 9497a1ec7ec99f5075421732b0ec37781ba739a9 xend_config_format : 4VMs - Sender and Receiver
- Distro: Ubuntu 24.04
- Kernel: 6.8.0-36-generic #36-Ubuntu SMP PREEMPT_DYNAMIC
- vCPUs: 32
- RAM: 4GB
have you tested without the 8.3 updates? The results seem still low. Any improvement?
-
@bullerwins Unfortunately I didn't. In hindsight I wish I did.
-
These latest 8.3 update speeds are still slower than a 13 year-old Xeon E3 1230.
-
I can unfortunately share that from ongoing ticket investigations in this, It is far more deeply rooted than something that a patch of going from one major kernel to another will "just fix" There are multiple leads being investigated and multiple vendors involved.
-
I'd like to check something to see if it's coherent with our tests, by using 2x similar VMs (4vCPUs/4G RAM):
- iperf monothread speed on a "fresh" Debian 10 install (4.19 kernel)
- the same bench with 5.10.0 kernel from backports (add
deb http://deb.debian.org/debian buster-backports main contrib non-freein your source list and then apt install linux-image-5.10, don't forget to reboot to be on that kernel)
Do you see a performance diff between those?
-
@olivierlambert said in Epyc VM to VM networking slow:
I'd like to check something to see if it's coherent with our tests, by using 2x similar VMs (4vCPUs/4G RAM):
- iperf monothread speed on a "fresh" Debian 10 install (4.19 kernel)
- the same bench with 5.10.0 kernel from backports (add
deb http://deb.debian.org/debian buster-backports main contrib non-freein your source list and then apt install linux-image-5.10, don't forget to reboot to be on that kernel)
Do you see a performance diff between those?
FYI, getting a Debian 10 backports or non-backports packages are going to now be extremely difficult. The Debian Linux 10 LTS has reached EOL. Now currently in ELTS from the beginning of this month until 30/06/2029, though covering only a subset of the packages.
-
I had no issue to test it quickly. The thing is for the sake of testing and try to identify a potential regression, not for production usage or whatnot.
-
I identified a specific regression in a Debian kernel build since 5.10, we are investigating the "why" (starting from this exact build: https://snapshot.debian.org/package/linux/5.10.92-1/)
-
@olivierlambert
Would it be possible for you to either offer a ISO to download? Or maybe seed one? I really want to help test this. But I'm getting lost with how Debian provides their legacy images and this jig-boo (intentionally misspelled)
-
May someone could graph their vm.
Comparing a slow vm with a full speed could bring light into darknes.https://www.brendangregg.com/Articles/Linux_Kernel_Performance_Flame_Graphs.pdf
-
@probain Debian 10 is available in the XOA Hub.
-
@olivierlambert
I wasn't aware. Thanks! Downloading for doing a test, right awayTest done:
Run1 Run2 Run3 Sender: Debian10 kernel 4.19 4.81Gb 4.81Gb 4.83Gb Reveiver: Debian10 kernel 4.19 Sender: Debian10 kernel 5.10 5.13Gb 5.02Gb 5.12Gb Reveiver: Debian10 kernel 4.19 Sender: Debian10 kernel 5.10 4.98Gb 5.02Gb 4.97Gb Reveiver: Debian10 kernel 5.10sender runs 'iperf -c <IP-to-receiver> -t 60'
Kernel 4.19 = 4.19.0-6-amd64
Kernel 5.10 = 5.10.0-0.deb10.24-amd64CPU 4 cores (AMD EPYC 7302P)
RAM 4GBCreated from XOA-hub
-
Thanks @probain , now can you try
iperf -sin the Dom0 andiperf -c <IP dom0>in the Debian guest? -
@olivierlambert
vm -> dom0 results in "no route to host": firewall?Results will be shown for dom0 -> vm. Listed by each kernel installed on vm.
Just as earlier. VM is installed via XOA Hub, with 4 CPU and 4GB RAM. Host CPU running on AMD EPYC 7302P.
VM kernel ver. Run1 Run2 Run3 kernel 4.19.0 8.47Gb 8.82Gb 8.43Gb kernel 5.10.0 7.12Gb 7.07Gb 7.11Gb -
yes disable the fw first (only in a testing lab obviously) with
iptables -F -
@olivierlambert how do I restore the iptables again afterwards? Other than reboot ofc
Update: Tests done
vm -> dom0 Run1 Run2 Run3 kernel 4.19.0 5.84Gb 5.77Gb 5.85Gb kernel 5.10.0 1.25Gb 1.26Gb 1.28Specs are just as previous post.
-
Thanks so at least it confirms something we are also spotting in here. We found the exact commit.
-
Here are the opterons with dropped firewall:
source destination OS Kernel Speed Average vm dom debian 10 4.19.0-6-amd64 6.57 Gbits/sec dom vm debian 10 4.19.0-6-amd64 1.79 Gbits/sec vm dom truenas 6.6.20 2.01 Gbits/sec dom vm truenas 6.6.20 1.82 Gbits/sec host vm debian 10 4.19.0-6-amd64 5.32 Gbits/sec host vm truenas 6.6.20 1.92 Gbits/sec host dom debian 4.19.0+1 8.97 Gbits/sec
