@bleader I've been investigating this issue on my own system and came across this discussion. I know this is a somewhat old thread so I hope it's ok to contribute more data here!
Host:
- CPU: EPYC 7302p
- Number of sockets: 1
- CPU pinning: no
- XCP-NG version: 8.3 beta 2, Xen 4.17 (everything current as of the time of writing)
- Output of
xl info -n:
host : xcp-ng
release : 4.19.0+1
version : #1 SMP Wed Jan 24 17:19:11 CET 2024
machine : x86_64
nr_cpus : 32
max_cpu_id : 31
nr_nodes : 1
cores_per_socket : 16
threads_per_core : 2
cpu_mhz : 3000.001
hw_caps : 178bf3ff:7ed8320b:2e500800:244037ff:0000000f:219c91a9:00400004:00000780
virt_caps : pv hvm hvm_directio pv_directio hap gnttab-v1 gnttab-v2
total_memory : 130931
free_memory : 24740
sharing_freed_memory : 0
sharing_used_memory : 0
outstanding_claims : 0
free_cpus : 0
cpu_topology :
cpu: core socket node
0: 0 0 0
1: 0 0 0
2: 1 0 0
3: 1 0 0
4: 4 0 0
5: 4 0 0
6: 5 0 0
7: 5 0 0
8: 8 0 0
9: 8 0 0
10: 9 0 0
11: 9 0 0
12: 12 0 0
13: 12 0 0
14: 13 0 0
15: 13 0 0
16: 16 0 0
17: 16 0 0
18: 17 0 0
19: 17 0 0
20: 20 0 0
21: 20 0 0
22: 21 0 0
23: 21 0 0
24: 24 0 0
25: 24 0 0
26: 25 0 0
27: 25 0 0
28: 28 0 0
29: 28 0 0
30: 29 0 0
31: 29 0 0
device topology :
device node
No device topology data available
numa_info :
node: memsize memfree distances
0: 132338 24740 10
xen_major : 4
xen_minor : 17
xen_extra : .3-3
xen_version : 4.17.3-3
xen_caps : xen-3.0-x86_64 hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : $Format:%H$, pq ???
xen_commandline : dom0_mem=7568M,max:7568M watchdog ucode=scan dom0_max_vcpus=1-16 crashkernel=256M,below=4G console=vga vga=mode-0x0311
cc_compiler : gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
cc_compile_by : mockbuild
cc_compile_domain : [unknown]
cc_compile_date : Wed Feb 28 10:12:19 CET 2024
build_id : 9a011a28e29a21a7643376b36aec959253587d42
xend_config_format : 4
Test set 1:
Server and client were both Debian 12 (Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64) with 4 cores.
VM to VM (1 thread):
iperf -c 192.168.1.66 -t 60
------------------------------------------------------------
Client connecting to 192.168.1.66, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.69 port 55530 connected with 192.168.1.66 port 5001 (icwnd/mss/irtt=14/1448/551)
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-60.0213 sec 38.2 GBytes 5.47 Gbits/sec
xentop: 100 / 150 / 250
VM to VM (4 threads):
iperf -c 192.168.1.66 -t 60 -P4
------------------------------------------------------------
Client connecting to 192.168.1.66, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 2] local 192.168.1.69 port 35702 connected with 192.168.1.66 port 5001 (icwnd/mss/irtt=14/1448/531)
[ 4] local 192.168.1.69 port 35708 connected with 192.168.1.66 port 5001 (icwnd/mss/irtt=14/1448/576)
[ 1] local 192.168.1.69 port 35714 connected with 192.168.1.66 port 5001 (icwnd/mss/irtt=14/1448/458)
[ 3] local 192.168.1.69 port 35692 connected with 192.168.1.66 port 5001 (icwnd/mss/irtt=14/1448/744)
[ ID] Interval Transfer Bandwidth
[ 2] 0.0000-60.0141 sec 12.4 GBytes 1.77 Gbits/sec
[ 1] 0.0000-60.0129 sec 13.9 GBytes 1.99 Gbits/sec
[ 3] 0.0000-60.0141 sec 14.5 GBytes 2.07 Gbits/sec
[ 4] 0.0000-60.0301 sec 12.2 GBytes 1.75 Gbits/sec
[SUM] 0.0000-60.0071 sec 53.0 GBytes 7.58 Gbits/sec
xentop: 165 / 200 / 380
Host to VM (1 thread):
iperf -c 192.168.1.66 -t 60
------------------------------------------------------------
Client connecting to 192.168.1.66, TCP port 5001
TCP window size: 297 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.1 port 37804 connected with 192.168.1.66 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 6.58 GBytes 942 Mbits/sec
xentop: N/A / 135 / 145
Host to VM (4 threads):
iperf -c 192.168.1.66 -t 60 -P4
------------------------------------------------------------
Client connecting to 192.168.1.66, TCP port 5001
TCP window size: 112 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.1.1 port 37812 connected with 192.168.1.66 port 5001
[ 3] local 192.168.1.1 port 37808 connected with 192.168.1.66 port 5001
[ 6] local 192.168.1.1 port 37814 connected with 192.168.1.66 port 5001
[ 4] local 192.168.1.1 port 37810 connected with 192.168.1.66 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 1.63 GBytes 233 Mbits/sec
[ 6] 0.0-60.0 sec 2.08 GBytes 298 Mbits/sec
[ 4] 0.0-60.0 sec 1.07 GBytes 154 Mbits/sec
[ 5] 0.0-60.0 sec 1.80 GBytes 257 Mbits/sec
[SUM] 0.0-60.0 sec 6.58 GBytes 942 Mbits/sec
xentop: N/A / 155 / 155
Test set 2:
Server: FreeBSD 14 (FreeBSD 14.0-RELEASE (GENERIC) #0 releng/14.0-n265380-f9716eee8ab4: Fri Nov 10 05:57:23 UTC 2023) with 4 cores.
Client: Debian 12 (Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64) with 4 cores.
VM to VM (1 thread):
iperf -c 192.168.1.64 -t 60
------------------------------------------------------------
Client connecting to 192.168.1.64, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 1] local 192.168.1.69 port 38572 connected with 192.168.1.64 port 5001 (icwnd/mss/irtt=14/1448/905)
[ ID] Interval Transfer Bandwidth
[ 1] 0.0000-60.0089 sec 21.3 GBytes 3.04 Gbits/sec
xentop: 125 / 355 / 325
VM to VM (4 threads):
iperf -c 192.168.1.64 -t 60 -P4
------------------------------------------------------------
Client connecting to 192.168.1.64, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.69 port 50068 connected with 192.168.1.64 port 5001 (icwnd/mss/irtt=14/1448/753)
[ 1] local 192.168.1.69 port 50078 connected with 192.168.1.64 port 5001 (icwnd/mss/irtt=14/1448/513)
[ 4] local 192.168.1.69 port 50088 connected with 192.168.1.64 port 5001 (icwnd/mss/irtt=14/1448/411)
[ 2] local 192.168.1.69 port 50070 connected with 192.168.1.64 port 5001 (icwnd/mss/irtt=14/1448/676)
[ ID] Interval Transfer Bandwidth
[ 4] 0.0000-60.0299 sec 9.48 GBytes 1.36 Gbits/sec
[ 1] 0.0000-60.0299 sec 6.56 GBytes 938 Mbits/sec
[ 3] 0.0000-60.0301 sec 11.2 GBytes 1.60 Gbits/sec
[ 2] 0.0000-60.0293 sec 6.61 GBytes 947 Mbits/sec
[SUM] 0.0000-60.0146 sec 33.8 GBytes 4.84 Gbits/sec
xentop: 220 / 400 / 730
Host to VM (1 thread):
iperf -c 192.168.1.64 -t 60
------------------------------------------------------------
Client connecting to 192.168.1.64, TCP port 5001
TCP window size: 212 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.1 port 58464 connected with 192.168.1.64 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-60.0 sec 6.58 GBytes 941 Mbits/sec
xentop: N/A / 295 / 205
Host to VM (4 threads):
iperf -c 192.168.1.64 -t 60 -P4
------------------------------------------------------------
Client connecting to 192.168.1.64, TCP port 5001
TCP window size: 130 KByte (default)
------------------------------------------------------------
[ 5] local 192.168.1.1 port 58470 connected with 192.168.1.64 port 5001
[ 3] local 192.168.1.1 port 58468 connected with 192.168.1.64 port 5001
[ 4] local 192.168.1.1 port 58472 connected with 192.168.1.64 port 5001
[ 6] local 192.168.1.1 port 58474 connected with 192.168.1.64 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-60.0 sec 1.73 GBytes 247 Mbits/sec
[ 3] 0.0-60.0 sec 1.56 GBytes 224 Mbits/sec
[ 4] 0.0-60.0 sec 1.73 GBytes 247 Mbits/sec
[ 6] 0.0-60.0 sec 1.56 GBytes 224 Mbits/sec
[SUM] 0.0-60.0 sec 6.58 GBytes 942 Mbits/sec
xentop: N/A / 280 / 205
Conclusion:
No special tuning on any of the VMs, just a fresh install from the netboot ISO for each respective OS.
I also don't fully understand why my host seems to be limited to 1Gb. The management interface is 1Gb, but that shouldn't matter? The other physical NIC is 10Gb SFP+, just for the sake of completeness.
Please let me know if there's anything at all that I can do to help with this!