Epyc VM to VM networking slow
-
OEL 8 & 9 wouldn't contain the fix unless they applied extra patches for this to the RHEL 8 & 9 kernel(s). I'll let the hypervisor team check the current status.
-
kernel-4.18.0-553.71.1.0.1.el8_10(OL8) andkernel-5.14.0-570.37.1.0.1.el9_6(OL9) do not contain the fix.kernel-6.12.0-55.29.1.0.1.el10_0(OL10) does. -
-
Tested the new updates on my prod EPYC 7402P pool with
iperf3. Seems like quite a good uplift
Ubuntu 24.04 VM (6 cores) -> bare metal server (6 cores) over a 2x25Gbit LACP link.
Pre-patch
- iperf3 -P1 : 9.72Gbit/s
- iperf3 -P6 : 14.6GBis/s
Post Patch
- iperf3 -P1 : 11.3GBit/s
- iperf3 -P6 : 24.2GBit/s
Ubuntu 24.04 VM (6 cores) -> Ubuntu 24.04 VM (6 cores) on the same host
Pre Patch
Forgot to test this...
Post Patch
- iperf3 -P1 : 13.7GBit/s
- iperf3 -P6 : 30.8GBit/s
- iperf3 -P24 : 40.4GBit/s
Our servers have
Last-Level Cache (LLC) as NUMA Nodeenabled as most our VMs do not have huge amount of vCPUs assigned. This means for the EPYC 7402P (24c/48t) we have 8 NUMA nodes. We however do not usexl cpupool-numa-split. -
That's nice! It means it scales relatively well with many threads, it's a good result
-
@olivierlambert Can we get an updated XOA with this patch?
-
@Forza What patch are you referring to that would relate to XOA?
-
@stormi said in Epyc VM to VM networking slow:
@Forza What patch are you referring to that would relate to XOA?
It seems only recent kernels can take advantage of the improvements. From the blog post mentioned above:
The change only affects Linux guests. To make it effective, their kernel must support the feature which enables this fix.
Iperf inside XOA is much slower than other VMs (like the Ubundu 24.04 above).
When I run on the same host (pool master) against our NFS SR:
Ubuntu 24.04 (kernel 6.8):
# iperf3 -c 10.12.9.4 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 16.0 GBytes 13.7 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 16.0 GBytes 13.7 Gbits/sec receiver # iperf3 -c 10.12.9.4 P4 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.26 GBytes 5.37 Gbits/sec 8861 sender [ 5] 0.00-10.00 sec 6.25 GBytes 5.37 Gbits/sec receiver [ 7] 0.00-10.00 sec 8.57 GBytes 7.36 Gbits/sec 8372 sender [ 7] 0.00-10.00 sec 8.57 GBytes 7.36 Gbits/sec receiver [ 9] 0.00-10.00 sec 9.05 GBytes 7.77 Gbits/sec 10192 sender [ 9] 0.00-10.00 sec 9.05 GBytes 7.77 Gbits/sec receiver [ 11] 0.00-10.00 sec 6.12 GBytes 5.25 Gbits/sec 7144 sender [ 11] 0.00-10.00 sec 6.11 GBytes 5.25 Gbits/sec receiver [SUM] 0.00-10.00 sec 30.0 GBytes 25.8 Gbits/sec 34569 sender [SUM] 0.00-10.00 sec 30.0 GBytes 25.8 Gbits/sec receiverXOA 2025.08 (kernel 6.1):
# iperf3 -c 10.12.9.4 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 6.26 GBytes 5.37 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 6.25 GBytes 5.37 Gbits/sec receiver # iperf3 -c 10.12.9.4 -P4 - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 4.71 GBytes 4.05 Gbits/sec 3987 sender [ 5] 0.00-10.00 sec 4.71 GBytes 4.05 Gbits/sec receiver [ 7] 0.00-10.00 sec 4.61 GBytes 3.96 Gbits/sec 3086 sender [ 7] 0.00-10.00 sec 4.61 GBytes 3.96 Gbits/sec receiver [ 9] 0.00-10.00 sec 6.77 GBytes 5.81 Gbits/sec 7745 sender [ 9] 0.00-10.00 sec 6.77 GBytes 5.81 Gbits/sec receiver [ 11] 0.00-10.00 sec 5.42 GBytes 4.65 Gbits/sec 629 sender [ 11] 0.00-10.00 sec 5.42 GBytes 4.65 Gbits/sec receiver [SUM] 0.00-10.00 sec 21.5 GBytes 18.5 Gbits/sec 15447 sender [SUM] 0.00-10.00 sec 21.5 GBytes 18.5 Gbits/sec receiver -
XOA's kernel should have the capability already, as it's a Debian 12 with stock kernel. Also, the bottleneck is ONLY between VMs on the same host.
-
@olivierlambert said in Epyc VM to VM networking slow:
XOA's kernel should have the capability already, as it's a Debian 12 with stock kernel. Also, the bottleneck is ONLY between VMs on the same host.
OK, but then I do not understand the huge difference.
Tested some VM-VM traffic on the same host:
Ubuntu (kernel 6.8) -> Alpine (kernel 6.12): 13.7Gbit/s, P4 = 23.8 Gbit/s
XOA (kernel 6.1) -> Ubuntu (kernel 6.8) : 5.5 Gbit/s, P4 = 17.5 Gbit/s
XOA (kernel 6.1) -> Alpine (kernel 6.12): 11.9 Gbit/s, P4 = 13.6 Gbit/sAnd for measure against the bare metal NFS server:
Alpine -> NFS SR (kernel 6.12): 13.7 Gbit/s, P4 = 23.4 Gbit/s -
@Forza By default XOA VM has 2 vcpus, how many vcpus do your ubuntu have? Althrough iperf isn't running multithreaded in your test, there is one queue on the kernel side of the VM per vcpu to process packets.
-
@bleader said in Epyc VM to VM networking slow:
@Forza By default XOA VM has 2 vcpus, how many vcpus do your ubuntu have? Althrough iperf isn't running multithreaded in your test, there is one queue on the kernel side of the VM per vcpu to process packets.
I have 8 CPUS on XOA, 4 CPUs in the Alpine VM and 6 in the Ubuntu VM
P4uses 4 threads with iperf3.@bleader Are you saying you do not see any performance differences between XOA and VMs with more recent kernels?
-
If you want to compare perf between kernels, do that with the same number of vCPUs ideally (and also the same env, ie different kernel versions with the same iperf version)
-
@olivierlambert said in Epyc VM to VM networking slow:
If you want to compare perf between kernels, do that with the same number of vCPUs ideally (and also the same env, ie different kernel versions with the same iperf version)
Sure. but in this context it is not that relevant as all other VMs got a boost while XOA didnt as much.
Does the Debian 6.1 kernel that XOA uses have the backported fixes mentioned in https://xcp-ng.org/blog/2025/09/01/september-2025-maintenance-update-for-xcp-ng-8-3/
Even if it has, it is clear that more resent kernels are much faster. Why not release a XOA with more recent kernels?
-
Kernel version could have an impact and being unrelated to the fix we provided. Get an even more recent kernel on your XOA to test (eg one from testing)
-
@olivierlambert said in Epyc VM to VM networking slow:
Kernel version could have an impact and being unrelated to the fix we provided. Get an even more recent kernel on your XOA to test (eg one from testing)
I was not able to update the kernel due to some issue in XOA installation. I commented on this at https://xcp-ng.org/forum/post/97522
But to clarify — my goal is to reach the same performance in XOA as with our other VMs. I had assumed it lacked the kernel support, and that led to the confusion. Sorry for that.
-
