XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Epyc VM to VM networking slow

    Scheduled Pinned Locked Moved Compute
    260 Posts 28 Posters 178.6k Views 29 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • gduperreyG Offline
      gduperrey Vates 🪐 XCP-ng Team
      last edited by gduperrey

      The fix, which was proposed as a test to resolve some of the issues encountered, has been integrated into an official update candidate which will be released to everyone next time we publish updates. For more information on this update, you can consult the following post: https://xcp-ng.org/forum/post/96135

      1 Reply Last reply Reply Quote 5
      • K Offline
        ksyblast
        last edited by

        Hello!
        First of all thank you very much for providing the fix!

        Following the blog post https://xcp-ng.org/blog/2025/09/01/september-2025-maintenance-update-for-xcp-ng-8-3/:

        The change only affects Linux guests. To make it effective, their kernel must support the feature which enables this fix. Linux distributions that have recent enough kernels or apply fixes from the mainline LTS kernels support it. Older ones don't (example: Ubuntu 20.04). Some currently supported LTS distros don't have the required patch yet, notably RHEL 8 and 9 and their derivatives. This might change if we can convince them to apply the patch to their kernel.

        Where can we find the list of the supported OS and kernels to run some?

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Because that would be a pretty big list. Which distros you have in mind?

          K 1 Reply Last reply Reply Quote 0
          • K Offline
            ksyblast @olivierlambert
            last edited by

            @olivierlambert we are using Oracle Linux, is OEL8/9/10 supported? As I understand OEL8 and 9 do contain a fix. We can also try with any other OS suggested by you. Thanks!

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Question for @Team-Hypervisor-Kernel

              1 Reply Last reply Reply Quote 0
              • stormiS Offline
                stormi Vates 🪐 XCP-ng Team
                last edited by

                OEL 8 & 9 wouldn't contain the fix unless they applied extra patches for this to the RHEL 8 & 9 kernel(s). I'll let the hypervisor team check the current status.

                1 Reply Last reply Reply Quote 1
                • D Offline
                  dinhngtu Vates 🪐 XCP-ng Team
                  last edited by

                  kernel-4.18.0-553.71.1.0.1.el8_10 (OL8) and kernel-5.14.0-570.37.1.0.1.el9_6 (OL9) do not contain the fix. kernel-6.12.0-55.29.1.0.1.el10_0 (OL10) does.

                  1 Reply Last reply Reply Quote 4
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    Thanks @dinhngtu and @stormi

                    ForzaF 1 Reply Last reply Reply Quote 0
                    • ForzaF Offline
                      Forza @olivierlambert
                      last edited by Forza

                      Tested the new updates on my prod EPYC 7402P pool with iperf3. Seems like quite a good uplift 🙂

                      Ubuntu 24.04 VM (6 cores) -> bare metal server (6 cores) over a 2x25Gbit LACP link.

                      Pre-patch

                      • iperf3 -P1 : 9.72Gbit/s
                      • iperf3 -P6 : 14.6GBis/s

                      Post Patch

                      • iperf3 -P1 : 11.3GBit/s
                      • iperf3 -P6 : 24.2GBit/s

                      Ubuntu 24.04 VM (6 cores) -> Ubuntu 24.04 VM (6 cores) on the same host

                      Pre Patch

                      Forgot to test this...

                      Post Patch

                      • iperf3 -P1 : 13.7GBit/s
                      • iperf3 -P6 : 30.8GBit/s
                      • iperf3 -P24 : 40.4GBit/s

                      Our servers have Last-Level Cache (LLC) as NUMA Node enabled as most our VMs do not have huge amount of vCPUs assigned. This means for the EPYC 7402P (24c/48t) we have 8 NUMA nodes. We however do not use xl cpupool-numa-split.

                      1 Reply Last reply Reply Quote 2
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        That's nice! It means it scales relatively well with many threads, it's a good result 🙂

                        ForzaF 1 Reply Last reply Reply Quote 1
                        • ForzaF Offline
                          Forza @olivierlambert
                          last edited by

                          @olivierlambert Can we get an updated XOA with this patch?

                          stormiS 1 Reply Last reply Reply Quote 1
                          • stormiS Offline
                            stormi Vates 🪐 XCP-ng Team @Forza
                            last edited by stormi

                            @Forza What patch are you referring to that would relate to XOA?

                            ForzaF 1 Reply Last reply Reply Quote 0
                            • ForzaF Offline
                              Forza @stormi
                              last edited by Forza

                              @stormi said in Epyc VM to VM networking slow:

                              @Forza What patch are you referring to that would relate to XOA?

                              It seems only recent kernels can take advantage of the improvements. From the blog post mentioned above:

                              The change only affects Linux guests. To make it effective, their kernel must support the feature which enables this fix.

                              Iperf inside XOA is much slower than other VMs (like the Ubundu 24.04 above).

                              When I run on the same host (pool master) against our NFS SR:

                              Ubuntu 24.04 (kernel 6.8):

                              # iperf3 -c 10.12.9.4
                              - - - - - - - - - - - - - - - - - - - - - - - - -
                              [ ID] Interval           Transfer     Bitrate         Retr
                              [  5]   0.00-10.00  sec  16.0 GBytes  13.7 Gbits/sec    0             sender
                              [  5]   0.00-10.00  sec  16.0 GBytes  13.7 Gbits/sec                  receiver
                              
                              # iperf3 -c 10.12.9.4 P4
                              - - - - - - - - - - - - - - - - - - - - - - - - -
                              [ ID] Interval           Transfer     Bitrate         Retr
                              [  5]   0.00-10.00  sec  6.26 GBytes  5.37 Gbits/sec  8861             sender
                              [  5]   0.00-10.00  sec  6.25 GBytes  5.37 Gbits/sec                  receiver
                              [  7]   0.00-10.00  sec  8.57 GBytes  7.36 Gbits/sec  8372             sender
                              [  7]   0.00-10.00  sec  8.57 GBytes  7.36 Gbits/sec                  receiver
                              [  9]   0.00-10.00  sec  9.05 GBytes  7.77 Gbits/sec  10192             sender
                              [  9]   0.00-10.00  sec  9.05 GBytes  7.77 Gbits/sec                  receiver
                              [ 11]   0.00-10.00  sec  6.12 GBytes  5.25 Gbits/sec  7144             sender
                              [ 11]   0.00-10.00  sec  6.11 GBytes  5.25 Gbits/sec                  receiver
                              [SUM]   0.00-10.00  sec  30.0 GBytes  25.8 Gbits/sec  34569             sender
                              [SUM]   0.00-10.00  sec  30.0 GBytes  25.8 Gbits/sec                  receiver
                              
                              

                              XOA 2025.08 (kernel 6.1):

                              # iperf3 -c 10.12.9.4
                              - - - - - - - - - - - - - - - - - - - - - - - - -
                              [ ID] Interval           Transfer     Bitrate         Retr
                              [  5]   0.00-10.00  sec  6.26 GBytes  5.37 Gbits/sec    0             sender
                              [  5]   0.00-10.00  sec  6.25 GBytes  5.37 Gbits/sec                  receiver
                              
                              # iperf3 -c 10.12.9.4 -P4
                              - - - - - - - - - - - - - - - - - - - - - - - - -
                              [ ID] Interval           Transfer     Bitrate         Retr
                              [  5]   0.00-10.00  sec  4.71 GBytes  4.05 Gbits/sec  3987             sender
                              [  5]   0.00-10.00  sec  4.71 GBytes  4.05 Gbits/sec                  receiver
                              [  7]   0.00-10.00  sec  4.61 GBytes  3.96 Gbits/sec  3086             sender
                              [  7]   0.00-10.00  sec  4.61 GBytes  3.96 Gbits/sec                  receiver
                              [  9]   0.00-10.00  sec  6.77 GBytes  5.81 Gbits/sec  7745             sender
                              [  9]   0.00-10.00  sec  6.77 GBytes  5.81 Gbits/sec                  receiver
                              [ 11]   0.00-10.00  sec  5.42 GBytes  4.65 Gbits/sec  629             sender
                              [ 11]   0.00-10.00  sec  5.42 GBytes  4.65 Gbits/sec                  receiver
                              [SUM]   0.00-10.00  sec  21.5 GBytes  18.5 Gbits/sec  15447             sender
                              [SUM]   0.00-10.00  sec  21.5 GBytes  18.5 Gbits/sec                  receiver
                              
                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by olivierlambert

                                XOA's kernel should have the capability already, as it's a Debian 12 with stock kernel. Also, the bottleneck is ONLY between VMs on the same host.

                                ForzaF 1 Reply Last reply Reply Quote 0
                                • ForzaF Offline
                                  Forza @olivierlambert
                                  last edited by Forza

                                  @olivierlambert said in Epyc VM to VM networking slow:

                                  XOA's kernel should have the capability already, as it's a Debian 12 with stock kernel. Also, the bottleneck is ONLY between VMs on the same host.

                                  OK, but then I do not understand the huge difference.

                                  Tested some VM-VM traffic on the same host:

                                  Ubuntu (kernel 6.8) -> Alpine (kernel 6.12): 13.7Gbit/s, P4 = 23.8 Gbit/s
                                  XOA (kernel 6.1) -> Ubuntu (kernel 6.8) : 5.5 Gbit/s, P4 = 17.5 Gbit/s
                                  XOA (kernel 6.1) -> Alpine (kernel 6.12): 11.9 Gbit/s, P4 = 13.6 Gbit/s

                                  And for measure against the bare metal NFS server:
                                  Alpine -> NFS SR (kernel 6.12): 13.7 Gbit/s, P4 = 23.4 Gbit/s

                                  bleaderB 1 Reply Last reply Reply Quote 0
                                  • bleaderB Offline
                                    bleader Vates 🪐 XCP-ng Team @Forza
                                    last edited by

                                    @Forza By default XOA VM has 2 vcpus, how many vcpus do your ubuntu have? Althrough iperf isn't running multithreaded in your test, there is one queue on the kernel side of the VM per vcpu to process packets.

                                    ForzaF 1 Reply Last reply Reply Quote 0
                                    • ForzaF Offline
                                      Forza @bleader
                                      last edited by Forza

                                      @bleader said in Epyc VM to VM networking slow:

                                      @Forza By default XOA VM has 2 vcpus, how many vcpus do your ubuntu have? Althrough iperf isn't running multithreaded in your test, there is one queue on the kernel side of the VM per vcpu to process packets.

                                      I have 8 CPUS on XOA, 4 CPUs in the Alpine VM and 6 in the Ubuntu VM

                                      P4 uses 4 threads with iperf3.

                                      @bleader Are you saying you do not see any performance differences between XOA and VMs with more recent kernels?

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        If you want to compare perf between kernels, do that with the same number of vCPUs ideally (and also the same env, ie different kernel versions with the same iperf version)

                                        ForzaF 1 Reply Last reply Reply Quote 0
                                        • ForzaF Offline
                                          Forza @olivierlambert
                                          last edited by Forza

                                          @olivierlambert said in Epyc VM to VM networking slow:

                                          If you want to compare perf between kernels, do that with the same number of vCPUs ideally (and also the same env, ie different kernel versions with the same iperf version)

                                          Sure. but in this context it is not that relevant as all other VMs got a boost while XOA didnt as much.

                                          Does the Debian 6.1 kernel that XOA uses have the backported fixes mentioned in https://xcp-ng.org/blog/2025/09/01/september-2025-maintenance-update-for-xcp-ng-8-3/

                                          Even if it has, it is clear that more resent kernels are much faster. Why not release a XOA with more recent kernels?

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by olivierlambert

                                            Kernel version could have an impact and being unrelated to the fix we provided. Get an even more recent kernel on your XOA to test (eg one from testing)

                                            ForzaF 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post