XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Epyc VM to VM networking slow

    Scheduled Pinned Locked Moved Compute
    237 Posts 25 Posters 108.4k Views 28 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ForzaF Offline
      Forza
      last edited by Forza

      It's a good discovery that having XOA outside the pool can make the backup performance much better.

      How is the problem solving going for the root cause? We too have quite poor network performance and would really like to see the end of this. Can we get a summary of the actions taken so far and what the prognosis is for a solution?

      Did anyone try plain Xen on a new 6.x kernel to see if the networking is the same there?

      1 Reply Last reply Reply Quote 1
      • TeddyAstieT Offline
        TeddyAstie Vates 🪐 XCP-ng Team Xen Guru
        last edited by

        Is there a difference when running iperf3 with -C bbr flag on the client side ?
        In my testing with AMD EPYC and some other CPUs, results are more consistent overall with BBR, and better on the AMD EPYC side (but no miracle, it's still far from perfect).

        AMD EPYC 7262
        vm to vm, 4 threads, iperf3
        Without BBR : 4.5-6 Gbps (sometimes more; varies a lot)
        With BBR : 7-8 Gbps

        ForzaF 1 Reply Last reply Reply Quote 0
        • ForzaF Offline
          Forza @TeddyAstie
          last edited by Forza

          @TeddyAstie That is interesting. I had a look. The default seems to be cubic, but bbr is available using modprobe tcp_bbr. I also wonder if different queuing disciplines (tc qdisc) can help. For example mqprio that spreads packes across the available NIC HW queues?

          TeddyAstieT 1 Reply Last reply Reply Quote 0
          • TeddyAstieT Offline
            TeddyAstie Vates 🪐 XCP-ng Team Xen Guru @Forza
            last edited by TeddyAstie

            @Forza the default one seems cubic which in my testing causes chaotic (either good or bad) network performance on XCP-ng (even on non-EPYC platforms) where BBR is more consistent (and also better on AMD EPYC).

            I also wonder if different queuing disciplines (tc qdisc) can help. For example mqprio that spreads packes across the available NIC HW queues?

            Regarding PV network, I don't think queue management will change anything as netfront/netback is single-queue. It's multi-queue so maybe it changes something.

            D 1 Reply Last reply Reply Quote 0
            • TeddyAstieT Offline
              TeddyAstie Vates 🪐 XCP-ng Team Xen Guru
              last edited by TeddyAstie

              For those who have AMD EPYC 7003 (Zen 3 EPYCs), you may find in Processor settings in firmware

              • Enhanced REP MOVSB/STOSB (ERMS)
              • Fast Short REP MOVSB (FSRM)

              Which is apparently disabled by default.
              It could be interesting to enable them and see if it changes anything performance-wise. I am not sure if it's just for showing a flag, or if it changes anything in the CPU behavior though.

              You can also try REP-MOV/STOS Streaming to see it changes anything too.

              1 Reply Last reply Reply Quote 0
              • D Offline
                dknight-bg @TeddyAstie
                last edited by

                @TeddyAstie

                I'm attaching results with Epyc 9004 / AS-1015CS-TNR-EU / MB: H13SSW

                user cpu family market v2v 1T v2v 4T h2v 1T h2v 4T notes
                dknight-bg EPYC 9354P Zen4 server 5.10 G (130/150/250) 6.24 G (131/254/348) 11.1 G (0/131/216) 11.1 G (0/187/302) Disabled: Enhanced REP MOVSB/STOSB (ERMS), Fast Short REP MOVSB (FSRM)
                dknight-bg EPYC 9354P Zen4 server 6.71 G (112/223/269) 7.11 G (122/261/342) 11.3 G (0/145/190) 11.5G (0/179/282) Enabled: Enhanced REP MOVSB/STOSB (ERMS), Fast Short REP MOVSB (FSRM)

                I couldn't find a setting for REP-MOV/STOPS Streaming in the BIOS, nor in the MB manual.

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  In VM to VM it's already a "free" +30% perf for 1T, impressive. Also 14% for 4T, not bad.

                  N 1 Reply Last reply Reply Quote 0
                  • N Offline
                    nicols @olivierlambert
                    last edited by nicols

                    @olivierlambert yes, but this is only a small improvement to overall problem. Still, compared to similar intel platform, v2v network performance is very low. Are there any news for the final solution to this problem?

                    M 1 Reply Last reply Reply Quote 0
                    • M Offline
                      manilx @nicols
                      last edited by

                      @nicols We're on 2 HP's with AMD EPYC 7543P 32-Core and it is a pain.
                      Had to deploy Intel server just for running the backups, which improved a lot (but that's stupid with 2 beasts of hosts)

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        No obvious solution yet, it's likely due to an architecture problem on AMD, because of CCDs and how CPUs are made. So the solution (if there's any) will be likely a sum of various small improvements to make it bearable.

                        I'm going to Santa Clara to discuss that with AMD directly (among other things).

                        M ForzaF 2 Replies Last reply Reply Quote 2
                        • M Offline
                          manilx @olivierlambert
                          last edited by

                          @olivierlambert No more EPYC's here, that's for sure. In the future it'll be Intel again (even if they then belong to Broadcom and TMC :p)

                          1 Reply Last reply Reply Quote 0
                          • ForzaF Offline
                            Forza @olivierlambert
                            last edited by

                            @olivierlambert said in Epyc VM to VM networking slow:

                            No obvious solution yet, it's likely due to an architecture problem on AMD, because of CCDs and how CPUs are made. So the solution (if there's any) will be likely a sum of various small improvements to make it bearable.

                            I'm going to Santa Clara to discuss that with AMD directly (among other things).

                            Do we have other data to back this? The issue is not really common outside of Xen. I do hope some solution comes out from the meeting with AMD.

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.

                              D ForzaF 2 Replies Last reply Reply Quote 0
                              • D Offline
                                Davidj 0 @olivierlambert
                                last edited by

                                @olivierlambert
                                Can we rule out extra_guest_irqs as the root cause of this problem?

                                https://docs.xcp-ng.org/compute/#nvme-storage-devices-on-linux

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  It's probably completely unrelated, but feel free to test 🙂

                                  1 Reply Last reply Reply Quote 0
                                  • ForzaF Offline
                                    Forza @olivierlambert
                                    last edited by

                                    @olivierlambert said in Epyc VM to VM networking slow:

                                    If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.

                                    Hi, is there anything new to report on this? We have very powerful machines, but unfortunately limited by this stubborn issue.

                                    M TeddyAstieT 2 Replies Last reply Reply Quote 0
                                    • M Offline
                                      manilx @Forza
                                      last edited by

                                      @Forza Dito. A 15.000€ EPYC HP monster is slower than a 1.600€ Protectli Intel...
                                      This is a joke and had we known this we'd NEVER jumped on the AMD wagon 😞

                                      1 Reply Last reply Reply Quote 0
                                      • TeddyAstieT Offline
                                        TeddyAstie Vates 🪐 XCP-ng Team Xen Guru @Forza
                                        last edited by

                                        @Forza said in Epyc VM to VM networking slow:

                                        olivierlambert said in Epyc VM to VM networking slow:

                                        If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.

                                        Hi, is there anything new to report on this? We have very powerful machines, but unfortunately limited by this stubborn issue.

                                        Can you test https://xcp-ng.org/forum/topic/10862/early-testable-pvh-support ?

                                        We observe very significant improvements on AMD EPYC with PVH.

                                        We're still pin-pointing the issue with HVM, the current hypothesis is a issue regarding memory typing (grant-table accessed as uncacheable(UC) which is very slow) related to grant-table positionning in HVM.

                                        ForzaF 1 Reply Last reply Reply Quote 0
                                        • ForzaF Offline
                                          Forza @TeddyAstie
                                          last edited by

                                          @TeddyAstie Unfortunately not. This is a production pool on 8.2.1 so I do not want to try too experimental things.

                                          Do we know if the issue happens on plain Xen on a modern (6.12-15) dom0 kernel?

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by olivierlambert

                                            It's on any Xen and Linux version. Vates is now the spearhead on finding the problem and a solution, there's no upstream with a fix anywhere.

                                            ForzaF 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post