XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Updated XOA with kernel >5.3 to support nconnect nfs option

    Scheduled Pinned Locked Moved Xen Orchestra
    34 Posts 5 Posters 4.5k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ForzaF Offline
      Forza @olivierlambert
      last edited by Forza

      That would explain why the transfer rate is uneven, as some bandwidth and IO is used by the coalescing processes.

      Overall the transfer is somewhat faster. about 1.5 hours for 253GiB on 1Gbit/s connection. Before it was maybe 2ish hours.

      I lost the backup logs in XOA since I switched to the new version, but looking back in the Netdata stats from the backup server we can see the following pattern for the previous run of the same backup job. Just looking at the timestamps it looks pretty similar, while the bandwidth used looks less. Maybe some rounding errors in Netdata graphs? 😕

      579d6d7b-7e7b-4ba7-af22-285bb913d38b-image.png

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by olivierlambert

        It would be lovely to have the stats view on your NFS server with the BR mounted before and after adding nconnect, ideally with different number of parallel TCP connections 😄 In case you have some time to do it, that would be great!

        ForzaF 1 Reply Last reply Reply Quote 1
        • ForzaF Offline
          Forza @olivierlambert
          last edited by

          Changing nconnect requires restart of XOA (seems remount doesn't take a new value for nconnect), and I am heading home now, so I will try some additional benchmarks later in the week. 🙂

          Btw, does xcp-ng 8.3 support nconnect? 8.2 that I am using does not.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Nope, it will be just for the traffic between XO and the Backup Repo (BR)

            ForzaF 1 Reply Last reply Reply Quote 0
            • ForzaF Offline
              Forza @olivierlambert
              last edited by

              Ah, that's a shame, but reasonable for a point release. Maybe next major release?

              This was an interesting read regarding nconnect on Azure https://learn.microsoft.com/en-us/azure/storage/files/nfs-performance

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Yes, XCP-ng 9.x will likely able to use it 🙂

                ForzaF 1 Reply Last reply Reply Quote 0
                • ForzaF Offline
                  Forza @olivierlambert
                  last edited by Forza

                  @olivierlambert

                  I did a 3 x backup of a single VM hosted on an SSD drive on the same host as XOA is running, which is also pool master.

                  This is with nconnect=1:
                  a46273be-551b-443e-a3cc-4a15a11e1341-image.png

                  This is with nconnect=16:
                  1885cf86-af94-4383-ba56-10d458a7009e-image.png

                  The transfer speed according to XOA is slightly less, but looking at the bandwidth graph, it looks like the LACP bonded network on the storage server is reaching a higher max throughput.

                  I will test some more with incremental backups and see if there's a difference with them.

                  If we ignore the nconnect for a second, and just look at the graphs, it seems we have a lot of possibilities to improve the backup performance if we could make the transfer more even. What is causing this type of pattern? I do not believe any coalesce was happening during this test.

                  florentF 1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    That's a question for @florent

                    1 Reply Last reply Reply Quote 0
                    • florentF Offline
                      florent Vates 🪐 XO Team @Forza
                      last edited by

                      On the read side :

                      • the legacy mode : xapi build an expor from the vhd chain
                      • NBD : we read individual block on the storage repository

                      On the write Side :

                      • the default file mode : we write 1 big file per disk
                      • block mode : we write a compressed block per 2MB data

                      We don't have a lot of room on the legacy mode. The NBD + block gives us more freedom, and , instinctively , should gain more from nconnect, since we will read and write multiple small blocks in parallel

                      What mode are you using ?

                      @Forza said in Updated XOA with kernel >5.3 to support nconnect nfs option:

                      @olivierlambert

                      I did a 3 x backup of a single VM hosted on an SSD drive on the same host as XOA is running, which is also pool master.

                      This is with nconnect=1:
                      a46273be-551b-443e-a3cc-4a15a11e1341-image.png

                      This is with nconnect=16:
                      1885cf86-af94-4383-ba56-10d458a7009e-image.png

                      The transfer speed according to XOA is slightly less, but looking at the bandwidth graph, it looks like the LACP bonded network on the storage server is reaching a higher max throughput.

                      I will test some more with incremental backups and see if there's a difference with them.

                      If we ignore the nconnect for a second, and just look at the graphs, it seems we have a lot of possibilities to improve the backup performance if we could make the transfer more even. What is causing this type of pattern? I do not believe any coalesce was happening during this test.

                      ForzaF 1 Reply Last reply Reply Quote 0
                      • ForzaF Offline
                        Forza @florent
                        last edited by

                        @florent hi, for the full backup test above, I used normal mode with zstd enabled. There were no snapshots of the source VM and it was stored on local ssd storage on the pool master.

                        florentF 1 Reply Last reply Reply Quote 0
                        • A Offline
                          acomav
                          last edited by

                          Just replying to thank you for pointing this out. I have been having very poor backup speeds for over a month and this sorted it out.
                          I have only used nconnect=4 and 6 for my NFS shares.

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Hi @acomav

                            Can you give more details on your setup and the result before/after? We might use this to update our XO guide with the new values 🙂

                            A 1 Reply Last reply Reply Quote 0
                            • A Offline
                              acomav @olivierlambert
                              last edited by

                              @olivierlambert Hi Olivier. I'll see what I can do. I've spent the weekend cleaning up my backups and catching up on mirror transfers. Once completed, I'll do a few custom backups at various nconnect values.

                              1 Reply Last reply Reply Quote 2
                              • florentF Offline
                                florent Vates 🪐 XO Team @Forza
                                last edited by

                                @Forza said in Updated XOA with kernel >5.3 to support nconnect nfs option:

                                @florent hi, for the full backup test above, I used normal mode with zstd enabled. There were no snapshots of the source VM and it was stored on local ssd storage on the pool master.

                                can you test a backup on an empty remote with block mode ?

                                M 1 Reply Last reply Reply Quote 0
                                • M Offline
                                  manilx @florent
                                  last edited by

                                  @florent Did a test on my 8.3 homelab.
                                  Using XO on a Debian 12 VM.

                                  2 full (delta) backups using "nconnect=6" of all my 10 VM's

                                  With block defined at remote:
                                  ScreenShot 2024-04-29 at 10.38.11.png
                                  ScreenShot 2024-04-29 at 10.52.12.png

                                  without block (my normal mode):
                                  ScreenShot 2024-04-29 at 10.56.04.png
                                  ScreenShot 2024-04-29 at 10.51.57.png

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    I'm not sure to get it with your graph. Can you write down the average values for each scenario?

                                    M 1 Reply Last reply Reply Quote 0
                                    • M Offline
                                      manilx @olivierlambert
                                      last edited by

                                      @olivierlambert

                                      with block: 100-150 MB/s approx
                                      without: 200-300 MB/s

                                      Values from observed NAS traffic.

                                      ForzaF 1 Reply Last reply Reply Quote 0
                                      • ForzaF Offline
                                        Forza @manilx
                                        last edited by

                                        @manilx Do you mean block as NBD mode?

                                        M 1 Reply Last reply Reply Quote 0
                                        • M Offline
                                          manilx @Forza
                                          last edited by manilx

                                          @Forza Did another test:

                                          A: Remote as nfs share with "Store backup as multiple data blocks"
                                          B: same but without multiple data blocks

                                          1st backup to A,"Backup"
                                          ScreenShot 2024-04-29 at 13.45.47.png
                                          2nd backup to B, "Backup"
                                          ScreenShot 2024-04-29 at 13.45.57.png
                                          3rd backup to A, "Backup"
                                          ScreenShot 2024-04-29 at 13.46.04.png
                                          4th backup to B, "Delta Backup"
                                          ScreenShot 2024-04-29 at 13.46.10.png

                                          ScreenShot 2024-04-29 at 13.44.27.png

                                          M 1 Reply Last reply Reply Quote 0
                                          • M Offline
                                            manilx @manilx
                                            last edited by

                                            @manilx This was just one VM. Did not use NBD (forgot to turn on)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post