XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Updated XOA with kernel >5.3 to support nconnect nfs option

    Scheduled Pinned Locked Moved Xen Orchestra
    34 Posts 5 Posters 4.6k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      XOA is based on Debian 12. I'm not against doing a custom kernel if it's really a big bonus in terms of performance 🙂

      ForzaF 1 Reply Last reply Reply Quote 0
      • ForzaF Offline
        Forza @olivierlambert
        last edited by Forza

        @olivierlambert said in Updated XOA with kernel >5.3 to support nconnect nfs option:

        XOA is based on Debian 12. I'm not against doing a custom kernel if it's really a big bonus in terms of performance 🙂

        Debian 12 should use kernel 6.1, so that would be ok. Maybe I am having an old XOA as my kernel is 4.19. This is what shows when I login:

        login as: xoa
        xoa@<redacted>'s password:
        Linux <redacted> 4.19.0-13-amd64 #1 SMP Debian 4.19.160-2 (2020-11-28) x86_64
         __   __             ____           _               _
         \ \ / /            / __ \         | |             | |
          \ V / ___ _ __   | |  | |_ __ ___| |__   ___  ___| |_ _ __ __ _
           > < / _ \ '_ \  | |  | | '__/ __| '_ \ / _ \/ __| __| '__/ _` |
          / . \  __/ | | | | |__| | | | (__| | | |  __/\__ \ |_| | | (_| |
         /_/ \_\___|_| |_|  \____/|_|  \___|_| |_|\___||___/\__|_|  \__,_|
        
        Welcome to XOA Unified Edition, with Pro Support.
        
        * Restart XO: sudo systemctl restart xo-server.service
        * Display status: sudo systemctl status xo-server.service
        * Display logs: sudo journalctl -u xo-server.service
        * Register your XOA: sudo xoa-updater --register
        * Update your XOA: sudo xoa-updater --upgrade
        
        OFFICIAL XOA DOCUMENTATION HERE: https://xen-orchestra.com/docs/xoa.html
        
        Support available at https://xen-orchestra.com/#!/member/support
        
        In case of issues, use `xoa check` for a quick health check.
        
        Build number: 21.01.02
        
        Based on Debian GNU/Linux 10 (Stable) 64bits in PVHVM mode
        
        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Your XOA is old, you can deploy a new one and do a compare by mounting the backup repo with and without the option.

          ForzaF 1 Reply Last reply Reply Quote 0
          • ForzaF Offline
            Forza @olivierlambert
            last edited by

            I will do.

            It would be nice to have an notice that a new XOA is available, in addition to the normal updates. Perhaps something on the https://xoa/#/xoa/support page?

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              We notified that when it happened in our release blog post, which we advertise every month via our XO newsletter. See https://xen-orchestra.com/blog/xen-orchestra-5-92/#🪐-new-xoa for more details

              ForzaF 1 Reply Last reply Reply Quote 0
              • ForzaF Offline
                Forza @olivierlambert
                last edited by Forza

                I usually read the blog posts, but I must have missed it that time. 🙂

                As for performance it is a little difficult to test i a structured manner, but the initial incremental backup seems to run very well. Normally I have about 60-90MB/s as max before, but also often around 20-30MB/s. This time the backup took 6 minutes, and normal is around 15 minutes. This is over a 1 Gbit/s network.

                1d35feab-ed67-4b17-b666-3f5e18367089-image.png

                ForzaF 1 Reply Last reply Reply Quote 0
                • ForzaF Offline
                  Forza @Forza
                  last edited by Forza

                  Testing full backups. Seems at least XOA is saturating the link for larger VMs. It does have quite a bit of delay outside of the transfer itself, so the overall BW that is logged in the web interface is lower.

                  This is bmon running inside XOA.
                  f7c57582-9427-4eff-ad7c-52afa7ae4ccc-image.png

                  ee96db72-9a48-47de-9876-e3efd4f56c91-image.png

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    That's interesting 🙂 Are you checking the speed via in XO graphs on the XOA VM?

                    ForzaF 1 Reply Last reply Reply Quote 0
                    • ForzaF Offline
                      Forza @olivierlambert
                      last edited by

                      @olivierlambert
                      Yes, I am checking on the XOA VM itself.

                      Here is a view over the XOA VM. You can see that the full backup was started around 3:10 pm:
                      12d42d0a-59e5-4c51-b147-5a6c828e8251-image.png

                      This is the NFS stats from the backup server (Remote) for the same period:
                      3825e305-9842-49f8-bb1b-ed1f6e787b77-image.png

                      Why is the server doing reads? Maybe coalesce? But is there coalesce on full backups?

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Thanks for the stats 🙂 About the read, yes, it's all the workers coalescing the chain on the BR 🙂

                        ForzaF 1 Reply Last reply Reply Quote 0
                        • ForzaF Offline
                          Forza @olivierlambert
                          last edited by Forza

                          That would explain why the transfer rate is uneven, as some bandwidth and IO is used by the coalescing processes.

                          Overall the transfer is somewhat faster. about 1.5 hours for 253GiB on 1Gbit/s connection. Before it was maybe 2ish hours.

                          I lost the backup logs in XOA since I switched to the new version, but looking back in the Netdata stats from the backup server we can see the following pattern for the previous run of the same backup job. Just looking at the timestamps it looks pretty similar, while the bandwidth used looks less. Maybe some rounding errors in Netdata graphs? 😕

                          579d6d7b-7e7b-4ba7-af22-285bb913d38b-image.png

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by olivierlambert

                            It would be lovely to have the stats view on your NFS server with the BR mounted before and after adding nconnect, ideally with different number of parallel TCP connections 😄 In case you have some time to do it, that would be great!

                            ForzaF 1 Reply Last reply Reply Quote 1
                            • ForzaF Offline
                              Forza @olivierlambert
                              last edited by

                              Changing nconnect requires restart of XOA (seems remount doesn't take a new value for nconnect), and I am heading home now, so I will try some additional benchmarks later in the week. 🙂

                              Btw, does xcp-ng 8.3 support nconnect? 8.2 that I am using does not.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Nope, it will be just for the traffic between XO and the Backup Repo (BR)

                                ForzaF 1 Reply Last reply Reply Quote 0
                                • ForzaF Offline
                                  Forza @olivierlambert
                                  last edited by

                                  Ah, that's a shame, but reasonable for a point release. Maybe next major release?

                                  This was an interesting read regarding nconnect on Azure https://learn.microsoft.com/en-us/azure/storage/files/nfs-performance

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    Yes, XCP-ng 9.x will likely able to use it 🙂

                                    ForzaF 1 Reply Last reply Reply Quote 0
                                    • ForzaF Offline
                                      Forza @olivierlambert
                                      last edited by Forza

                                      @olivierlambert

                                      I did a 3 x backup of a single VM hosted on an SSD drive on the same host as XOA is running, which is also pool master.

                                      This is with nconnect=1:
                                      a46273be-551b-443e-a3cc-4a15a11e1341-image.png

                                      This is with nconnect=16:
                                      1885cf86-af94-4383-ba56-10d458a7009e-image.png

                                      The transfer speed according to XOA is slightly less, but looking at the bandwidth graph, it looks like the LACP bonded network on the storage server is reaching a higher max throughput.

                                      I will test some more with incremental backups and see if there's a difference with them.

                                      If we ignore the nconnect for a second, and just look at the graphs, it seems we have a lot of possibilities to improve the backup performance if we could make the transfer more even. What is causing this type of pattern? I do not believe any coalesce was happening during this test.

                                      florentF 1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        That's a question for @florent

                                        1 Reply Last reply Reply Quote 0
                                        • florentF Offline
                                          florent Vates 🪐 XO Team @Forza
                                          last edited by

                                          On the read side :

                                          • the legacy mode : xapi build an expor from the vhd chain
                                          • NBD : we read individual block on the storage repository

                                          On the write Side :

                                          • the default file mode : we write 1 big file per disk
                                          • block mode : we write a compressed block per 2MB data

                                          We don't have a lot of room on the legacy mode. The NBD + block gives us more freedom, and , instinctively , should gain more from nconnect, since we will read and write multiple small blocks in parallel

                                          What mode are you using ?

                                          @Forza said in Updated XOA with kernel >5.3 to support nconnect nfs option:

                                          @olivierlambert

                                          I did a 3 x backup of a single VM hosted on an SSD drive on the same host as XOA is running, which is also pool master.

                                          This is with nconnect=1:
                                          a46273be-551b-443e-a3cc-4a15a11e1341-image.png

                                          This is with nconnect=16:
                                          1885cf86-af94-4383-ba56-10d458a7009e-image.png

                                          The transfer speed according to XOA is slightly less, but looking at the bandwidth graph, it looks like the LACP bonded network on the storage server is reaching a higher max throughput.

                                          I will test some more with incremental backups and see if there's a difference with them.

                                          If we ignore the nconnect for a second, and just look at the graphs, it seems we have a lot of possibilities to improve the backup performance if we could make the transfer more even. What is causing this type of pattern? I do not believe any coalesce was happening during this test.

                                          ForzaF 1 Reply Last reply Reply Quote 0
                                          • ForzaF Offline
                                            Forza @florent
                                            last edited by

                                            @florent hi, for the full backup test above, I used normal mode with zstd enabled. There were no snapshots of the source VM and it was stored on local ssd storage on the pool master.

                                            florentF 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post