XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Our future backup code: test it!

    Scheduled Pinned Locked Moved Backup
    128 Posts 10 Posters 7.7k Views 11 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Tristis OrisT Offline
      Tristis Oris Top contributor @Tristis Oris
      last edited by

      @Tristis-Oris probably after https://github.com/vatesfr/xen-orchestra/pull/8432/files/04be6032180e7245a8db9c5f78177ac7999882c2

      /@vates/generator-toolbox# yarn build
      yarn run v1.22.22
      $ tsc
      /bin/sh: 1: tsc: not found
      error Command failed with exit code 127.
      info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
      

      ok, npm install -g typescript

      yarn build
      yarn run v1.22.22
      $ tsc
      src/synchronized.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
      
      1 import assert from 'node:assert'
                           ~~~~~~~~~~~~~
      
      src/synchronized.test.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
      
      1 import assert from 'node:assert'
                           ~~~~~~~~~~~~~
      
      src/synchronized.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.
      
      2 import { suite, test } from 'node:test'
                                    ~~~~~~~~~~~
      
      src/throttle.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
      
      1 import assert from 'node:assert'
                           ~~~~~~~~~~~~~
      
      src/throttle.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.
      
      2 import { suite, test } from 'node:test'
                                    ~~~~~~~~~~~
      
      src/throttle.test.mts:3:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
      
      3 import assert from 'node:assert'
                           ~~~~~~~~~~~~~
      
      src/throttle.test.mts:7:11 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`.
      
      7     yield Buffer.allocUnsafe(chunkSize)
                  ~~~~~~
      
      src/timeout.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
      
      1 import assert from 'node:assert'
                           ~~~~~~~~~~~~~
      
      src/timeout.test.mts:1:30 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.
      
      1 import { describe, it } from 'node:test'
                                     ~~~~~~~~~~~
      
      src/timeout.test.mts:2:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
      
      2 import assert from 'node:assert'
                           ~~~~~~~~~~~~~
      
      
      Found 10 errors in 6 files.
      
      Errors  Files
           1  src/synchronized.mts:1
           2  src/synchronized.test.mts:1
           1  src/throttle.mts:1
           3  src/throttle.test.mts:2
           1  src/timeout.mts:1
           2  src/timeout.test.mts:1
      error Command failed with exit code 2.
      info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
      
      fbeauchamp opened this pull request in vatesfr/xen-orchestra

      closed feat(backups): use generator instead of streams for backup and replication #8432

      Tristis OrisT 1 Reply Last reply Reply Quote 0
      • Tristis OrisT Offline
        Tristis Oris Top contributor @Tristis Oris
        last edited by

        @Tristis-Oris
        from root directory.
        /xen-orchestra
        yarn - succeess
        yarn build - failed
        again to cd @vates/generator-toolbox/
        yarn build

        yarn run v1.22.22
        $ tsc
        Done in 5.50s.
        

        cd -
        yarn build

         Tasks:    29 successful, 29 total
        Cached:    13 cached, 29 total
          Time:    1m17.02s
        
        Done in 77.34s.
        

        again at root

        cd packages/xo-server
        mkdir -p ~/.config/xo-server
        cp sample.config.toml ~/.config/xo-server/config.toml
        yarn start
        
        1 Reply Last reply Reply Quote 0
        • Tristis OrisT Offline
          Tristis Oris Top contributor
          last edited by

          delta, first full.

          • no NBD
          2025-04-11T12:19:15.169Z xo:backups:worker INFO starting backup
          2025-04-11T12:22:29.509Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
            path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T121931Z.json',
            actual: 27271364608,
            expected: 27278188544
          }
          

          2 vms but 1 log at cli.

          delta

          [2025-04-11T12:27:41.428Z xo:backups:worker INFO starting backup
          2025-04-11T12:27:48.644Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
            path: '/xo-vm-backups/f3cfdc19-0e6a-04d1-5651-82e1dbb25191/20250411T122746Z.json',
            actual: 44040192,
            expected: 46314496
          }
          2025-04-11T12:28:03.055Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
            path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T122801Z.json',
            actual: 46137344,
            expected: 48412160
          }
          2025-04-11T12:28:03.081Z xo:backups:worker INFO backup has ended
          2025-04-11T12:28:03.181Z xo:backups:worker INFO process will exit {
            duration: 21752192,
            exitCode: 0,
            resourceUsage: {
              userCPUTime: 3794057,
              systemCPUTime: 468414,
              maxRSS: 69592,
              sharedMemorySize: 0,
              unsharedDataSize: 0,
              unsharedStackSize: 0,
              minorPageFault: 42277,
              majorPageFault: 0,
              swappedOut: 0,
              fsRead: 13624,
              fsWrite: 185840,
              ipcSent: 0,
              ipcReceived: 0,
              signalsCount: 0,
              voluntaryContextSwitches: 3591,
              involuntaryContextSwitches: 441
            },
            summary: { duration: '22s', cpuUsage: '20%', memoryUsage: '67.96 MiB' }
          }
          

          can't get NBD to work. It enabled on nic, but not used for backups. XO settings copied from prod instance.

          1 Reply Last reply Reply Quote 0
          • D Offline
            Davidj 0
            last edited by

            Like @Tristis-Oris , my first tests were done using a clone of my production XO. They failed with "No NBD" errors.

            Creating a new backup works.

            Tristis OrisT 1 Reply Last reply Reply Quote 0
            • Tristis OrisT Offline
              Tristis Oris Top contributor @Davidj 0
              last edited by

              @Davidj-0 No NBD usage even with new task( Main XO is on same host, same nic.

              D 1 Reply Last reply Reply Quote 0
              • D Offline
                Davidj 0 @Tristis Oris
                last edited by

                @Tristis-Oris My setup is similar to yours, and I get similar errors on any existing backup job.

                However, if I create a new backup job, then it works without any error.

                ping @florent , maybe these data points are useful.

                1 Reply Last reply Reply Quote 0
                • florentF Offline
                  florent Vates πŸͺ XO Team
                  last edited by

                  thanks for your tests . I will recheck NBD backup

                  florentF 1 Reply Last reply Reply Quote 0
                  • florentF Offline
                    florent Vates πŸͺ XO Team @florent
                    last edited by

                    ih @Tristis-Oris and @Davidj-0 I pushed an update to better handle NBD error, please keep us informed

                    Tristis OrisT 1 Reply Last reply Reply Quote 0
                    • Tristis OrisT Offline
                      Tristis Oris Top contributor @florent
                      last edited by Tristis Oris

                      @florent i'm not sure how to update from this branch(

                      git checkout .
                      git pull --ff-only
                      yarn
                      yarn build
                      
                      Updating 04be60321..6641580e6
                      Fast-forward
                       @xen-orchestra/xapi/disks/Xapi.mjs             | 17 ++++++++++-------
                       @xen-orchestra/xapi/disks/XapiVhdCbt.mjs       |  3 ++-
                       @xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs |  3 ---
                       @xen-orchestra/xapi/disks/utils.mjs            | 29 ++++++++++++++++-------------
                       4 files changed, 28 insertions(+), 24 deletions(-)
                      
                      git checkout feat_generator_backups
                      Already on 'feat_generator_backups'
                      Your branch is up to date with 'origin/feat_generator_backups'.
                      
                      
                      git pull origin feat_generator_backups
                      From https://github.com/vatesfr/xen-orchestra
                       * branch                feat_generator_backups -> FETCH_HEAD
                      Already up to date.
                      

                      but still no NBD at backup.

                      1 Reply Last reply Reply Quote 0
                      • Tristis OrisT Offline
                        Tristis Oris Top contributor
                        last edited by

                        i see significant speed difference from master branch.

                        prod
                        NBD
                        160.71 MiB
                        no NBD
                        169

                        test
                        NBD (broken)
                        110-130 MiB
                        no NBD
                        115

                        florentF 1 Reply Last reply Reply Quote 0
                        • florentF Offline
                          florent Vates πŸͺ XO Team @Tristis Oris
                          last edited by

                          @Tristis-Oris ouch that is quite costly
                          Can you describe which backup you run ?
                          Can you check if the duration is different (maybe this is a measurement error and not a slow speed) ?

                          Tristis OrisT 1 Reply Last reply Reply Quote 0
                          • Tristis OrisT Offline
                            Tristis Oris Top contributor @florent
                            last edited by

                            @florent

                            1 vm, 1 storage, NBD connections: 1. delta, first full.

                            Duration: 3 minutes
                            Size: 26.54 GiB
                            Speed: 160.71 MiB/s

                            Duration: 4 minutes
                            Size: 26.53 GiB
                            Speed: 113.74 MiB/s

                            florentF 1 Reply Last reply Reply Quote 1
                            • florentF Offline
                              florent Vates πŸͺ XO Team @Tristis Oris
                              last edited by

                              @Tristis-Oris Do you have the same performance without NBD ?
                              Does your storage use blocks ?

                              Tristis OrisT 1 Reply Last reply Reply Quote 0
                              • Tristis OrisT Offline
                                Tristis Oris Top contributor @florent
                                last edited by

                                @florent blocks! i forgot about that checkbox)

                                better but not same.

                                Duration: 3 minutes
                                Size: 26.53 GiB
                                Speed: 143.7 MiB/s

                                Speed: 146.09 MiB/s

                                florentF 1 Reply Last reply Reply Quote 0
                                • florentF Offline
                                  florent Vates πŸͺ XO Team @Tristis Oris
                                  last edited by

                                  @Tristis-Oris I made a little change, can you update (like the last time ) and retest ?

                                  Tristis OrisT 1 Reply Last reply Reply Quote 0
                                  • Tristis OrisT Offline
                                    Tristis Oris Top contributor @florent
                                    last edited by

                                    @florent
                                    same speed after fix.
                                    142.56 MiB/s - 145.63 MiB/s

                                    maybe i miss something else? same database as prod, only another LUN connected for backups.

                                    florentF 1 Reply Last reply Reply Quote 0
                                    • florentF Offline
                                      florent Vates πŸͺ XO Team @Tristis Oris
                                      last edited by florent

                                      @Tristis-Oris no it's on our end

                                      Could you retry nbd + target a block based directory ?
                                      ON my test setup, with the latest changes I get better speed than master ( 190MB/s per disk vs 130-170 depending on the run and settings on master)

                                      I got quite a huge variation between the same runs (40MB/s)

                                      Tristis OrisT 1 Reply Last reply Reply Quote 0
                                      • Tristis OrisT Offline
                                        Tristis Oris Top contributor @florent
                                        last edited by

                                        @florent yep, now it equal. Maybe my hw bottleneck? i can also check with ssd storage to see max speed.

                                        Duration: 3 minutes
                                        Size: 26.53 GiB
                                        Speed: 157.78 MiB/s
                                        Speed: 149.39 MiB/s
                                        Speed: 163.76 MiB/s

                                        No more errors incorrect backup size in metadata.

                                        But still no NBD(

                                        florentF 1 Reply Last reply Reply Quote 0
                                        • florentF Offline
                                          florent Vates πŸͺ XO Team @Tristis Oris
                                          last edited by

                                          @Tristis-Oris that is already a good news.

                                          I pushed an additional fix : the NBD info was not shown on the UI
                                          79c6d837-2b12-4b66-b66d-59f887c088a5-image.png

                                          Tristis OrisT 1 Reply Last reply Reply Quote 0
                                          • Tristis OrisT Offline
                                            Tristis Oris Top contributor @florent
                                            last edited by

                                            @florent haha, now it works too good)

                                            60174b41-7179-4e31-8d02-bf0c9ec405a6-ΠΈΠ·ΠΎΠ±Ρ€Π°ΠΆΠ΅Π½ΠΈΠ΅.png

                                            39e6a5fa-942f-4895-933d-51bd765ddde5-ΠΈΠ·ΠΎΠ±Ρ€Π°ΠΆΠ΅Π½ΠΈΠ΅.png

                                            983f9e40-8f8b-417e-8360-85d195aeba2b-ΠΈΠ·ΠΎΠ±Ρ€Π°ΠΆΠ΅Π½ΠΈΠ΅.png

                                            Tristis OrisT 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post