XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Our future backup code: test it!

    Scheduled Pinned Locked Moved Backup
    77 Posts 10 Posters 2.6k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Tristis OrisT Offline
      Tristis Oris Top contributor @florent
      last edited by

      @florent yep it works now.

      Tristis OrisT 1 Reply Last reply Reply Quote 0
      • J Offline
        john.c
        last edited by

        So is it this new backup code which the tape back up support will be plugging into?

        If so I have been thinking may be have it when doing the tape backup support, get the XOA to install the packages when required after prompting the user. Also have xo-server instruct the creation of a new disk image for attaching to xoa. The new disk image would act as storage or cache for the extra data used by tape drive backups.

        The tape drive related software wouldn't be installed by default, unless specified as such during deployment with the XOA deploy tool. But instead installed at user request, by selecting to use tape backup destination when setting up back up job, clicking on the shim widget about tape backups or restoring from an online Vates account backup of XOA settings.

        florentF 1 Reply Last reply Reply Quote 0
        • D Offline
          Davidj 0 @florent
          last edited by Davidj 0

          @florent I'm doing this from source.
          Running git pull --rebase seems to have fixed the build problems.

          1 Reply Last reply Reply Quote 2
          • florentF Offline
            florent Vates 🪐 XO Team @john.c
            last edited by florent

            @john.c the backup on tape is still on the roadmap but won't be for this step

            The main change is that we'll have to rework part of the code to ensure we write data sequentially + the management tools ( which backup is on which tape )

            You can try the FS on tape , but the performance penalty is heavy

            J 1 Reply Last reply Reply Quote 0
            • J Offline
              john.c @florent
              last edited by

              @florent said in Our future backup code: test it!:

              @john.c the backup on tape is still on the roadmap but won't be for this step

              The main change is that we'll have to rework part of the code to ensure we write data sequentially + the management tools ( which backup is on which tape )

              You can try the FS on tape , but the performance penalty is heavy

              Even if its not part of this rewrite some of what I wrote is my idea or view as to how it could be implemented later on. So that it's implemented in a way, that permits people who don't need tape back ups to still have small Xen Orchestra instances. But at the same time enable those who use tape backups, to have the capacity and an appropriately sized Xen Orchestra.

              1 Reply Last reply Reply Quote 0
              • D Offline
                Davidj 0
                last edited by

                Testing a continuous replication backup, I get Error: can't connect to any nbd client

                Running the same job on my regular XO works fine. Let me know what logs you want to see.

                mail (h2)
                
                   Snapshot
                   Start: 2025-04-10 18:18
                   End: 2025-04-10 18:18
                   h3 local storage (130.02 GiB free) - h3
                   Start: 2025-04-10 18:18
                
                Start: 2025-04-10 18:18
                End: 2025-04-10 18:18
                Duration: a few seconds
                Error: can't connect to any nbd client
                Type: delta
                
                1 Reply Last reply Reply Quote 0
                • Tristis OrisT Offline
                  Tristis Oris Top contributor @Tristis Oris
                  last edited by

                  @Tristis-Oris probably after https://github.com/vatesfr/xen-orchestra/pull/8432/files/04be6032180e7245a8db9c5f78177ac7999882c2

                  /@vates/generator-toolbox# yarn build
                  yarn run v1.22.22
                  $ tsc
                  /bin/sh: 1: tsc: not found
                  error Command failed with exit code 127.
                  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
                  

                  ok, npm install -g typescript

                  yarn build
                  yarn run v1.22.22
                  $ tsc
                  src/synchronized.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
                  
                  1 import assert from 'node:assert'
                                       ~~~~~~~~~~~~~
                  
                  src/synchronized.test.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
                  
                  1 import assert from 'node:assert'
                                       ~~~~~~~~~~~~~
                  
                  src/synchronized.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.
                  
                  2 import { suite, test } from 'node:test'
                                                ~~~~~~~~~~~
                  
                  src/throttle.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
                  
                  1 import assert from 'node:assert'
                                       ~~~~~~~~~~~~~
                  
                  src/throttle.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.
                  
                  2 import { suite, test } from 'node:test'
                                                ~~~~~~~~~~~
                  
                  src/throttle.test.mts:3:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
                  
                  3 import assert from 'node:assert'
                                       ~~~~~~~~~~~~~
                  
                  src/throttle.test.mts:7:11 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`.
                  
                  7     yield Buffer.allocUnsafe(chunkSize)
                              ~~~~~~
                  
                  src/timeout.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
                  
                  1 import assert from 'node:assert'
                                       ~~~~~~~~~~~~~
                  
                  src/timeout.test.mts:1:30 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.
                  
                  1 import { describe, it } from 'node:test'
                                                 ~~~~~~~~~~~
                  
                  src/timeout.test.mts:2:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.
                  
                  2 import assert from 'node:assert'
                                       ~~~~~~~~~~~~~
                  
                  
                  Found 10 errors in 6 files.
                  
                  Errors  Files
                       1  src/synchronized.mts:1
                       2  src/synchronized.test.mts:1
                       1  src/throttle.mts:1
                       3  src/throttle.test.mts:2
                       1  src/timeout.mts:1
                       2  src/timeout.test.mts:1
                  error Command failed with exit code 2.
                  info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
                  
                  fbeauchamp opened this pull request in vatesfr/xen-orchestra

                  closed feat(backups): use generator instead of streams for backup and replication #8432

                  Tristis OrisT 1 Reply Last reply Reply Quote 0
                  • Tristis OrisT Offline
                    Tristis Oris Top contributor @Tristis Oris
                    last edited by

                    @Tristis-Oris
                    from root directory.
                    /xen-orchestra
                    yarn - succeess
                    yarn build - failed
                    again to cd @vates/generator-toolbox/
                    yarn build

                    yarn run v1.22.22
                    $ tsc
                    Done in 5.50s.
                    

                    cd -
                    yarn build

                     Tasks:    29 successful, 29 total
                    Cached:    13 cached, 29 total
                      Time:    1m17.02s
                    
                    Done in 77.34s.
                    

                    again at root

                    cd packages/xo-server
                    mkdir -p ~/.config/xo-server
                    cp sample.config.toml ~/.config/xo-server/config.toml
                    yarn start
                    
                    1 Reply Last reply Reply Quote 0
                    • Tristis OrisT Offline
                      Tristis Oris Top contributor
                      last edited by

                      delta, first full.

                      • no NBD
                      2025-04-11T12:19:15.169Z xo:backups:worker INFO starting backup
                      2025-04-11T12:22:29.509Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
                        path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T121931Z.json',
                        actual: 27271364608,
                        expected: 27278188544
                      }
                      

                      2 vms but 1 log at cli.

                      delta

                      [2025-04-11T12:27:41.428Z xo:backups:worker INFO starting backup
                      2025-04-11T12:27:48.644Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
                        path: '/xo-vm-backups/f3cfdc19-0e6a-04d1-5651-82e1dbb25191/20250411T122746Z.json',
                        actual: 44040192,
                        expected: 46314496
                      }
                      2025-04-11T12:28:03.055Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
                        path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T122801Z.json',
                        actual: 46137344,
                        expected: 48412160
                      }
                      2025-04-11T12:28:03.081Z xo:backups:worker INFO backup has ended
                      2025-04-11T12:28:03.181Z xo:backups:worker INFO process will exit {
                        duration: 21752192,
                        exitCode: 0,
                        resourceUsage: {
                          userCPUTime: 3794057,
                          systemCPUTime: 468414,
                          maxRSS: 69592,
                          sharedMemorySize: 0,
                          unsharedDataSize: 0,
                          unsharedStackSize: 0,
                          minorPageFault: 42277,
                          majorPageFault: 0,
                          swappedOut: 0,
                          fsRead: 13624,
                          fsWrite: 185840,
                          ipcSent: 0,
                          ipcReceived: 0,
                          signalsCount: 0,
                          voluntaryContextSwitches: 3591,
                          involuntaryContextSwitches: 441
                        },
                        summary: { duration: '22s', cpuUsage: '20%', memoryUsage: '67.96 MiB' }
                      }
                      

                      can't get NBD to work. It enabled on nic, but not used for backups. XO settings copied from prod instance.

                      1 Reply Last reply Reply Quote 0
                      • D Offline
                        Davidj 0
                        last edited by

                        Like @Tristis-Oris , my first tests were done using a clone of my production XO. They failed with "No NBD" errors.

                        Creating a new backup works.

                        Tristis OrisT 1 Reply Last reply Reply Quote 0
                        • Tristis OrisT Offline
                          Tristis Oris Top contributor @Davidj 0
                          last edited by

                          @Davidj-0 No NBD usage even with new task( Main XO is on same host, same nic.

                          D 1 Reply Last reply Reply Quote 0
                          • D Offline
                            Davidj 0 @Tristis Oris
                            last edited by

                            @Tristis-Oris My setup is similar to yours, and I get similar errors on any existing backup job.

                            However, if I create a new backup job, then it works without any error.

                            ping @florent , maybe these data points are useful.

                            1 Reply Last reply Reply Quote 0
                            • florentF Offline
                              florent Vates 🪐 XO Team
                              last edited by

                              thanks for your tests . I will recheck NBD backup

                              florentF 1 Reply Last reply Reply Quote 0
                              • florentF Offline
                                florent Vates 🪐 XO Team @florent
                                last edited by

                                ih @Tristis-Oris and @Davidj-0 I pushed an update to better handle NBD error, please keep us informed

                                Tristis OrisT 1 Reply Last reply Reply Quote 0
                                • Tristis OrisT Offline
                                  Tristis Oris Top contributor @florent
                                  last edited by Tristis Oris

                                  @florent i'm not sure how to update from this branch(

                                  git checkout .
                                  git pull --ff-only
                                  yarn
                                  yarn build
                                  
                                  Updating 04be60321..6641580e6
                                  Fast-forward
                                   @xen-orchestra/xapi/disks/Xapi.mjs             | 17 ++++++++++-------
                                   @xen-orchestra/xapi/disks/XapiVhdCbt.mjs       |  3 ++-
                                   @xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs |  3 ---
                                   @xen-orchestra/xapi/disks/utils.mjs            | 29 ++++++++++++++++-------------
                                   4 files changed, 28 insertions(+), 24 deletions(-)
                                  
                                  git checkout feat_generator_backups
                                  Already on 'feat_generator_backups'
                                  Your branch is up to date with 'origin/feat_generator_backups'.
                                  
                                  
                                  git pull origin feat_generator_backups
                                  From https://github.com/vatesfr/xen-orchestra
                                   * branch                feat_generator_backups -> FETCH_HEAD
                                  Already up to date.
                                  

                                  but still no NBD at backup.

                                  1 Reply Last reply Reply Quote 0
                                  • Tristis OrisT Offline
                                    Tristis Oris Top contributor
                                    last edited by

                                    i see significant speed difference from master branch.

                                    prod
                                    NBD
                                    160.71 MiB
                                    no NBD
                                    169

                                    test
                                    NBD (broken)
                                    110-130 MiB
                                    no NBD
                                    115

                                    florentF 1 Reply Last reply Reply Quote 0
                                    • florentF Offline
                                      florent Vates 🪐 XO Team @Tristis Oris
                                      last edited by

                                      @Tristis-Oris ouch that is quite costly
                                      Can you describe which backup you run ?
                                      Can you check if the duration is different (maybe this is a measurement error and not a slow speed) ?

                                      Tristis OrisT 1 Reply Last reply Reply Quote 0
                                      • Tristis OrisT Offline
                                        Tristis Oris Top contributor @florent
                                        last edited by

                                        @florent

                                        1 vm, 1 storage, NBD connections: 1. delta, first full.

                                        Duration: 3 minutes
                                        Size: 26.54 GiB
                                        Speed: 160.71 MiB/s

                                        Duration: 4 minutes
                                        Size: 26.53 GiB
                                        Speed: 113.74 MiB/s

                                        florentF 1 Reply Last reply Reply Quote 1
                                        • florentF Offline
                                          florent Vates 🪐 XO Team @Tristis Oris
                                          last edited by

                                          @Tristis-Oris Do you have the same performance without NBD ?
                                          Does your storage use blocks ?

                                          Tristis OrisT 1 Reply Last reply Reply Quote 0
                                          • Tristis OrisT Offline
                                            Tristis Oris Top contributor @florent
                                            last edited by

                                            @florent blocks! i forgot about that checkbox)

                                            better but not same.

                                            Duration: 3 minutes
                                            Size: 26.53 GiB
                                            Speed: 143.7 MiB/s

                                            Speed: 146.09 MiB/s

                                            florentF 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post