Our future backup code: test it!

john.c

@florent said in Our future backup code: test it!:

@john.c the backup on tape is still on the roadmap but won't be for this step

The main change is that we'll have to rework part of the code to ensure we write data sequentially + the management tools ( which backup is on which tape )

You can try the FS on tape , but the performance penalty is heavy

Even if its not part of this rewrite some of what I wrote is my idea or view as to how it could be implemented later on. So that it's implemented in a way, that permits people who don't need tape back ups to still have small Xen Orchestra instances. But at the same time enable those who use tape backups, to have the capacity and an appropriately sized Xen Orchestra.

Davidj 0

Testing a continuous replication backup, I get Error: can't connect to any nbd client

Running the same job on my regular XO works fine. Let me know what logs you want to see.

mail (h2)

   Snapshot
   Start: 2025-04-10 18:18
   End: 2025-04-10 18:18
   h3 local storage (130.02 GiB free) - h3
   Start: 2025-04-10 18:18

Start: 2025-04-10 18:18
End: 2025-04-10 18:18
Duration: a few seconds
Error: can't connect to any nbd client
Type: delta

Tristis Oris

@Tristis-Oris probably after https://github.com/vatesfr/xen-orchestra/pull/8432/files/04be6032180e7245a8db9c5f78177ac7999882c2

/@vates/generator-toolbox# yarn build
yarn run v1.22.22
$ tsc
/bin/sh: 1: tsc: not found
error Command failed with exit code 127.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

ok, npm install -g typescript

yarn build
yarn run v1.22.22
$ tsc
src/synchronized.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.

1 import assert from 'node:assert'
                     ~~~~~~~~~~~~~

src/synchronized.test.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.

1 import assert from 'node:assert'
                     ~~~~~~~~~~~~~

src/synchronized.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.

2 import { suite, test } from 'node:test'
                              ~~~~~~~~~~~

src/throttle.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.

1 import assert from 'node:assert'
                     ~~~~~~~~~~~~~

src/throttle.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.

2 import { suite, test } from 'node:test'
                              ~~~~~~~~~~~

src/throttle.test.mts:3:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.

3 import assert from 'node:assert'
                     ~~~~~~~~~~~~~

src/throttle.test.mts:7:11 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`.

7     yield Buffer.allocUnsafe(chunkSize)
            ~~~~~~

src/timeout.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.

1 import assert from 'node:assert'
                     ~~~~~~~~~~~~~

src/timeout.test.mts:1:30 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations.

1 import { describe, it } from 'node:test'
                               ~~~~~~~~~~~

src/timeout.test.mts:2:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations.

2 import assert from 'node:assert'
                     ~~~~~~~~~~~~~


Found 10 errors in 6 files.

Errors  Files
     1  src/synchronized.mts:1
     2  src/synchronized.test.mts:1
     1  src/throttle.mts:1
     3  src/throttle.test.mts:2
     1  src/timeout.mts:1
     2  src/timeout.test.mts:1
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed feat(backups): use generator instead of streams for backup and replication #8432

Tristis Oris

@Tristis-Oris
from root directory.
/xen-orchestra
yarn - succeess
yarn build - failed
again to cd @vates/generator-toolbox/
yarn build

yarn run v1.22.22
$ tsc
Done in 5.50s.

cd -
yarn build

 Tasks:    29 successful, 29 total
Cached:    13 cached, 29 total
  Time:    1m17.02s

Done in 77.34s.

again at root

cd packages/xo-server
mkdir -p ~/.config/xo-server
cp sample.config.toml ~/.config/xo-server/config.toml
yarn start

Tristis Oris

delta, first full.

no NBD

2025-04-11T12:19:15.169Z xo:backups:worker INFO starting backup
2025-04-11T12:22:29.509Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
  path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T121931Z.json',
  actual: 27271364608,
  expected: 27278188544
}

2 vms but 1 log at cli.

delta

[2025-04-11T12:27:41.428Z xo:backups:worker INFO starting backup
2025-04-11T12:27:48.644Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
  path: '/xo-vm-backups/f3cfdc19-0e6a-04d1-5651-82e1dbb25191/20250411T122746Z.json',
  actual: 44040192,
  expected: 46314496
}
2025-04-11T12:28:03.055Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
  path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T122801Z.json',
  actual: 46137344,
  expected: 48412160
}
2025-04-11T12:28:03.081Z xo:backups:worker INFO backup has ended
2025-04-11T12:28:03.181Z xo:backups:worker INFO process will exit {
  duration: 21752192,
  exitCode: 0,
  resourceUsage: {
    userCPUTime: 3794057,
    systemCPUTime: 468414,
    maxRSS: 69592,
    sharedMemorySize: 0,
    unsharedDataSize: 0,
    unsharedStackSize: 0,
    minorPageFault: 42277,
    majorPageFault: 0,
    swappedOut: 0,
    fsRead: 13624,
    fsWrite: 185840,
    ipcSent: 0,
    ipcReceived: 0,
    signalsCount: 0,
    voluntaryContextSwitches: 3591,
    involuntaryContextSwitches: 441
  },
  summary: { duration: '22s', cpuUsage: '20%', memoryUsage: '67.96 MiB' }
}

can't get NBD to work. It enabled on nic, but not used for backups. XO settings copied from prod instance.

Davidj 0

Like @Tristis-Oris , my first tests were done using a clone of my production XO. They failed with "No NBD" errors.

Creating a new backup works.

Tristis Oris

@Davidj-0 No NBD usage even with new task( Main XO is on same host, same nic.

Davidj 0

@Tristis-Oris My setup is similar to yours, and I get similar errors on any existing backup job.

However, if I create a new backup job, then it works without any error.

ping @florent , maybe these data points are useful.

florent

thanks for your tests . I will recheck NBD backup

florent

ih @Tristis-Oris and @Davidj-0 I pushed an update to better handle NBD error, please keep us informed

Tristis Oris

@florent i'm not sure how to update from this branch(

git checkout .
git pull --ff-only
yarn
yarn build

Updating 04be60321..6641580e6
Fast-forward
 @xen-orchestra/xapi/disks/Xapi.mjs             | 17 ++++++++++-------
 @xen-orchestra/xapi/disks/XapiVhdCbt.mjs       |  3 ++-
 @xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs |  3 ---
 @xen-orchestra/xapi/disks/utils.mjs            | 29 ++++++++++++++++-------------
 4 files changed, 28 insertions(+), 24 deletions(-)

git checkout feat_generator_backups
Already on 'feat_generator_backups'
Your branch is up to date with 'origin/feat_generator_backups'.


git pull origin feat_generator_backups
From https://github.com/vatesfr/xen-orchestra
 * branch                feat_generator_backups -> FETCH_HEAD
Already up to date.

but still no NBD at backup.

Tristis Oris

i see significant speed difference from master branch.

prod
NBD
160.71 MiB
no NBD
169

test
NBD (broken)
110-130 MiB
no NBD
115

florent

@Tristis-Oris ouch that is quite costly
Can you describe which backup you run ?
Can you check if the duration is different (maybe this is a measurement error and not a slow speed) ?

Tristis Oris

@florent

1 vm, 1 storage, NBD connections: 1. delta, first full.

Duration: 3 minutes
Size: 26.54 GiB
Speed: 160.71 MiB/s

Duration: 4 minutes
Size: 26.53 GiB
Speed: 113.74 MiB/s

florent

@Tristis-Oris Do you have the same performance without NBD ?
Does your storage use blocks ?

Tristis Oris

@florent blocks! i forgot about that checkbox)

better but not same.

Duration: 3 minutes
Size: 26.53 GiB
Speed: 143.7 MiB/s

Speed: 146.09 MiB/s

florent

@Tristis-Oris I made a little change, can you update (like the last time ) and retest ?

Tristis Oris

@florent
same speed after fix.
142.56 MiB/s - 145.63 MiB/s

maybe i miss something else? same database as prod, only another LUN connected for backups.

florent

@Tristis-Oris no it's on our end

Could you retry nbd + target a block based directory ?
ON my test setup, with the latest changes I get better speed than master ( 190MB/s per disk vs 130-170 depending on the run and settings on master)

I got quite a huge variation between the same runs (40MB/s)

Tristis Oris

@florent yep, now it equal. Maybe my hw bottleneck? i can also check with ssd storage to see max speed.

Duration: 3 minutes
Size: 26.53 GiB
Speed: 157.78 MiB/s
Speed: 149.39 MiB/s
Speed: 163.76 MiB/s

No more errors incorrect backup size in metadata.

But still no NBD(