Our future backup code: test it!

Tristis Oris

delta, first full.

no NBD

2025-04-11T12:19:15.169Z xo:backups:worker INFO starting backup
2025-04-11T12:22:29.509Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
  path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T121931Z.json',
  actual: 27271364608,
  expected: 27278188544
}

2 vms but 1 log at cli.

delta

[2025-04-11T12:27:41.428Z xo:backups:worker INFO starting backup
2025-04-11T12:27:48.644Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
  path: '/xo-vm-backups/f3cfdc19-0e6a-04d1-5651-82e1dbb25191/20250411T122746Z.json',
  actual: 44040192,
  expected: 46314496
}
2025-04-11T12:28:03.055Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
  path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T122801Z.json',
  actual: 46137344,
  expected: 48412160
}
2025-04-11T12:28:03.081Z xo:backups:worker INFO backup has ended
2025-04-11T12:28:03.181Z xo:backups:worker INFO process will exit {
  duration: 21752192,
  exitCode: 0,
  resourceUsage: {
    userCPUTime: 3794057,
    systemCPUTime: 468414,
    maxRSS: 69592,
    sharedMemorySize: 0,
    unsharedDataSize: 0,
    unsharedStackSize: 0,
    minorPageFault: 42277,
    majorPageFault: 0,
    swappedOut: 0,
    fsRead: 13624,
    fsWrite: 185840,
    ipcSent: 0,
    ipcReceived: 0,
    signalsCount: 0,
    voluntaryContextSwitches: 3591,
    involuntaryContextSwitches: 441
  },
  summary: { duration: '22s', cpuUsage: '20%', memoryUsage: '67.96 MiB' }
}

can't get NBD to work. It enabled on nic, but not used for backups. XO settings copied from prod instance.

Davidj 0

Like @Tristis-Oris , my first tests were done using a clone of my production XO. They failed with "No NBD" errors.

Creating a new backup works.

Tristis Oris

@Davidj-0 No NBD usage even with new task( Main XO is on same host, same nic.

Davidj 0

@Tristis-Oris My setup is similar to yours, and I get similar errors on any existing backup job.

However, if I create a new backup job, then it works without any error.

ping @florent , maybe these data points are useful.

florent

thanks for your tests . I will recheck NBD backup

florent

ih @Tristis-Oris and @Davidj-0 I pushed an update to better handle NBD error, please keep us informed

Tristis Oris

@florent i'm not sure how to update from this branch(

git checkout .
git pull --ff-only
yarn
yarn build

Updating 04be60321..6641580e6
Fast-forward
 @xen-orchestra/xapi/disks/Xapi.mjs             | 17 ++++++++++-------
 @xen-orchestra/xapi/disks/XapiVhdCbt.mjs       |  3 ++-
 @xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs |  3 ---
 @xen-orchestra/xapi/disks/utils.mjs            | 29 ++++++++++++++++-------------
 4 files changed, 28 insertions(+), 24 deletions(-)

git checkout feat_generator_backups
Already on 'feat_generator_backups'
Your branch is up to date with 'origin/feat_generator_backups'.


git pull origin feat_generator_backups
From https://github.com/vatesfr/xen-orchestra
 * branch                feat_generator_backups -> FETCH_HEAD
Already up to date.

but still no NBD at backup.

Tristis Oris

i see significant speed difference from master branch.

prod
NBD
160.71 MiB
no NBD
169

test
NBD (broken)
110-130 MiB
no NBD
115

florent

@Tristis-Oris ouch that is quite costly
Can you describe which backup you run ?
Can you check if the duration is different (maybe this is a measurement error and not a slow speed) ?

Tristis Oris

@florent

1 vm, 1 storage, NBD connections: 1. delta, first full.

Duration: 3 minutes
Size: 26.54 GiB
Speed: 160.71 MiB/s

Duration: 4 minutes
Size: 26.53 GiB
Speed: 113.74 MiB/s

florent

@Tristis-Oris Do you have the same performance without NBD ?
Does your storage use blocks ?

Tristis Oris

@florent blocks! i forgot about that checkbox)

better but not same.

Duration: 3 minutes
Size: 26.53 GiB
Speed: 143.7 MiB/s

Speed: 146.09 MiB/s

florent

@Tristis-Oris I made a little change, can you update (like the last time ) and retest ?

Tristis Oris

@florent
same speed after fix.
142.56 MiB/s - 145.63 MiB/s

maybe i miss something else? same database as prod, only another LUN connected for backups.