Our future backup code: test it!
-
Testing a continuous replication backup, I get Error: can't connect to any nbd client
Running the same job on my regular XO works fine. Let me know what logs you want to see.
mail (h2) Snapshot Start: 2025-04-10 18:18 End: 2025-04-10 18:18 h3 local storage (130.02 GiB free) - h3 Start: 2025-04-10 18:18 Start: 2025-04-10 18:18 End: 2025-04-10 18:18 Duration: a few seconds Error: can't connect to any nbd client Type: delta -
@Tristis-Oris probably after https://github.com/vatesfr/xen-orchestra/pull/8432/files/04be6032180e7245a8db9c5f78177ac7999882c2
/@vates/generator-toolbox# yarn build yarn run v1.22.22 $ tsc /bin/sh: 1: tsc: not found error Command failed with exit code 127. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.ok,
npm install -g typescriptyarn build yarn run v1.22.22 $ tsc src/synchronized.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/synchronized.test.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/synchronized.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations. 2 import { suite, test } from 'node:test' ~~~~~~~~~~~ src/throttle.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/throttle.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations. 2 import { suite, test } from 'node:test' ~~~~~~~~~~~ src/throttle.test.mts:3:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 3 import assert from 'node:assert' ~~~~~~~~~~~~~ src/throttle.test.mts:7:11 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`. 7 yield Buffer.allocUnsafe(chunkSize) ~~~~~~ src/timeout.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/timeout.test.mts:1:30 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations. 1 import { describe, it } from 'node:test' ~~~~~~~~~~~ src/timeout.test.mts:2:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 2 import assert from 'node:assert' ~~~~~~~~~~~~~ Found 10 errors in 6 files. Errors Files 1 src/synchronized.mts:1 2 src/synchronized.test.mts:1 1 src/throttle.mts:1 3 src/throttle.test.mts:2 1 src/timeout.mts:1 2 src/timeout.test.mts:1 error Command failed with exit code 2. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. -
@Tristis-Oris
from root directory.
/xen-orchestra
yarn- succeess
yarn build- failed
again tocd @vates/generator-toolbox/
yarn buildyarn run v1.22.22 $ tsc Done in 5.50s.cd -
yarn buildTasks: 29 successful, 29 total Cached: 13 cached, 29 total Time: 1m17.02s Done in 77.34s.again at root
cd packages/xo-server mkdir -p ~/.config/xo-server cp sample.config.toml ~/.config/xo-server/config.toml yarn start -
delta, first full.
- no NBD
2025-04-11T12:19:15.169Z xo:backups:worker INFO starting backup 2025-04-11T12:22:29.509Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata { path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T121931Z.json', actual: 27271364608, expected: 27278188544 }2 vms but 1 log at cli.
delta
[2025-04-11T12:27:41.428Z xo:backups:worker INFO starting backup 2025-04-11T12:27:48.644Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata { path: '/xo-vm-backups/f3cfdc19-0e6a-04d1-5651-82e1dbb25191/20250411T122746Z.json', actual: 44040192, expected: 46314496 } 2025-04-11T12:28:03.055Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata { path: '/xo-vm-backups/07e5d245-7ed3-b690-e386-3461556ad41f/20250411T122801Z.json', actual: 46137344, expected: 48412160 } 2025-04-11T12:28:03.081Z xo:backups:worker INFO backup has ended 2025-04-11T12:28:03.181Z xo:backups:worker INFO process will exit { duration: 21752192, exitCode: 0, resourceUsage: { userCPUTime: 3794057, systemCPUTime: 468414, maxRSS: 69592, sharedMemorySize: 0, unsharedDataSize: 0, unsharedStackSize: 0, minorPageFault: 42277, majorPageFault: 0, swappedOut: 0, fsRead: 13624, fsWrite: 185840, ipcSent: 0, ipcReceived: 0, signalsCount: 0, voluntaryContextSwitches: 3591, involuntaryContextSwitches: 441 }, summary: { duration: '22s', cpuUsage: '20%', memoryUsage: '67.96 MiB' } }can't get NBD to work. It enabled on nic, but not used for backups. XO settings copied from prod instance.
-
Like @Tristis-Oris , my first tests were done using a clone of my production XO. They failed with "No NBD" errors.
Creating a new backup works.
-
@Davidj-0 No NBD usage even with new task( Main XO is on same host, same nic.
-
@Tristis-Oris My setup is similar to yours, and I get similar errors on any existing backup job.
However, if I create a new backup job, then it works without any error.
ping @florent , maybe these data points are useful.
-
thanks for your tests . I will recheck NBD backup
-
ih @Tristis-Oris and @Davidj-0 I pushed an update to better handle NBD error, please keep us informed
-
@florent i'm not sure how to update from this branch(
git checkout . git pull --ff-only yarn yarn buildUpdating 04be60321..6641580e6 Fast-forward @xen-orchestra/xapi/disks/Xapi.mjs | 17 ++++++++++------- @xen-orchestra/xapi/disks/XapiVhdCbt.mjs | 3 ++- @xen-orchestra/xapi/disks/XapiVhdStreamNbd.mjs | 3 --- @xen-orchestra/xapi/disks/utils.mjs | 29 ++++++++++++++++------------- 4 files changed, 28 insertions(+), 24 deletions(-)git checkout feat_generator_backups Already on 'feat_generator_backups' Your branch is up to date with 'origin/feat_generator_backups'. git pull origin feat_generator_backups From https://github.com/vatesfr/xen-orchestra * branch feat_generator_backups -> FETCH_HEAD Already up to date.but still no NBD at backup.
-
i see significant speed difference from master branch.
prod
NBD
160.71 MiB
no NBD
169test
NBD (broken)
110-130 MiB
no NBD
115 -
@Tristis-Oris ouch that is quite costly
Can you describe which backup you run ?
Can you check if the duration is different (maybe this is a measurement error and not a slow speed) ? -
1 vm, 1 storage, NBD connections: 1. delta, first full.
Duration: 3 minutes
Size: 26.54 GiB
Speed: 160.71 MiB/sDuration: 4 minutes
Size: 26.53 GiB
Speed: 113.74 MiB/s -
@Tristis-Oris Do you have the same performance without NBD ?
Does your storage use blocks ? -
@florent blocks! i forgot about that checkbox)
better but not same.
Duration: 3 minutes
Size: 26.53 GiB
Speed: 143.7 MiB/sSpeed: 146.09 MiB/s
-
@Tristis-Oris I made a little change, can you update (like the last time ) and retest ?
-
@florent
same speed after fix.
142.56 MiB/s - 145.63 MiB/smaybe i miss something else? same database as prod, only another LUN connected for backups.
-
@Tristis-Oris no it's on our end
Could you retry nbd + target a block based directory ?
ON my test setup, with the latest changes I get better speed than master ( 190MB/s per disk vs 130-170 depending on the run and settings on master)I got quite a huge variation between the same runs (40MB/s)
-
@florent yep, now it equal. Maybe my hw bottleneck? i can also check with ssd storage to see max speed.
Duration: 3 minutes
Size: 26.53 GiB
Speed: 157.78 MiB/s
Speed: 149.39 MiB/s
Speed: 163.76 MiB/sNo more errors
incorrect backup size in metadata.But still no NBD(
-
@Tristis-Oris that is already a good news.
I pushed an additional fix : the NBD info was not shown on the UI
