Our future backup code: test it!
-
@flakpyro today the backup code use binary stream in a the vhd format. This format is limited, by design , to 2TB disks
xcp-ng team introduce the qcow2 format to handle bigger disk
By using a independant format, we'll be able to handle both vhd and qcow2 on the backup side without multiplying complexity. We'll also be able to build the adapter to handle the various vmdk sub format (rax, cowd, sesparse and stream optimized) used by v2v and import bigger disks directly -
Could i test it on prod, along with real backups? or better to setup 2nd XO instance.
-
It's your call, but there's no guarantee it won't break for now, as the code is relatively young.
-
@olivierlambert but could it change old backup tasks\compability or something else? tests with real data obviously more helpful.
-
I will let @florent answering this one
-
I'm still getting "Cannot find module '@vates/generator-toolbox'" errors when I try to build it. I'm using the 'feat_generator_backups" branch.
-
Thanks for your feedback, @florent will take a look.
-
@Davidj-0 Are you using it from source or through a thrid party script ?
because this dependency is present in the branch, but still not published on npm
-
@Tristis-Oris said in Our future backup code: test it!:
@olivierlambert but could it change old backup tasks\compability or something else? tests with real data obviously more helpful.
It could gives us more informaitons, but in the other hand we are still in the validation phase.
-
F florent referenced this topic on
-
same error as above
@xen-orchestra/disk-transform:build: cache miss, executing 6b8daba7020cf148 @xen-orchestra/disk-transform:build: yarn run v1.22.22 @xen-orchestra/disk-transform:build: $ tsc @xen-orchestra/disk-transform:build: src/SynchronizedDisk.mts(2,30): error TS2307: Cannot find module '@vates/generator-toolbox' or its corresponding type declarations. @xen-orchestra/disk-transform:build: error Command failed with exit code 2. @xen-orchestra/disk-transform:build: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. @xen-orchestra/disk-transform:build: ERROR: command finished with error: command (/opt/xo/xo-builds/xen-orchestra-202504071027/@xen-orchestra/disk-transform) /tmp/yarn--1744010914016-0.7668824784043369/yarn run build exited (2) @xen-orchestra/disk-transform#build: command (/opt/xo/xo-builds/xen-orchestra-202504071027/@xen-orchestra/disk-transform) /tmp/yarn--1744010914016-0.7668824784043369/yarn run build exited (2) Tasks: 24 successful, 28 total Cached: 0 cached, 28 total Time: 11.251s Failed: @xen-orchestra/disk-transform#build ERROR run failed: command exited (2) error Command failed with exit code 2. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. + rm -rf /opt/xo/xo-builds/xen-orchestra-202504071027 -
@Tristis-Oris did you run yarn from the root of the project to install the new dependencies ?
-
yarn install v1.22.22 [1/5] Validating package.json... [2/5] Resolving packages... [3/5] Fetching packages... warning Pattern ["@intlify/shared@latest"] is trying to unpack in the same destination "/usr/local/share/.cache/yarn/v6/npm-@intlify-shared-11.1.2-d780bc8eb4e3c3ef8a3e45c421f0f5ecf59651f2-integrity/node_modules/@intlify/shared" as pattern ["@intlify/shared@11.1.2","@intlify/shared@11.1.2","@intlify/shared@11.1.2"]. This could result in non-deterministic behavior, skipping. warning Pattern ["@intlify/shared@next"] is trying to unpack in the same destination "/usr/local/share/.cache/yarn/v6/npm-@intlify-shared-11.0.0-rc.1-52a67aa12fccd9303027b48ebf017b75b7350283-integrity/node_modules/@intlify/shared" as pattern ["@intlify/shared@11.0.0-rc.1"]. This could result in non-deterministic behavior, skipping. warning bare-os@3.4.0: The engine "bare" appears to be invalid. [4/5] Linking dependencies... warning "@commitlint/cli > @commitlint/load > cosmiconfig-typescript-loader@6.1.0" has unmet peer dependency "@types/node@*". warning "@commitlint/cli > @commitlint/load > cosmiconfig-typescript-loader@6.1.0" has unmet peer dependency "typescript@>=5". warning "@typescript-eslint/eslint-plugin > ts-api-utils@1.4.3" has unmet peer dependency "typescript@>=4.2.0". warning " > @xen-orchestra/web-core@0.18.0" has unmet peer dependency "pinia@^3.0.1". warning " > @xen-orchestra/web-core@0.18.0" has unmet peer dependency "vue@~3.5.13". warning " > @xen-orchestra/web-core@0.18.0" has unmet peer dependency "vue-i18n@^11.1.2". warning " > @xen-orchestra/web-core@0.18.0" has unmet peer dependency "vue-router@^4.5.0". warning "workspace-aggregator-9e56f98d-aa3d-4c5b-aae2-45a491674f9d > @xen-orchestra/rest-api > inversify@6.2.2" has unmet peer dependency "reflect-metadata@~0.2.2". warning "workspace-aggregator-9e56f98d-aa3d-4c5b-aae2-45a491674f9d > @xen-orchestra/rest-api > swagger-ui-express@5.0.1" has unmet peer dependency "express@>=4.0.0 || >=5.0.0-beta". warning "workspace-aggregator-9e56f98d-aa3d-4c5b-aae2-45a491674f9d > @vates/event-listeners-manager > tap > @tapjs/test > @isaacs/ts-node-temp-fork-for-pr-2009@10.9.7" has unmet peer dependency "@types/node@*". warning "workspace-aggregator-9e56f98d-aa3d-4c5b-aae2-45a491674f9d > @xen-orchestra/rest-api > inversify > @inversifyjs/core > @inversifyjs/reflect-metadata-utils@0.2.4" has unmet peer dependency "reflect-metadata@0.2.2". warning "workspace-aggregator-9e56f98d-aa3d-4c5b-aae2-45a491674f9d > @vates/event-listeners-manager > tap > @tapjs/asserts > tcompare > react-element-to-jsx-string@15.0.0" has unmet peer dependency "react@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1 || ^18.0.0". warning "workspace-aggregator-9e56f98d-aa3d-4c5b-aae2-45a491674f9d > @vates/event-listeners-manager > tap > @tapjs/asserts > tcompare > react-element-to-jsx-string@15.0.0" has unmet peer dependency "react-dom@^0.14.8 || ^15.0.1 || ^16.0.0 || ^17.0.1 || ^18.0.0". warning Workspaces can only be enabled in private projects. [5/5] Building fresh packages... $ husky install husky - Git hooks installed Done in 39.86s.but same error.
-
@Tristis-Oris it looks like the build from completly empty repo don't work as expected
can you try this :
cd @vates/generator-toolbox/ yarn build cd - yarn buildI am working on improving the build process in the meantimes
-
@florent yep it works now.
-
So is it this new backup code which the tape back up support will be plugging into?
If so I have been thinking may be have it when doing the tape backup support, get the XOA to install the packages when required after prompting the user. Also have xo-server instruct the creation of a new disk image for attaching to xoa. The new disk image would act as storage or cache for the extra data used by tape drive backups.
The tape drive related software wouldn't be installed by default, unless specified as such during deployment with the XOA deploy tool. But instead installed at user request, by selecting to use tape backup destination when setting up back up job, clicking on the shim widget about tape backups or restoring from an online Vates account backup of XOA settings.
-
@florent I'm doing this from source.
Runninggit pull --rebaseseems to have fixed the build problems. -
@john.c the backup on tape is still on the roadmap but won't be for this step
The main change is that we'll have to rework part of the code to ensure we write data sequentially + the management tools ( which backup is on which tape )
You can try the FS on tape , but the performance penalty is heavy
-
@florent said in Our future backup code: test it!:
@john.c the backup on tape is still on the roadmap but won't be for this step
The main change is that we'll have to rework part of the code to ensure we write data sequentially + the management tools ( which backup is on which tape )
You can try the FS on tape , but the performance penalty is heavy
Even if its not part of this rewrite some of what I wrote is my idea or view as to how it could be implemented later on. So that it's implemented in a way, that permits people who don't need tape back ups to still have small Xen Orchestra instances. But at the same time enable those who use tape backups, to have the capacity and an appropriately sized Xen Orchestra.
-
Testing a continuous replication backup, I get Error: can't connect to any nbd client
Running the same job on my regular XO works fine. Let me know what logs you want to see.
mail (h2) Snapshot Start: 2025-04-10 18:18 End: 2025-04-10 18:18 h3 local storage (130.02 GiB free) - h3 Start: 2025-04-10 18:18 Start: 2025-04-10 18:18 End: 2025-04-10 18:18 Duration: a few seconds Error: can't connect to any nbd client Type: delta -
@Tristis-Oris probably after https://github.com/vatesfr/xen-orchestra/pull/8432/files/04be6032180e7245a8db9c5f78177ac7999882c2
/@vates/generator-toolbox# yarn build yarn run v1.22.22 $ tsc /bin/sh: 1: tsc: not found error Command failed with exit code 127. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.ok,
npm install -g typescriptyarn build yarn run v1.22.22 $ tsc src/synchronized.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/synchronized.test.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/synchronized.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations. 2 import { suite, test } from 'node:test' ~~~~~~~~~~~ src/throttle.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/throttle.test.mts:2:29 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations. 2 import { suite, test } from 'node:test' ~~~~~~~~~~~ src/throttle.test.mts:3:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 3 import assert from 'node:assert' ~~~~~~~~~~~~~ src/throttle.test.mts:7:11 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`. 7 yield Buffer.allocUnsafe(chunkSize) ~~~~~~ src/timeout.mts:1:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 1 import assert from 'node:assert' ~~~~~~~~~~~~~ src/timeout.test.mts:1:30 - error TS2307: Cannot find module 'node:test' or its corresponding type declarations. 1 import { describe, it } from 'node:test' ~~~~~~~~~~~ src/timeout.test.mts:2:20 - error TS2307: Cannot find module 'node:assert' or its corresponding type declarations. 2 import assert from 'node:assert' ~~~~~~~~~~~~~ Found 10 errors in 6 files. Errors Files 1 src/synchronized.mts:1 2 src/synchronized.test.mts:1 1 src/throttle.mts:1 3 src/throttle.test.mts:2 1 src/timeout.mts:1 2 src/timeout.test.mts:1 error Command failed with exit code 2. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
