<![CDATA[Our future backup code: test it!]]>A big change is coming!
As we prepare to add qcow2 support for backups, we took the opportunity to redesign major parts of the backup engine. The result? A much more flexible and abstracted system that can better handle various scenarios like V2V, qcow2, VHD, and more.

We're also moving from traditional streams to Node generators, adding backup throttling, and laying the groundwork for future improvements.

🚧 It's not production-ready yet, but that’s where you come in!
If you're working from sources, you can test it by switching to the branch: feat_generator_backups. Bug reports and feedback are more than welcome!

More tests from you means we could put it in production sooner 🙂

For reference, the PR is here: https://github.com/vatesfr/xen-orchestra/pull/8432

Adding directly @florent in the loop because he's the guy to talk to 😉

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed feat(backups): use generator instead of streams for backup and replication #8432

]]>https://xcp-ng.org/forum/topic/10664/our-future-backup-code-test-itRSS for NodeWed, 15 Jul 2026 00:55:24 GMTWed, 26 Mar 2025 19:00:12 GMT60<![CDATA[Reply to Our future backup code: test it! on Fri, 30 May 2025 08:03:38 GMT]]>I created a new CR job for another VM and it worked. However, it didn't work with the XO VM.
So maybe the root cause of the problem is that the old CR copies have disappeared. Maybe they still exist, but I can't find them?

]]>https://xcp-ng.org/forum/post/93450https://xcp-ng.org/forum/post/93450Fri, 30 May 2025 08:03:38 GMT<![CDATA[Reply to Our future backup code: test it! on Wed, 28 May 2025 14:17:51 GMT]]>Seems fixed in XOA 5.107.1

]]>https://xcp-ng.org/forum/post/93397https://xcp-ng.org/forum/post/93397Wed, 28 May 2025 14:17:51 GMT<![CDATA[Reply to Our future backup code: test it! on Wed, 28 May 2025 10:31:59 GMT]]>@olivierlambert @flakpyro Commit 2fd5759 seems to solve the issue for me.

]]>https://xcp-ng.org/forum/post/93388https://xcp-ng.org/forum/post/93388Wed, 28 May 2025 10:31:59 GMT<![CDATA[Reply to Our future backup code: test it! on Wed, 28 May 2025 06:24:01 GMT]]>@olivierlambert I think so! All my regular Delta+CR combination jobs run as usual tonight without issue which is good news after such a massive change.Ii can see the newly replicated copies when looking in XO-Lite or XCP-NG center so likely just a UI issue that is hopefully a quick and easy fix!

]]>https://xcp-ng.org/forum/post/93381https://xcp-ng.org/forum/post/93381Wed, 28 May 2025 06:24:01 GMT<![CDATA[Reply to Our future backup code: test it! on Wed, 28 May 2025 06:17:48 GMT]]>The windows VM disappearing is likely a web UI bug. I'm reporting it.

]]>https://xcp-ng.org/forum/post/93380https://xcp-ng.org/forum/post/93380Wed, 28 May 2025 06:17:48 GMT<![CDATA[Reply to Our future backup code: test it! on Wed, 28 May 2025 04:44:17 GMT]]>@Andrew weird, never seen that.
b4952b74-fafb-417c-8c5a-86f89203768f-изображение.png
too many tasks maybe?

]]>https://xcp-ng.org/forum/post/93378https://xcp-ng.org/forum/post/93378Wed, 28 May 2025 04:44:17 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 23:15:32 GMT]]>@flakpyro Nice catch! I see the same problem.

@olivierlambert @florent Running XO source (master 6ed35). I see the Windows VMs in the CR backup job report, but they no longer appear on the destination host as a VM. Some have updated tools, others do not. It worked before... Non-windows VMs seems to be fine. These are all old VMs, so it's a new problem on existing systems.

]]>https://xcp-ng.org/forum/post/93372https://xcp-ng.org/forum/post/93372Tue, 27 May 2025 23:15:32 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 22:06:01 GMT]]>@florent so testing a CR job with the latest XOA release. The job runs successfully and i can see the virtual disk on the remote SR however the replica does not appear in XOAs VM inventory view.

Edits:
They do appear in XCP-NG Center,and XO Lite
They do NOT appear to show up using xo-cli list-objects
This also appears to ONLY happen to Windows VMs. Maybe related to missing templates?

VMs that have not yet even ran their daily replication job also seem affected. Replicas that ran last night and appeared in inventory before the update now now longer appear in the VM inventory list. This might be related to something else that was changed in the new XOA update?

]]>https://xcp-ng.org/forum/post/93366https://xcp-ng.org/forum/post/93366Tue, 27 May 2025 22:06:01 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 17:30:58 GMT]]>@olivierlambert @florent I still hit a single thread limit in the XO VM that makes backups CPU bound limiting performance as before.

fe320cc2-13f5-4698-ae52-3c88fa9a2de8-image.png

]]>https://xcp-ng.org/forum/post/93334https://xcp-ng.org/forum/post/93334Tue, 27 May 2025 17:30:58 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 15:34:24 GMT]]>@flakpyro nope. same error.

]]>https://xcp-ng.org/forum/post/93331https://xcp-ng.org/forum/post/93331Tue, 27 May 2025 15:34:24 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 15:29:13 GMT]]>@Tristis-Oris have CR jobs started working for you with the latest master code?
@florent Understod! Our XOA appliance has been on the latest branch since we deployed it almost a year and a half ago. Though with all these changes i may consider dropping it down to stable for a bit as backups are pretty critical. We only use Delta jobs (Which seem to be working well) and Continuous replication jobs. (Which i have seen some issues with recently in this thread).

]]>https://xcp-ng.org/forum/post/93330https://xcp-ng.org/forum/post/93330Tue, 27 May 2025 15:29:13 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 10:13:13 GMT]]>@flakpyro the stable branch, is the stable xoa. The latest release is " it is tested, but on limited hardware". The "pull from master and use it" is on best effort, I wouldn't advise to use it on any critical use.

the april's release is stable and will be promoted to stable today.
the may's release probably won't be promoted to stable in june, given the massive changes. my guess is that the june's release maybe , if we deem it stable enough

]]>https://xcp-ng.org/forum/post/93322https://xcp-ng.org/forum/post/93322Tue, 27 May 2025 10:13:13 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 08:29:05 GMT]]>@flakpyro Very good question. I really don't need the aggravation of any backup job breaking on our production site......

]]>https://xcp-ng.org/forum/post/93315https://xcp-ng.org/forum/post/93315Tue, 27 May 2025 08:29:05 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 07:37:59 GMT]]>So are we now at the point that Delta backups, Mirrors and CR backups are all working and considered stable? I have only been able to test delta backups with my home lab setup which have been working without missing a beat since the May 22nd fix for tasks not clearing. (MS-01 running XCP-NG, TrueNAS for storage)

We do use make use of CR jobs pretty extensively with the full XOA appliance for our DR site at work but because its full XOA and "production" i have not tested anything there. That said hoping for no nasty surprises with the may XO update if this is planned on being pushed to the stable XOA release!

]]>https://xcp-ng.org/forum/post/93312https://xcp-ng.org/forum/post/93312Tue, 27 May 2025 07:37:59 GMT<![CDATA[Reply to Our future backup code: test it! on Tue, 27 May 2025 06:38:57 GMT]]>@manilx said in Our future backup code: test it!:

@florent waiting for final master

this will reach master today, and probably be released in the may version of XO

]]>https://xcp-ng.org/forum/post/93311https://xcp-ng.org/forum/post/93311Tue, 27 May 2025 06:38:57 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 20:50:56 GMT]]>@florent waiting for final master

]]>https://xcp-ng.org/forum/post/93307https://xcp-ng.org/forum/post/93307Mon, 26 May 2025 20:50:56 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 18:19:19 GMT]]>@manilx said in Our future backup code: test it!:

@florent Simple NFS, no encryption.

I have a potential fix here on the branch fix_backup
( https://github.com/vatesfr/xen-orchestra/pull/8635/files)

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(disk-transform): backup omnibus fix #8635

]]>https://xcp-ng.org/forum/post/93306https://xcp-ng.org/forum/post/93306Mon, 26 May 2025 18:19:19 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 15:37:53 GMT]]>@florent Simple NFS, no encryption.

]]>https://xcp-ng.org/forum/post/93304https://xcp-ng.org/forum/post/93304Mon, 26 May 2025 15:37:53 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 15:36:00 GMT]]>At least I reproduce it locally , when going from a non block remote to an encrypted remote

Is your target remote encrypted ?

]]>https://xcp-ng.org/forum/post/93303https://xcp-ng.org/forum/post/93303Mon, 26 May 2025 15:36:00 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 10:30:47 GMT]]>@florent Giving up on mirror backups. Switching to direct backups to remore storage.
At least until this works again as expected.

]]>https://xcp-ng.org/forum/post/93296https://xcp-ng.org/forum/post/93296Mon, 26 May 2025 10:30:47 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 08:45:57 GMT]]>@florent All my backups have this set! And this only happens with the new backup code, never with the old.

]]>https://xcp-ng.org/forum/post/93293https://xcp-ng.org/forum/post/93293Mon, 26 May 2025 08:45:57 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 08:29:25 GMT]]>this error is known : this when a mirror backup start before the end of the asynchronoux cleaning of previous backup is not done

you can check the "merge backup synchronously" on the advanced setting of the first backup

]]>https://xcp-ng.org/forum/post/93292https://xcp-ng.org/forum/post/93292Mon, 26 May 2025 08:29:25 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 08:19:35 GMT]]>@florent The original backup is delta. Using NBD+CBT to a NFS storage.

Also get these frequently:

ScreenShot 2025-05-26 at 09.18.45.png

But nothing is running in parallel.

]]>https://xcp-ng.org/forum/post/93291https://xcp-ng.org/forum/post/93291Mon, 26 May 2025 08:19:35 GMT<![CDATA[Reply to Our future backup code: test it! on Mon, 26 May 2025 08:15:49 GMT]]>I am checking it
Can you confirm that this a delta backup job and you are not using block.

]]>https://xcp-ng.org/forum/post/93290https://xcp-ng.org/forum/post/93290Mon, 26 May 2025 08:15:49 GMT