Continuous Replication jobs creates full backups every time since 2025-09-06 (xo from source)
-
Anyone else experiencing the same problem, or is it just a problem of my imagination?
Recap:
Since 2025-09-06, all Continuous Replication jobs are creating full backups instead of incrementals.
Environment: XO from source, updated to commit c2144 (2 commits behind at the time of test).
Replication target: a standalone host that is part of the pool, on the same local network, with local SSD storage.
Logs and status screenshots were attached in my first post.
What I’ve done so far:
2025-09-11- Updated Xen Orchestra to the latest commits (as of 2025-09-11).
- Restarted the XO VM and the source VM after update.
2025-09-12:
- Applied pool patches as they became available.
- Installed host updates and rebooted hosts.
- Restarted the source and target hosts (since all hosts were restarted).
- Verified behavior again today → still a full backup for the “Admin Ubuntu 24” VM.
Observation:
The issue began exactly after the 2025-09-05 updates (which included a set of host patches).
Since then, every daily replication runs as a full. -
You can try to pinpoint the exact commit to be 100% sure, with git bissect
-
@olivierlambert said in Continuous Replication jobs creates full backups every time since 2025-09-06 (xo from source):
git bissect
That might be quite a bunch of commits in between the working one and when I discovered the problems (I actually let it do it's thing a couple of days, quickly filling up the backup drive)
I keep only the current and the previous build of XO, so I only know it's somewhere between Aug 16 (from the 'lastlog' to find out when I was logged in) and late Sep 5. -
Bissect is rather fast, it… bissects between 2 commits. If you can pinpoint the exact commit generating that, it means we could probably solve this a LOT faster
For ref: https://en.wikipedia.org/wiki/Bisection_(software_engineering)
-
@olivierlambert Has the problem been confirmed ? I can try to pinpoint where the always-full replications were introduced, but more time efficient to do it on a machine that is fast do replicate, not the one that now takes 2.5 hours (a few minutes for the diff before it borked starting at the backup made 6 Sept)
-
No, no confirmation so far, that's why we need more to investigate
-
@olivierlambert said in Continuous Replication jobs creates full backups every time since 2025-09-06 (xo from source):
No, no confirmation so far, that's why we need more to investigate
Great (or maybe not) news.. At least on my system this is very easy to replicate.. On a tiny test machine that only have had a normal backup before, first time doing replication copy (to H2 SSD) it sends it as full as expected. Just did some simple operations on the machine (like apt update, no upgrade), then retried:
-
Too bad it becomes that unexplainable slow when it has to fall back to a full backup:
The first (full) backup was 10x faster:
-
@olivierlambert This happens to me too with a459015ca91c159123bb682f16237b4371a312a6.
I did open an issue https://github.com/vatesfr/xen-orchestra/issues/8969
-
@Andrew And it doesn't with the commit just before?
-
@olivierlambert Correct. Running commit 4944ea902ff19f172b1b86ec96ad989e322bec2c works.
-
@florent so it's like https://github.com/vatesfr/xen-orchestra/commit/a459015ca91c159123bb682f16237b4371a312a6 might introduced a regression?
