<![CDATA[Delta Backup Changes in 5.66 ?]]>I updated XOA to 5.66 2 days ago and it messed up with my delta backups.

Instead of 2 hours it took 20 hours yesterday and same thing again today.

I don't see any notes in the changelogs that would explain this.

Anyone else?

]]>https://xcp-ng.org/forum/topic/5371/delta-backup-changes-in-5-66RSS for NodeMon, 08 Jun 2026 05:31:35 GMTThu, 23 Dec 2021 08:02:11 GMT60<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Wed, 05 Jan 2022 15:41:08 GMT]]>@andrewreid XO team has released a new version 5.66.2 for XOA and new sources for the XO community that should fix most of the recent backup issue, so please update to latest and report back if needed.

]]>https://xcp-ng.org/forum/post/45493https://xcp-ng.org/forum/post/45493Wed, 05 Jan 2022 15:41:08 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Wed, 05 Jan 2022 02:38:56 GMT]]>I'm not sure if mine is an edge case or another "me too", but d3071 has partially fixed the always a full backup issue on my end.

I'm not sure exactly what's going on, but this morning I did wonder if it's to do with having two delta backup jobs (backing up to two different remotes), and whether something strange was happening with the snapshots between the two jobs.

For example, from my backup report for the first backup to a local NFS remote:

f2136443-ab35-4881-ab3e-8ac3f9c8bf31-image.png

Half an hour or so later, that same VM yields a full backup-sized transfer:

6f6bf1a5-418e-4d8f-9508-c45ce9014338-image.png

(Which, incidentally, I don't understand how it's a 3.33GiB transfer, when the VHD size on the SR is ~111MB and the output of df -h on the VM shows 1.8GB used. I'm confused!)

I'll roll onto feat-add-debug-backup and see what that yields!

]]>https://xcp-ng.org/forum/post/45460https://xcp-ng.org/forum/post/45460Wed, 05 Jan 2022 02:38:56 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 23:58:27 GMT]]>I'm running XO as well, not from sources.. but today, everthing worked as it should??

Lets see what happens tomorrow.

]]>https://xcp-ng.org/forum/post/45459https://xcp-ng.org/forum/post/45459Tue, 04 Jan 2022 23:58:27 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 20:37:29 GMT]]>We seem to experience the same issues. We are on commit d3071. It seems that xoa mislables the backups as well.
Even though we enforced a full backup through the schedule option, it is labled as a delta in the restore process menu.
The xoa backup "log" in the interface shows full as well instead of delta and the amount of data transferred is clearly a full.

]]>https://xcp-ng.org/forum/post/45458https://xcp-ng.org/forum/post/45458Tue, 04 Jan 2022 20:37:29 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 15:45:15 GMT]]>@s-pam I answered on the ticket and deployed the fix.

]]>https://xcp-ng.org/forum/post/45435https://xcp-ng.org/forum/post/45435Tue, 04 Jan 2022 15:45:15 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 15:26:23 GMT]]>@florent said in Delta Backup Changes in 5.66 ?:

@s-pam this issue caused quite a few tickets, which one is yours ? I can connect right away

It is Ticket#774300

]]>https://xcp-ng.org/forum/post/45431https://xcp-ng.org/forum/post/45431Tue, 04 Jan 2022 15:26:23 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 15:13:30 GMT]]>@s-pam this issue caused quite a few tickets, which one is yours ? I can connect right away

]]>https://xcp-ng.org/forum/post/45426https://xcp-ng.org/forum/post/45426Tue, 04 Jan 2022 15:13:30 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 15:04:39 GMT]]>@florent said in Delta Backup Changes in 5.66 ?:

@s-pam it's a great choice 🙂
If you can launch a backup , I can connect through a support tunnel and deploy the fix

then you can launch the backup and we can celebrate the success of this fix

Do you mean a second XOA? I already have a support ticket+tunnel open on it. Originally due to failed incremental backups, but now it includes this issue with full backups being made when they shouldn't.

]]>https://xcp-ng.org/forum/post/45424https://xcp-ng.org/forum/post/45424Tue, 04 Jan 2022 15:04:39 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 14:42:39 GMT]]>@s-pam it's a great choice 🙂
If you can launch a backup , I can connect through a support tunnel and deploy the fix

then you can launch the backup and we can celebrate the success of this fix

]]>https://xcp-ng.org/forum/post/45421https://xcp-ng.org/forum/post/45421Tue, 04 Jan 2022 14:42:39 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 14:22:40 GMT]]>@florent unfortunately not. We use XOA.

]]>https://xcp-ng.org/forum/post/45420https://xcp-ng.org/forum/post/45420Tue, 04 Jan 2022 14:22:40 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 10:12:27 GMT]]>@pnunn @S-Pam If you use the sources, can you try to use this branch feat-add-debug-backup it's master with additionnal debug and a potential fix ( potential , because I can't reproduce it regularly in our lab)

Then you can activate more debug by this

[logs]
# Display all logs matching this filter, regardless of their level
filter = 'xo:backups'

# Display all logs with level >=, regardless of their namespace
level = 'debug'

in your config file ( should be in ~/.config/xo-server/config.toml

]]>https://xcp-ng.org/forum/post/45419https://xcp-ng.org/forum/post/45419Tue, 04 Jan 2022 10:12:27 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 09:22:53 GMT]]>We are doing progress in the investigation. @florent will keep you posted!

]]>https://xcp-ng.org/forum/post/45414https://xcp-ng.org/forum/post/45414Tue, 04 Jan 2022 09:22:53 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 04 Jan 2022 01:18:23 GMT]]>Not wanting to be a "me too" but I'm seeing exactly the same thing.

Some VM's (unfortunately in my case large ones) and doing daily fulls rather than deltas.

Backup taking 14 hours instead of 2 or 3.

Similar etup to @S-Pam with nfs as the remote target.

]]>https://xcp-ng.org/forum/post/45406https://xcp-ng.org/forum/post/45406Tue, 04 Jan 2022 01:18:23 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Wed, 29 Dec 2021 08:25:31 GMT]]>@julien-f said in Delta Backup Changes in 5.66 ?:

@s-pam we were not able to reproduce so far on our side. Also, most of the XO team (myself included) is on holidays until next week, thus I'm not sure we'll be able to fix it before.

If you are using an official XO appliance, you can rollback to the stable channel and if you are using XO from the sources, you can go back to a previous commit.

Yea, I have a support ticket active for another issue, but this popped up now in the middle of it. But then I'll wait until next week before downgrating so you can have a chance to look at it through the support tunnel. I've changed the backup schedule to daily instead of hourly to save some space on the remote 😮 .

]]>https://xcp-ng.org/forum/post/45306https://xcp-ng.org/forum/post/45306Wed, 29 Dec 2021 08:25:31 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Wed, 29 Dec 2021 08:20:11 GMT]]>@s-pam we were not able to reproduce so far on our side. Also, most of the XO team (myself included) is on holidays until next week, thus I'm not sure we'll be able to fix it before.

If you are using an official XO appliance, you can rollback to the stable channel and if you are using XO from the sources, you can go back to a previous commit.

]]>https://xcp-ng.org/forum/post/45305https://xcp-ng.org/forum/post/45305Wed, 29 Dec 2021 08:20:11 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Wed, 29 Dec 2021 07:26:21 GMT]]>@julien-f said in Delta Backup Changes in 5.66 ?:

We discussed this issue in a support ticket but for everyone else: it was a bug and a fix was shipped in XO 5.66.1 yesterday 🙂

We're on 5.66.1 and incrememntal backups still make full backups. This is now eating up quite a lot of storage space.

]]>https://xcp-ng.org/forum/post/45304https://xcp-ng.org/forum/post/45304Wed, 29 Dec 2021 07:26:21 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 17:53:18 GMT]]>@julien-f NFS

]]>https://xcp-ng.org/forum/post/45300https://xcp-ng.org/forum/post/45300Tue, 28 Dec 2021 17:53:18 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 17:50:52 GMT]]>@danp Which type of remote are you using for this job? SMB? NFS?

]]>https://xcp-ng.org/forum/post/45299https://xcp-ng.org/forum/post/45299Tue, 28 Dec 2021 17:50:52 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 17:21:26 GMT]]>@danp I don't think these are new or related to this issue.

]]>https://xcp-ng.org/forum/post/45298https://xcp-ng.org/forum/post/45298Tue, 28 Dec 2021 17:21:26 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 17:07:03 GMT]]>Unsure if this is related, but I've found some errors in the logs that I don't believe occurred prior to the most recent updates --

Dec 28 00:19:10 xo-server[257773]: 2021-12-28T06:19:10.418Z xo:main WARN WebSocket send: {
Dec 28 00:19:10 xo-server[257773]:   error: Error: Cannot call write after a stream was destroyed
Dec 28 00:19:10 xo-server[257773]:       at new NodeError (node:internal/errors:371:5)
Dec 28 00:19:10 xo-server[257773]:       at errorBuffer (node:internal/streams/writable:517:14)
Dec 28 00:19:10 xo-server[257773]:       at afterWrite (node:internal/streams/writable:501:5)
Dec 28 00:19:10 xo-server[257773]:       at onwrite (node:internal/streams/writable:477:7)
Dec 28 00:19:10 xo-server[257773]:       at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:91:12)
Dec 28 00:19:10 xo-server[257773]:       at WriteWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
Dec 28 00:19:10 xo-server[257773]:     code: 'ERR_STREAM_DESTROYED'
Dec 28 00:19:10 xo-server[257773]:   }
Dec 28 00:19:10 xo-server[257773]: }

Also this --

Dec 28 00:00:13 xo-server[360334]: 2021-12-28T06:00:13.194Z xo:xapi:vm WARN HANDLE_INVALID(VDI, OpaqueRef:83be31a3-81d7-4be3-9714-0ad1fa034d20) {
Dec 28 00:00:13 xo-server[360334]:   error: XapiError: HANDLE_INVALID(VDI, OpaqueRef:83be31a3-81d7-4be3-9714-0ad1fa034d20)
Dec 28 00:00:13 xo-server[360334]:       at Function.wrap (/opt/xen-orchestra/packages/xen-api/dist/_XapiError.js:26:12)
Dec 28 00:00:13 xo-server[360334]:       at /opt/xen-orchestra/packages/xen-api/dist/transports/json-rpc.js:58:30
Dec 28 00:00:13 xo-server[360334]:       at runMicrotasks (<anonymous>)
Dec 28 00:00:13 xo-server[360334]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Dec 28 00:00:13 xo-server[360334]:     code: 'HANDLE_INVALID',
Dec 28 00:00:13 xo-server[360334]:     params: [ 'VDI', 'OpaqueRef:83be31a3-81d7-4be3-9714-0ad1fa034d20' ],
Dec 28 00:00:13 xo-server[360334]:     call: { method: 'VDI.get_record', params: [Array] },
Dec 28 00:00:13 xo-server[360334]:     url: undefined,
Dec 28 00:00:13 xo-server[360334]:     task: undefined
Dec 28 00:00:13 xo-server[360334]:   }
Dec 28 00:00:13 xo-server[360334]: }
Dec 28 00:00:13 xo-server[360334]: 2021-12-28T06:00:13.393Z xo:xapi:vm WARN HANDLE_INVALID(VDI, OpaqueRef:83be31a3-81d7-4be3-9714-0ad1fa034d20) {
Dec 28 00:00:13 xo-server[360334]:   error: XapiError: HANDLE_INVALID(VDI, OpaqueRef:83be31a3-81d7-4be3-9714-0ad1fa034d20)
Dec 28 00:00:13 xo-server[360334]:       at Function.wrap (/opt/xen-orchestra/packages/xen-api/dist/_XapiError.js:26:12)
Dec 28 00:00:13 xo-server[360334]:       at /opt/xen-orchestra/packages/xen-api/dist/transports/json-rpc.js:58:30
Dec 28 00:00:13 xo-server[360334]:       at runMicrotasks (<anonymous>)
Dec 28 00:00:13 xo-server[360334]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Dec 28 00:00:13 xo-server[360334]:     code: 'HANDLE_INVALID',
Dec 28 00:00:13 xo-server[360334]:     params: [ 'VDI', 'OpaqueRef:83be31a3-81d7-4be3-9714-0ad1fa034d20' ],
Dec 28 00:00:13 xo-server[360334]:     call: { method: 'VDI.get_record', params: [Array] },
Dec 28 00:00:13 xo-server[360334]:     url: undefined,
Dec 28 00:00:13 xo-server[360334]:     task: undefined
Dec 28 00:00:13 xo-server[360334]:   }
Dec 28 00:00:13 xo-server[360334]: }
]]>https://xcp-ng.org/forum/post/45297https://xcp-ng.org/forum/post/45297Tue, 28 Dec 2021 17:07:03 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 16:51:32 GMT]]>@julien-f It's the same delta backup job that I've been using forever --
e439a8dd-f5fa-491b-a6ee-209280331616-image.png

It uses smart mode to backup all running VMs. Scheduled to run daily at midnight, backup retention is set to 12.

]]>https://xcp-ng.org/forum/post/45296https://xcp-ng.org/forum/post/45296Tue, 28 Dec 2021 16:51:32 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 16:17:43 GMT]]>@danp We were not able to reproduce the issue on our side, what's the config of your backup job?

]]>https://xcp-ng.org/forum/post/45295https://xcp-ng.org/forum/post/45295Tue, 28 Dec 2021 16:17:43 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 16:14:45 GMT]]>Thanks for the feedback @Danp XO team is investigating as we speak

]]>https://xcp-ng.org/forum/post/45294https://xcp-ng.org/forum/post/45294Tue, 28 Dec 2021 16:14:45 GMT<![CDATA[Reply to Delta Backup Changes in 5.66 ? on Tue, 28 Dec 2021 15:06:04 GMT]]>Currently on commit 5bc4436, I'm still experiencing repeated full backups but this isn't occurring on all VMs.

@julien-f Any suggestions on how to diagnose the cause?

]]>https://xcp-ng.org/forum/post/45291https://xcp-ng.org/forum/post/45291Tue, 28 Dec 2021 15:06:04 GMT