@dthenot Yes, VHD files on both local disk and NFS, same problem.

Testing one VM, I removed the snapshot, disk CBT setting, and removed the destination replica. First CR run does a full backup without issue (same NBD/CBT/purge enabled). Second run has the same problem (fell back to full). So, clearing out things does not fix it (with the same original setup).

Testing several combinations, just disabling the backup purge snapshot option makes the delta CR backup work again (NBD/CBT still enabled). It does a full backup the first run (fell back to full), but then does delta after that.

@rzr (edit) After upgrading two main pools, I'm having CR delta backup issues. Everything was working before the XCP update, now every VM has the same error of Backup fell back to a full. Using XO master db9c4, but the same XO setup was working just fine before the XCP update.

(edit 2) XO logs

Apr 15 22:55:40 xo1 xo-server[1409]: 2026-04-16T02:55:40.613Z xo:backups:worker INFO starting backup
Apr 15 22:55:42 xo1 xo-server[1409]: 2026-04-16T02:55:42.006Z xo:xapi:xapi-disks INFO export through vhd
Apr 15 22:55:44 xo1 xo-server[1409]: 2026-04-16T02:55:44.093Z xo:xapi:vdi INFO  OpaqueRef:07ac67ab-05cf-a066-5924-f28e15642d4e was already destroyed {
Apr 15 22:55:44 xo1 xo-server[1409]:   vdiRef: 'OpaqueRef:49ff18d3-5c18-176c-4930-0163c6727c2b',
Apr 15 22:55:44 xo1 xo-server[1409]:   vbdRef: 'OpaqueRef:07ac67ab-05cf-a066-5924-f28e15642d4e'
Apr 15 22:55:44 xo1 xo-server[1409]: }
Apr 15 22:55:44 xo1 xo-server[1409]: 2026-04-16T02:55:44.839Z xo:xapi:vdi INFO  OpaqueRef:e5fa3d00-f629-6983-6ff2-841e9edacf82 has been disconnected from dom0 {
Apr 15 22:55:44 xo1 xo-server[1409]:   vdiRef: 'OpaqueRef:02f9ba92-1ee2-88eb-f660-a2cf3eeb287d',
Apr 15 22:55:44 xo1 xo-server[1409]:   vbdRef: 'OpaqueRef:e5fa3d00-f629-6983-6ff2-841e9edacf82'
Apr 15 22:55:44 xo1 xo-server[1409]: }
Apr 15 22:55:44 xo1 xo-server[1409]: 2026-04-16T02:55:44.910Z xo:xapi:vm WARN _assertHealthyVdiChain, could not fetch VDI {
Apr 15 22:55:44 xo1 xo-server[1409]:   error: XapiError: UUID_INVALID(VDI, 8f233bfc-9deb-4a06-aa07-0510de7496a1)
Apr 15 22:55:44 xo1 xo-server[1409]:       at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/_XapiError.mjs:16:12)
Apr 15 22:55:44 xo1 xo-server[1409]:       at file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/transports/json-rpc.mjs:38:21
Apr 15 22:55:44 xo1 xo-server[1409]:       at process.processTicksAndRejections (node:internal/process/task_queues:104:5) {
Apr 15 22:55:44 xo1 xo-server[1409]:     code: 'UUID_INVALID',
Apr 15 22:55:44 xo1 xo-server[1409]:     params: [ 'VDI', '8f233bfc-9deb-4a06-aa07-0510de7496a1' ],
Apr 15 22:55:44 xo1 xo-server[1409]:     call: { duration: 3, method: 'VDI.get_by_uuid', params: [Array] },
Apr 15 22:55:44 xo1 xo-server[1409]:     url: undefined,
Apr 15 22:55:44 xo1 xo-server[1409]:     task: undefined
Apr 15 22:55:44 xo1 xo-server[1409]:   }
Apr 15 22:55:44 xo1 xo-server[1409]: }
Apr 15 22:55:46 xo1 xo-server[1409]: 2026-04-16T02:55:46.732Z xo:xapi:xapi-disks INFO Error in openNbdCBT XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated], )
Apr 15 22:55:46 xo1 xo-server[1409]:     at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/_XapiError.mjs:16:12)
Apr 15 22:55:46 xo1 xo-server[1409]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/_getTaskResult.mjs:13:29)
Apr 15 22:55:46 xo1 xo-server[1409]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/index.mjs:1078:24)
Apr 15 22:55:46 xo1 xo-server[1409]:     at file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/index.mjs:1112:14
Apr 15 22:55:46 xo1 xo-server[1409]:     at Array.forEach (<anonymous>)
Apr 15 22:55:46 xo1 xo-server[1409]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/index.mjs:1102:12)
Apr 15 22:55:46 xo1 xo-server[1409]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202604151415/packages/xen-api/index.mjs:1275:14)
Apr 15 22:55:46 xo1 xo-server[1409]:     at process.processTicksAndRejections (node:internal/process/task_queues:104:5) {
Apr 15 22:55:46 xo1 xo-server[1409]:   code: 'SR_BACKEND_FAILURE_460',
Apr 15 22:55:46 xo1 xo-server[1409]:   params: [
Apr 15 22:55:46 xo1 xo-server[1409]:     '',
Apr 15 22:55:46 xo1 xo-server[1409]:     'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Apr 15 22:55:46 xo1 xo-server[1409]:     ''
Apr 15 22:55:46 xo1 xo-server[1409]:   ],
Apr 15 22:55:46 xo1 xo-server[1409]:   call: undefined,
Apr 15 22:55:46 xo1 xo-server[1409]:   url: undefined,
Apr 15 22:55:46 xo1 xo-server[1409]:   task: task {
Apr 15 22:55:46 xo1 xo-server[1409]:     uuid: '8fae41b4-de82-789c-980a-5ff2d490d2d8',
Apr 15 22:55:46 xo1 xo-server[1409]:     name_label: 'Async.VDI.list_changed_blocks',
Apr 15 22:55:46 xo1 xo-server[1409]:     name_description: '',
Apr 15 22:55:46 xo1 xo-server[1409]:     allowed_operations: [],
Apr 15 22:55:46 xo1 xo-server[1409]:     current_operations: {},
Apr 15 22:55:46 xo1 xo-server[1409]:     created: '20260416T02:55:46Z',
Apr 15 22:55:46 xo1 xo-server[1409]:     finished: '20260416T02:55:46Z',
Apr 15 22:55:46 xo1 xo-server[1409]:     status: 'failure',
Apr 15 22:55:46 xo1 xo-server[1409]:     resident_on: 'OpaqueRef:7b987b11-ada0-99ce-d831-6e589bf34b50',
Apr 15 22:55:46 xo1 xo-server[1409]:     progress: 1,
Apr 15 22:55:46 xo1 xo-server[1409]:     type: '<none/>',
Apr 15 22:55:46 xo1 xo-server[1409]:     result: '',
Apr 15 22:55:46 xo1 xo-server[1409]:     error_info: [
Apr 15 22:55:46 xo1 xo-server[1409]:       'SR_BACKEND_FAILURE_460',
Apr 15 22:55:46 xo1 xo-server[1409]:       '',
Apr 15 22:55:46 xo1 xo-server[1409]:       'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Apr 15 22:55:46 xo1 xo-server[1409]:       ''
Apr 15 22:55:46 xo1 xo-server[1409]:     ],
Apr 15 22:55:46 xo1 xo-server[1409]:     other_config: {},
Apr 15 22:55:46 xo1 xo-server[1409]:     subtask_of: 'OpaqueRef:NULL',
Apr 15 22:55:46 xo1 xo-server[1409]:     subtasks: [],
Apr 15 22:55:46 xo1 xo-server[1409]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 150))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))'
Apr 15 22:55:46 xo1 xo-server[1409]:   }
Apr 15 22:55:46 xo1 xo-server[1409]: }
Apr 15 22:55:46 xo1 xo-server[1409]: 2026-04-16T02:55:46.735Z xo:xapi:xapi-disks INFO export through vhd
Apr 15 22:55:48 xo1 xo-server[1409]: 2026-04-16T02:55:48.115Z xo:xapi:vdi WARN invalid HTTP header in response body {
Apr 15 22:55:48 xo1 xo-server[1409]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Apr 15 22:55:48 xo1 xo-server[1409]:     'content-length: 318\r\n' +
Apr 15 22:55:48 xo1 xo-server[1409]:     'content-type: text/html\r\n' +
Apr 15 22:55:48 xo1 xo-server[1409]:     'connection: close\r\n' +
Apr 15 22:55:48 xo1 xo-server[1409]:     'cache-control: no-cache, no-store\r\n' +
Apr 15 22:55:48 xo1 xo-server[1409]:     '\r\n' +
Apr 15 22:55:48 xo1 xo-server[1409]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:3b37047e-11dd-f836-ebed-acfaff2072ac; CBT metadata ]</body></html>'
Apr 15 22:55:48 xo1 xo-server[1409]: }
Apr 15 22:55:48 xo1 xo-server[1409]: 2026-04-16T02:55:48.124Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:e7de1446-34fd-1ae8-4680-351b1e72b2dd from OpaqueRef:3b37047e-11dd-f836-ebed-acfaff2072ac, fallBack to a full {
Apr 15 22:55:48 xo1 xo-server[1409]:   error: Error: invalid HTTP header in response body
Apr 15 22:55:48 xo1 xo-server[1409]:       at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/xapi/vdi.mjs:37:19)
Apr 15 22:55:48 xo1 xo-server[1409]:       at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/xapi/vdi.mjs:261:5)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async XapiVhdStreamSource.init (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async #openExportStream (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/xapi/disks/Xapi.mjs:182:7)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/xapi/disks/Xapi.mjs:97:22)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/xapi/disks/Xapi.mjs:258:18)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Apr 15 22:55:48 xo1 xo-server[1409]:       at async file:///opt/xo/xo-builds/xen-orchestra-202604151415/@xen-orchestra/backups/_incrementalVm.mjs:66:5
Apr 15 22:55:48 xo1 xo-server[1409]: }
Apr 15 22:55:48 xo1 xo-server[1409]: 2026-04-16T02:55:48.126Z xo:xapi:xapi-disks INFO export through vhd
Apr 15 22:56:24 xo1 xo-server[1409]: 2026-04-16T02:56:24.047Z xo:backups:worker INFO backup has ended
Apr 15 22:56:24 xo1 xo-server[1409]: 2026-04-16T02:56:24.231Z xo:backups:worker INFO process will exit {
Apr 15 22:56:24 xo1 xo-server[1409]:   duration: 43618102,
Apr 15 22:56:24 xo1 xo-server[1409]:   exitCode: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:   resourceUsage: {
Apr 15 22:56:24 xo1 xo-server[1409]:     userCPUTime: 45307253,
Apr 15 22:56:24 xo1 xo-server[1409]:     systemCPUTime: 6674413,
Apr 15 22:56:24 xo1 xo-server[1409]:     maxRSS: 30928,
Apr 15 22:56:24 xo1 xo-server[1409]:     sharedMemorySize: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     unsharedDataSize: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     unsharedStackSize: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     minorPageFault: 287968,
Apr 15 22:56:24 xo1 xo-server[1409]:     majorPageFault: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     swappedOut: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     fsRead: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     fsWrite: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     ipcSent: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     ipcReceived: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     signalsCount: 0,
Apr 15 22:56:24 xo1 xo-server[1409]:     voluntaryContextSwitches: 14665,
Apr 15 22:56:24 xo1 xo-server[1409]:     involuntaryContextSwitches: 962
Apr 15 22:56:24 xo1 xo-server[1409]:   },
Apr 15 22:56:24 xo1 xo-server[1409]:   summary: { duration: '44s', cpuUsage: '119%', memoryUsage: '30.2 MiB' }
Apr 15 22:56:24 xo1 xo-server[1409]: }

@tsukraw It's not your fault, it's a bug (@florent, two bugs) in XO. Backup should fall back to non-NBD and warn there is a network issue, second it should not say delta at the bottom when it's doing a full backup. Known issues, I hit both last month without warnings...

I do hourly CR and I have the full interval set to 168 (once a week). You could set it longer if you have a slow link but backups are reliable. The full backup on CR is more of a data integrity issue. Deltas are just updates to the existing data. If there is a data problem with a delta then the copy is no longer 100% correct. The full backup (assuming it makes it over to the other site) ensures a full fresh copy without relying on past deltas being correct.

If you do CR nightly then setting full to 30 would be once a month. It would be nice to have a skew option to have full backups automatically done on some VMs but not all at the same time.

@rzr Always a reboot after big updates, as instructed/required.

@rzr Installed on test machines with some warnings:

  Updating   : blktap-3.55.5-6.4.xcpng8.3.x86_64                           9/87
cat: /usr/lib/udev/rules.d/65-md-incremental.rules: No such file or directory
warning: %triggerin(blktap-3.55.5-6.4.xcpng8.3.x86_64) scriptlet failed, exit status 1
Non-fatal <unknown> scriptlet failure in rpm package blktap-3.55.5-6.4.xcpng8.3. x86_64

  Updating   : sm-fairlock-3.2.12-17.2.xcpng8.3.x86_64                    32/87
Warning: fairlock@devicemapper.service changed on disk. Run 'systemctl daemon-reload' to reload units.

@tsukraw The type delta at the bottom is a known bug... Do you have NBD Connection enabled on a network interface (Check the pool network)?

@tsukraw It's running a full, not a delta. Do you have Use NBD enabled?

@McHenry What do you have Number of NBD connection per disk set to?

Node v20 is EOL at the end of the month (April 2026)...

Come on everyone! Click Here to vote to support HP IPMI info in XO!

@florent Here's one more thing that needs improvement related to full backups running when it should be a delta.

I am running a CR job between two servers (single server pool for each). Each time the CR delta job would say it was falling back to a full backup without a reason or error.

I had CBT and NBD selected. It did not matter if it deleted the snapshot or not. It always ran a full backup.... The problem/solution I found was the server did not have any interface with NBD enabled. Once I enabled the management interface as NBD then delta backup/CR worked correctly.

The backup job config says NBD, if available. Not being available seems to cause an issue.

It would be nice to have the delta backup not fail if there is no NBD interface, or have it give a warning/error so you know there's an issue. Or it should work as advertised and not use or require NBD that's not available.

@olivierlambert I would think HP servers would be a top tier for IPMI support.... please...

@gduperrey Up and working on several Intel pools.

@florent I updated XO to master 60ba5, and still having the issue.

I don't see any errors in the backup log. It just says "isFull": true for the ones stuck on full backups.

The XO log just reports:

Mar 24 16:26:47 xo1 xo-server[21199]: 2026-03-24T20:26:47.315Z xo:backups:worker INFO starting backup
Mar 24 16:26:52 xo1 xo-server[21199]: 2026-03-24T20:26:52.359Z xo:xapi:xapi-disks INFO export through vhd
Mar 24 16:26:54 xo1 xo-server[21199]: 2026-03-24T20:26:54.410Z xo:xapi:xapi-disks INFO export through vhd
Mar 24 16:29:22 xo1 xo-server[21199]: 2026-03-24T20:29:22.250Z xo:backups:worker INFO backup has ended

Note, it reports two export through vhd and does two full backups and one delta.

@acebmxer Yes, 22 or 24 should work correctly. Some early versions of 24 had issues, but not currently. I'm running 24 and it is working.

@florent I too am having issues with this... on some VMs...

Running XCP 8.3 (latest March update), XO (master 0286f).

I have a simple small CR job that replicates 3 VMs between two pools. Two VMs got stuck at Full, and one works correctly. CBT/NBD 1 enabled. Purge disabled.

It's also leaving snapshots each time. No CD in the drive this time.

I'll have to update XO and try again.

@acebmxer Use current Node LTS version... Currently 24.14.0
Or you can use 22.22.1.... use the latest version.

22.3 is the oldest version allowed to be supported (don't use one that old).

@florent I rebuilt my XCP hosting environment (everything is faster and bigger stuffed into one rack).... and this issue is now worse.

The main changes to this new setup are 2x40Gb networking, faster NFS NVMe NAS, faster pool servers, more memory, much faster CR destination machine with ZFS.

Running XCP 8.3 (March 2026 updates) and XO (master a2e33).

Replication is leaving many attached to control domain every day with NBD connection set to 2. Changing it to 1 seems to resolve the issue (no more stuck to control domain).

@yann Item Opened on Gitlab.

@olivierlambert Error causes tools to fail to load... >256 IP addresses on an interface (IPv4 and IPv6):

xen-guest-agent[514]: Error: Custom { kind: OutOfMemory, error: "no free slot for a new IP address" }

Best is to account for all the IP addresses.... second best is to ignore the extra IP addresses and not fail and exit.