" can't compute delta" & "can't connect through NBD, fall back to stream export" after 2026-07-28

probain

Since updating to XO from source on 2026-07-28 my backups have started showing issues:
Issue still persists on 2026-07-29 on latest commit (26c36) after the latest XCP-ng updates as well.

 can't compute delta OpaqueRef:4f676f19-a405-9bf7-ebb8-508240593020 from OpaqueRef:044a8fe1-7547-175f-612a-82e8df28524a, fall back to a full
 can't connect through NBD, fall back to stream export
 Backup fell back to a full

This happens to almost all of my VMs. But not on my Windows Server 2025 VM. Unsure if that is relevant though.

2026-07-27 didn't have any such notices.

Andrew

@florent @olivierlambert I also have the same issue. I see there have been a lot of XO updates to the backups. Maybe this is fixing a hidden problem?

CR works, but normal delta backups to S3 have the issue on some of the VMs (on a test run). All the VMs on this backup are Linux.

A second run of the same delta backups all worked. I'll have to see what happens for the normal nightly backup with a lot more VMs...

(edit fix logs):

 2026-07-28T23:17:34.544Z xo:xapi:vdi WARN invalid HTTP header in response body {
   body: 'HTTP/1.1 500 Internal Error\r\n' +
     'content-length: 318\r\n' +
     'content-type: text/html\r\n' +
     'connection: close\r\n' +
     'cache-control: no-cache, no-store\r\n' +
     '\r\n' +
     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:4a172cde-c772-2b6c-100d-f63ea5e4d842; CBT metadata ]</body></html>'
 }
 2026-07-28T23:17:34.552Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:6ec8ab35-d377-cd33-ce94-72a7a4e9e3b0 from OpaqueRef:4a172cde-c772-2b6c-100d-f63ea5e4d842, fall back to a full {
   error: Error: invalid HTTP header in response body
       at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/vdi.mjs:39:19)
       at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
       at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/vdi.mjs:263:5)
       at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)
       at async XapiVhdStreamSource.init (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)
       at async #openExportStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:186:7)
       at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:97:22)
       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:268:18)
       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
       at async file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/backups/_incrementalVm.mjs:104:5
 }
 2026-07-28T23:17:34.553Z xo:xapi:xapi-disks INFO export through vhd

acebmxer

I deployed a new XO last night - for just because. No issues with backups for me as of now.

florent

@Andrew @probain
XO was hiding some error message and fell back silently ( or fail later) , this onth updates gives us more information

are you using NBD ( it is mandatory for qcow2 disks) ?
is NBD enable on at least one network per pool ?
can XO access this network ?
is there a default backup network on the pool ? with NBD ? accessible by XO ?

any more logs on journalctl ( I am expecting some nbdinfos )

probain

@florent

Unfortunately I'm not able to connect logs and such until tomorrow. But I can share further info on the configs.

The backup type is Delta.
NBD is activated and reachable
The SR on the source drive is local
Target SR is truenas nfs. Transfers are as expected.

Starting with the commits on 27th. These infos and fallback to full started happening. They have not been seen before. Resulting in transfers becoming 300% larger than expected.

All vdi are vhd
I compile from source almost daily. Hence me being very certain this is due to recent commits.

Info warning and fallback is happening even after second attempt after latest updates as well. So issue persists between reboots as well.

Is there a vates upload service I can upload to?
Or if we reset my XO trial, I can open a support tunnel for you

probain

@florent

I'm doing more backup runs. And not all of the VMs trigger the warning and fall back. But it seems quite random so far.

I did one adjustment thoug.
My number of NBD connections are set quite high, 20. And after reading this post I decided to try if lowering to 1 would have an effect. And it seems that it does. The latest backup-run hasn't showed any warnings this time... Maybe it's a false lead. But it is a behaviour I noticed this time

Andrew

@florent

Running delta backup with CBT to S3.
NBD connections=1
Yes, NBD. No VHD only.
Yes, NBD is enabled and working on all pool hosts.
Yes, XO can access the network used for NBD.
No, a default backup network as not been assigned.
No more logs from journalctl.

My normal large nightly delta backup to S3 ran and did not produce any new errors. So far, just the one error once.