" can't compute delta" & "can't connect through NBD, fall back to stream export" after 2026-07-28

probain

Since updating to XO from source on 2026-07-28 my backups have started showing issues:
Issue still persists on 2026-07-29 on latest commit (26c36) after the latest XCP-ng updates as well.

 can't compute delta OpaqueRef:4f676f19-a405-9bf7-ebb8-508240593020 from OpaqueRef:044a8fe1-7547-175f-612a-82e8df28524a, fall back to a full
 can't connect through NBD, fall back to stream export
 Backup fell back to a full

This happens to almost all of my VMs. But not on my Windows Server 2025 VM. Unsure if that is relevant though.

2026-07-27 didn't have any such notices.

Andrew

@florent @olivierlambert I also have the same issue. I see there have been a lot of XO updates to the backups. Maybe this is fixing a hidden problem?

CR works, but normal delta backups to S3 have the issue on some of the VMs (on a test run). All the VMs on this backup are Linux.

A second run of the same delta backups all worked. I'll have to see what happens for the normal nightly backup with a lot more VMs...

(edit fix logs):

 2026-07-28T23:17:34.544Z xo:xapi:vdi WARN invalid HTTP header in response body {
   body: 'HTTP/1.1 500 Internal Error\r\n' +
     'content-length: 318\r\n' +
     'content-type: text/html\r\n' +
     'connection: close\r\n' +
     'cache-control: no-cache, no-store\r\n' +
     '\r\n' +
     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:4a172cde-c772-2b6c-100d-f63ea5e4d842; CBT metadata ]</body></html>'
 }
 2026-07-28T23:17:34.552Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:6ec8ab35-d377-cd33-ce94-72a7a4e9e3b0 from OpaqueRef:4a172cde-c772-2b6c-100d-f63ea5e4d842, fall back to a full {
   error: Error: invalid HTTP header in response body
       at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/vdi.mjs:39:19)
       at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
       at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/vdi.mjs:263:5)
       at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)
       at async XapiVhdStreamSource.init (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)
       at async #openExportStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:186:7)
       at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:97:22)
       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:268:18)
       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
       at async file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/backups/_incrementalVm.mjs:104:5
 }
 2026-07-28T23:17:34.553Z xo:xapi:xapi-disks INFO export through vhd

acebmxer

I deployed a new XO last night - for just because. No issues with backups for me as of now.

florent

@Andrew @probain
XO was hiding some error message and fell back silently ( or fail later) , this onth updates gives us more information

are you using NBD ( it is mandatory for qcow2 disks) ?
is NBD enable on at least one network per pool ?
can XO access this network ?
is there a default backup network on the pool ? with NBD ? accessible by XO ?

any more logs on journalctl ( I am expecting some nbdinfos )

probain

@florent

Unfortunately I'm not able to connect logs and such until tomorrow. But I can share further info on the configs.

The backup type is Delta.
NBD is activated and reachable
The SR on the source drive is local
Target SR is truenas nfs. Transfers are as expected.

Starting with the commits on 27th. These infos and fallback to full started happening. They have not been seen before. Resulting in transfers becoming 300% larger than expected.

All vdi are vhd
I compile from source almost daily. Hence me being very certain this is due to recent commits.

Info warning and fallback is happening even after second attempt after latest updates as well. So issue persists between reboots as well.

Is there a vates upload service I can upload to?
Or if we reset my XO trial, I can open a support tunnel for you

probain

@florent

I'm doing more backup runs. And not all of the VMs trigger the warning and fall back. But it seems quite random so far.

I did one adjustment thoug.
My number of NBD connections are set quite high, 20. And after reading this post I decided to try if lowering to 1 would have an effect. And it seems that it does. The latest backup-run hasn't showed any warnings this time... Maybe it's a false lead. But it is a behaviour I noticed this time

Andrew

@florent

Running delta backup with CBT to S3.
NBD connections=1
Yes, NBD. No VHD only.
Yes, NBD is enabled and working on all pool hosts.
Yes, XO can access the network used for NBD.
No, a default backup network as not been assigned.
No more logs from journalctl.

My normal large nightly delta backup to S3 ran and did not produce any new errors. So far, just the one error once.

poddingue

Small follow-up on the NBD connections idea, because I asked internally and it turns out that was not a false lead at all.
The number of NBD connections is capped per host, so 20 per disk is well past what the host will actually hand out, and it is worth turning down whatever else is going on in this thread.
I do not have the exact ceiling yet. I have asked the XAPI side for the real number and I will post it here once I have it, because guessing at it would not help anyone.
@Andrew, that fits your side too as far as I can tell, since you are at 1 and have seen this once rather than constantly.
probain, I will send you an upload link by direct message so you have somewhere to put the logs. I still do not know whether the fallback itself is new this week or simply newly visible, so please do not read anything into my message on that part.

probain

@poddingue

Thanks for looking into it further as well.. If there is a hard cap on the amount of NBD connections, then that is very interesting. Because the closest I've ever been able to find regarding this setting, was the general answer of "Try and see what works for you". So if ther is indeed a hard cap, then I would propose that this is limited inside the GUI itself. e.g. if the cap is 8, then don't let the integer exceed 8.

I'm doing increasing increments of the number of connections now. So far I've done 1,2,10. And 10 does seem to be breaking again, but only on some.

Thanks as well for the link. I'm collecting the logs through xen-bugtool --yestoall now. I do wish that something like this is implemented in GUI someday. And I even made a FR feedback on it. Since I really do want to provide good information for the devs. But collecting manually isn't the most convinient way unfortunately.

Logs should be up shortly

probain

5 nbd connections went fine.
8 is not.

olivierlambert

Happy to read the logs when you can provide them!

acebmxer

Just to confirm you are talking about this setting here in the backup job?

Screenshot 2026-07-30 153133.png

If so is this setting supposed to be set to the number of physical nics that can commicate to the backup remote? Or is this more about how many disks at once can be backup over those connections?

Previously had set to 2 or 4. Upped to 6 and see no difference.

olivierlambert

Logs are providing interesting results. Discussing internally about it in the next week to see what could be improved about it.

probain

@olivierlambert
Happy to hear that they are providing results to be looked at.
If there is anything else I can provide, please don't hesitate to ask. And I'll do my very best to be helpfull

IgorGlock

I am not sure if it is the same issue or not.
I doing daily replication from:
4 VMs nvme-1 SR vhd --> nvme-2 SR qcow2
2 VMs nvme-2 SR qcow2 --> nvme-1 SR vhd

That is homelab for testing only.
Xen Orchestra, commit 6a441

Standalone "Host up-to-date!" with latest august updates.

Replication qcow2 --> vhd works as expected.
Replication of vhd --> qcow2 creates daily new replica VM and reports this:

Log: 2026-08-30T18_42_49.945Z - backup NG.json.txt

IgorGlock

My reported issue is gone with build 7a187 from sourcecode.

Day before:

tasks	
0	
id	"0mtj9shwq-k2neggnx0sf"
properties	
id	"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeca"
name	"backup VM"
type	"VM"
name_label	"unifi-server"
progress	0
start	1788303601322
status	"success"
warnings	
0	
message	"can't compute delta OpaqueRef:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeee76 from OpaqueRef:aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeee32, fall back to a full"
tasks

Day after:

1	
id	"0mtktio1u-lrexahnlve"
properties	
id	"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeca"
name	"backup VM"
type	"VM"
name_label	"unifi-server"
progress	0
start	1788397201218
status	"success"
tasks	
0

probain

I've since reduced the number of NBD connections to 4. However, I'm actually still seing this happen. I thought it was resolved. But going back to confirm, proved I assumed wrong.

I have therefore uploaded a new bugtools archive from the host server on the link @poddingue sent me earlier.

I hope they can prove helpfull. If you'd like me to do anything else, then I'll happily help out.

Cheers!

florent

@probain the first checkc to do are network one :

is there a default backup network on he pool ? if yes the xo will only look at this network

check if :

the xo can access network(s) with nbd enabled
the host have an ip on those networks
are you on ipv4 or v6 ?
xo can access the ip of the host on the nbd newtork

jr-m4

@florent

I'm seing the same behaviour as well.
To answer the questions you're asking for:

XO from source (7a187)

XO is connected to three separate pools. Backups work most of the times. But I've started seeing the same error. However, it seems intermitent. And somwhat random. Since it will generate an error from one VM, but not on another on the same host with the same SR etc etc.

can't compute delta OpaqueRef: <ref>
from OpaqueRef:<ref>, fall back to a full
can't connect through NBD, fall back to stream export
Backup fell back to a full

There is an assigned backup-network, and it is the same as the management network. With NBD enabled
XO can access all of the hosts on the management/backup network
IPv4

probain

@florent

No explicit assigned migration or backup network set.

Backups generate errors somewhat randomly. As one VM won't get the "error" one time. But on the next backup run it will. And the third time it might not. This even happens on VMs that are turned off, and therefore shouldn't have any changed bytes between backups.

Host is reachable from XO
Network has NBD enabled
All VMs use VHD
IPv4
Backup type is Delta