" can't compute delta" & "can't connect through NBD, fall back to stream export" after 2026-07-28

[florent]

@Andrew @probain
XO was hiding some error message and fell back silently ( or fail later) , this onth updates gives us more information

are you using NBD ( it is mandatory for qcow2 disks) ?
is NBD enable on at least one network per pool ?
can XO access this network ?
is there a default backup network on the pool ? with NBD ? accessible by XO ?

any more logs on journalctl ( I am expecting some nbdinfos )

[probain]

@florent

Unfortunately I'm not able to connect logs and such until tomorrow. But I can share further info on the configs.

The backup type is Delta.
NBD is activated and reachable
The SR on the source drive is local
Target SR is truenas nfs. Transfers are as expected.

Starting with the commits on 27th. These infos and fallback to full started happening. They have not been seen before. Resulting in transfers becoming 300% larger than expected.

All vdi are vhd
I compile from source almost daily. Hence me being very certain this is due to recent commits.

Info warning and fallback is happening even after second attempt after latest updates as well. So issue persists between reboots as well.

Is there a vates upload service I can upload to?
Or if we reset my XO trial, I can open a support tunnel for you

[probain]

@florent

I'm doing more backup runs. And not all of the VMs trigger the warning and fall back. But it seems quite random so far.

I did one adjustment thoug.
My number of NBD connections are set quite high, 20. And after reading this post I decided to try if lowering to 1 would have an effect. And it seems that it does. The latest backup-run hasn't showed any warnings this time... Maybe it's a false lead. But it is a behaviour I noticed this time

[Andrew]

@florent

Running delta backup with CBT to S3.
NBD connections=1
Yes, NBD. No VHD only.
Yes, NBD is enabled and working on all pool hosts.
Yes, XO can access the network used for NBD.
No, a default backup network as not been assigned.
No more logs from journalctl.

My normal large nightly delta backup to S3 ran and did not produce any new errors. So far, just the one error once.

[poddingue]

Small follow-up on the NBD connections idea, because I asked internally and it turns out that was not a false lead at all.
The number of NBD connections is capped per host, so 20 per disk is well past what the host will actually hand out, and it is worth turning down whatever else is going on in this thread.
I do not have the exact ceiling yet. I have asked the XAPI side for the real number and I will post it here once I have it, because guessing at it would not help anyone.
@Andrew, that fits your side too as far as I can tell, since you are at 1 and have seen this once rather than constantly.
probain, I will send you an upload link by direct message so you have somewhere to put the logs. I still do not know whether the fallback itself is new this week or simply newly visible, so please do not read anything into my message on that part.

[probain]

@poddingue

Thanks for looking into it further as well.. If there is a hard cap on the amount of NBD connections, then that is very interesting. Because the closest I've ever been able to find regarding this setting, was the general answer of "Try and see what works for you". So if ther is indeed a hard cap, then I would propose that this is limited inside the GUI itself. e.g. if the cap is 8, then don't let the integer exceed 8.

I'm doing increasing increments of the number of connections now. So far I've done 1,2,10. And 10 does seem to be breaking again, but only on some.

Thanks as well for the link. I'm collecting the logs through xen-bugtool --yestoall now. I do wish that something like this is implemented in GUI someday. And I even made a FR feedback on it. Since I really do want to provide good information for the devs. But collecting manually isn't the most convinient way unfortunately.

Logs should be up shortly

[probain]

5 nbd connections went fine.
8 is not.

[olivierlambert]

Happy to read the logs when you can provide them!

[acebmxer]

Just to confirm you are talking about this setting here in the backup job?

If so is this setting supposed to be set to the number of physical nics that can commicate to the backup remote? Or is this more about how many disks at once can be backup over those connections?

Previously had set to 2 or 4. Upped to 6 and see no difference.

[olivierlambert]

Logs are providing interesting results. Discussing internally about it in the next week to see what could be improved about it.

[probain]

@olivierlambert
Happy to hear that they are providing results to be looked at.
If there is anything else I can provide, please don't hesitate to ask. And I'll do my very best to be helpfull