To add an update and to not leave on a cliff hanger.
We have since updated our XOA to the latest channel to attempt to fix an NBD issue.
This move broke a proxy of ours, but also all the backups are going through the XOA and after this the backups have not had an issue since.
So either the new NBD fixes, it being only on an XOA or something somehwere else resolved this problem for now.

We will be enabling the same in our other pool soon so will update if we have the same issues there.

@flakpyro ah I will try that once proxy for that pool is back
We upgrade XOA from stable channel to latest as we had another issue which is apparently resolved in that with NBD (causing some machines to go RO)
Once thats fixed I will try again to see if the above update and/or disabling "Purge snapshot" works as a workaround.

We have purge enabled (and would like it left enabled) as we use iSCSI (Dell SC5020's) so everything is a little fat, especially with some clients.
I shal update tommorow on what happens.

@rtjdamen Just trying that now
However it seems if i disable CBT on the vm, the backup (trying a new backup job for this testing) just re-enabled it.
Seems based on the job i can have NBD+CBT or neither.
Annoyingly we would like NBD to run to speed up backups as they take quite some time.

EDIT:
To add, the new test backup for the vm that failed before actually finished successfully
Just manually rrerunning it on the main job now
If it works there, the temproary workaround could be to just disable CBT and let the backup job re-enable it.

EDIT EDIT:
Re-running the backup on the vm in the original job still failed with the same error
Testing with the new job and making it the same with NBD connection set to 2, purge snapshots after, still passes fine
So i am guessing CBT is job dependant and no vm dependant?
Which would explain why a new job on the same VM to the same place works fine?

@Forza Tested this on a VM and seems i still get the same error sadly.

@rtjdamen Having a look i assumed we where on 8.3 as we updated yesterday and there is no available patches, but on 8.2.1

NAME="XCP-ng"
VERSION="8.2.1"
ID="xenenterprise"
ID_LIKE="centos rhel fedora"
VERSION_ID="8.2.1"
PRETTY_NAME="XCP-ng 8.2.1"

release/yangtze/master/58

Apologies forgot to check that and (wrongfully) assumed

We appear to have a similar issue to @flakpyro
We dont have NFS storage but using iSCSI from Dell SC5020's
We had backups with NBD and CBT enabled
We updated one of our pools to the latest (stable branch) yesterday to try and get rid of the iSCSI disconnecting bug, which meant all the vms where shuffled around and migrated.
This morning majority of the vms failed the backup with "can't create a stream from a metadata VDI, fall back to a base"
Quick searching brought me here and following what flak did i found one of the cbtlogs for one of the failed vms is also zero'd as shown below:

[09:40 xcp101 VG_XenStorage-6c2ec0ce-01ba-6975-741c-e2e86bc45e21]# cbt-util get -c -n cc2f2443-eb13-4eeb-951b-5faa3c7b8c55.cbtlog
00000000-0000-0000-0000-000000000000

We have an enterprise support with a ticket already open about NBD being slow (was on 1 NBD Connections) with a support tunnel open which I will update as well.
Hopefully that gives you another point of reference to check from.

Is it possible to force a clean fresh start for the backups similar to Veeam "Active Full"?

@julien-f Thanks for the help
Having a look at what you tried I had found my mistake on the testing being i removed %24 ($) and not %2F (/) when testing in postman
What appears to likley caused my confusion in the first place as well is when logging is set to Info, spring removed the trailing /, so it appeared as not there.
Setting logging to Debug shows the full URL string that it was sending including the trailing /
The other part that compounded the above is INFO throws a warning that URI will not encode certain characters, which i incorrectly followed as the source of the problem.

For the final "How I confused the hell out of myself", the inital URI was actually working, however it was returning a single json object (correctly as well), however Spring silently ignored this expecting an array and just returned an empty array with it. This is fixed by setting the body from a Record to "new ParameterizedTypeReference<>() {}"
Which I did at one point however did not test again with the original working string and by this time the / was either in the wrong place, which would return no objects, or %24 was removed causing a 500 as having 2 /'s isnt accepted.

Apologies for the confusion and the circle dance.

@julien-f
Afternoon
The filter i am using atm the moment as a test is a tag of Staff-RMD as shown below which was generated from the UI as well (selecting the tag from the tag dropdown)

Testing via Postman, it works if everything (including the colon and forward slash) are encoded as shown:

Problem with this is alot of libraries follow the RFC3986 (1.2.3) quite strictly and will not encode the colons or forward slash (Some even throw an invalid exception). This results in a query as below which returns an internal error:

And as a test checked running it with no encoding works and i get no results (but no error)

More than happy to test other methods or workarounds. Though I have noticed I can seem to override the URIBuilder for the Java/Spring RestClient sadly.

I have already created a StackOverflow question about this as having some issues around searching vms based on their tags using the api;
StackOverflow Q here

I am writing a Spring app to help us generate reports and daily checks with a per-client basis, which are tagged by the clients short hand names.
I have the basic curl query working for getting vms based on a tag however due to tags including : and / it gets a bit awkward with URI encoding.

From testing (as shown in the SO question) XO will only accept the tag filter if : and / are encoded, however the URI encoder used in java strictly abides by RFC 3986 and wont allow me to encode them (found this when checking in C# as well.). Even manually encoding them breaks as the % then gets encoded as well (so a bit of fun double encoding).

Is there a way to get XO to accept un-encoded : and / or if there is another way i can pass the tags across without having to put them in the URI?

Kind Regards
Rhodri MM-D