@ph7 said:

They run once a day @ 03:04 before the normal backup that starts @ 03:10 and they have all run OK since July 8, the failed one reported 8 days ago ( post ~85 )

Hi, just to be sure, are you saying it is working ok apart one task 8 days ago?

@ph7 Hi,
Thank you for your log, can you give me the logs before the backup operation if it happens again?
Another thing to check is the exact time between your XO and your host. Maybe you should resync your clock.

I ask that because it seems XAPI was unavailable before this backup or there was a gap of 5 min between the time the backup started and XAPI actually doing this backup.

Maybe you had a xapi db sync just before this backup, do you make these backup regularly? If it is too often (every 5 min for example), it can struggle to start the backup.

@pedro_udifar Hi,
This is a b2 side error. It could be just a transient error, did you try to restart the backup?

@bogikornel @jb @gregbinsd @majorp93
We tested metadata backups every 5 min and it works in XO. It seems it might be coming from XCP-NG, can you provide us a xensource.log from the problematic timeline?

@bogikornel Can you tell us on which versions you are please?
the ones where it works and the ones where it does not work.
Thanks for your help!

@MajorP93 I am currently working on this issue but to answer you quickly, yes we have automatic tests, unit and integration tests (you can see them in *.integ.mjs and *.test.mjs files, and @xen-orchestra/qatest folder), a long QA to validate at each release. We add a regression test at every bug we encounter.
The issue is we could not reproduce your exact issue yet.

@pedro_udifar Hi,
We think we have found two issues into one: can you test this branch please? It has the changes from previous branch you built: fix_throttled_stream

This should fix the stream issue and fix the warning.

@pedro_udifar Hi, just to be sure, this is a replication job with two s3? If yes, is the source s3 encrypted or is it only the target?

@Tristis-Oris Hi,
Can you upload your complete logs from one of these jobs, please? You can DM me if you prefer

@itservices Hi, xen-bugtools is in xcp-ng hosts. You can directly get the logs from the hosts if you don't have the command.

@itservices
Can you export your logs with xen-bugtool --yestoall please? If you prefer to avoid exporting everything, we especially need /var/log/xensource.log and /var/log/SMlog
@Team-Storage do you have an idea about the source of the issue?

@skipthompson81 Hi, as you have an XOA, can you open a support tunnel to allow us to investigate further into this issue please? You can open a ticket and ask for Pierre Brunet.

This warning is a log we added to be sure we check everything in the backups but it can be alarming indeed but the checks are hard to do and can be false positive so we don't want to panic everyone.

@itservices Can you check your health dashboard where you should see " Unhealthy VDIs" ? If there are too many VDI in it, they need to be coalesced before restarting a backup.

@itservices Hi,
We did not forget you, working on it!

@itservices It may be a storage or a xo issue. You can try to restart xo-server and retry the backup. If it is solved, it was an issue with xo-server not freeing your vdi. If it still happens, it will be a storage issue

@itservices Hi,
It seems the parent of the backup chain is not found (a delta backup links to another delta and another, until you have a full backup, it seems a backup on this chain is missing).
Can you restart a backup chain to have a full backup? You can do it by reducing the retention in your backup job.

@CGB Hi, are you running XO from source with root or another user?
If it is another user, it may be not accessible from it

@FagnerMoraes The delta backups are in vdis folder. You can see it just below cache.json.gz

@FagnerMoraes I just realized you are not listing the same folder that what is backed up in your NAS backup job: on the left, you are listing incremental backups from a given folder, but in the logs I analysed the xva are in another folder

@FagnerMoraes Is your backup encrypted?