@itservices Great news.

Don't hesitate to post here again if you encounter a similar issue in the future.

@itservices Hello,

I'm taking this issue from Pierre.

Did you perform a full restart of the XCP-ng host since the issue occured ? Or just restarted the toolstask ?

@acebmxer Hi,

Thanks to your help we were able to identify an issue with Redis that we think is the source of the v6 dashboard loading issue.

Could you try and checkout the fix_redis_encryption_issue branch, rebuild xo and restart ?

This should solve the 401 issues.

@acebmxer Ok thanks.

I'm guessing that switching to node 22 solved the dist issue but you still have the Redis issue.

We have identified a possible problem with Redis that looks like your issue, would you mind checking out this branch fix_redis_orphaned_entry ?

Try loading the v6 dashboard again after a rebuild and restart of your XO.

Thank for your help.

@acebmxer Hi,

The empty /dist might be due to the node version, we do not yet support node 24, could you change your node version to 22 and try to build xen-orchestra again ?

Thanks.

@JL457 Hi,

For now it looks like Wasabi is not sending us the correct date format, which is strange because we support this provider and don't usually have issues.

In order to allow us to investigate further, could you send us the full backup job logs ?

You can find them by clicking on the failed backup status and then on the download logs button:

Relevant XO logs would also help.

If you are a client, also don't hesitate to open a ticket with an open support tunnel.

Thanks.

@Pilow Thanks for the heads-up, you should be able to add back concurrency as it was before and get similar performance to before the refactoring.

Hi, thanks for the heads-up, we will see about doing some comparison with the backups refactoring on our dev environment to check if we lost some speed and try to fix it if so. Very happy to hear that the issue is mostly resolved.

We will patch this ASAP.

@MajorP93 Perfect, if the issue is confirmed resolved, we aim to release a patch early next week so no hurry

@pilow @majorp93

Hi again, we have found the most likely source of the issue and fixed it.

Could you checkout the fix_vhd_directory_merge branch and test that the backups take a reasonable length of time again ?

Thanks.

@MajorP93

Hi, yes this has clearly been a blind spot of our testing when we released the backups refactoring and our first step in fixing this issue is to add benchmarks to our testing suite to be sure this can't happen again.

@Pilow
Hi, thanks for the report we will look into this and get back to you ASAP.

@wralb Perfect, thanks for getting back to us.

So the issue only occurs when a delta backup tries to backup a disk that has had an increase in size, so you can either stay on the branch or go back to master if you are not planning on enlarging any disks in the near future.

Anyways, the fix should be available in the next patch or release.

Thanks.

@wmazren Hi, thanks for the report, we are looking into this.

Could you share you config.toml file ? It's most likely located in ~/.config/xo-server/.

Thanks.

Hi @wralb, thank you for the report.

We have identified the source of the issue and implemented a potential fix. It's currently on the branch fix_merge_out_of_range.

Could you check it out, retry the backup and confirm that the issue issue is fixed ?

Thanks.

Thanks for the heads-up and thanks for the help !

@acebmxer

Thanks a lot, the first issue was resolved but your log indicates a separate issue.
We think we resolved it on the "debug_backups" branch, could you test it again ?

Anyways, if this fixes the backup issue, a fix should be available tomorrow Paris time on master.
In the mean time you can revert to the commit a3139 as explained in the first post.

Thanks again for your help.

@acebmxer

Thanks a lot for the logs, we've been able to identify a possible cause for this issue, could you pull the "debug_backups" branch and try running the backups again ?

Hash is cb85e44ae

@manilx @acebmxer @marcoi

Hi, thanks for the reports, we were able to fix an issue with the health check but we weren't able to reproduce the backup issue.

Could you checkout the "debug_backups" branch, try to run the backups again and send the logs here ?

There should be no need to restart or rebuild.

Thanks !

@manilx What storage type/backup repository are you using ?