clean-vm (end) is stalling ?

Pilow

another example, on another client server :

yesterday we were on XOA 6.0.3

today we updated to XOA 6.3.2

clean-vm get from 4min to an hour on this single VM ! for barely 5Gb transfer size
same VMs, same NFS remote on same NAS, nothing changed except XOA version

whole backup window on this 12 VMs job is shifted +4hours (still not finished...)

olivierlambert

Ping @Team-XO-Backend

simonp

@Pilow
Hi, thanks for the report we will look into this and get back to you ASAP.

MajorP93

@Pilow I noticed the same thing.
After upgrading my XO instance my backups jobs started to take waaaay longer to finish.
I also noticed that the clean-vm / merging seems to slow down the backup jobs quite heavily.

@simonp
Maybe it would be a good idea to test these kind of things before releasing. Maybe by adding time thresholds to the unit tests as in "if step a or b takes longer than x seconds it is considered failed"?

simonp

@MajorP93

Hi, yes this has clearly been a blind spot of our testing when we released the backups refactoring and our first step in fixing this issue is to add benchmarks to our testing suite to be sure this can't happen again.

MajorP93

@simonp Awesome. Thanks for working on this.

simonp

@pilow @majorp93

Hi again, we have found the most likely source of the issue and fixed it.

Could you checkout the fix_vhd_directory_merge branch and test that the backups take a reasonable length of time again ?

Thanks.

MajorP93

@simonp I'll have to wait for the currently running backup job to finish.
I will update my XO instance afterwards and re-test.
I can probably give you feedback in this regard tomorrow or the day after at latest.
Thanks and best regards

simonp

@MajorP93 Perfect, if the issue is confirmed resolved, we aim to release a patch early next week so no hurry

MajorP93

@simonp Hello! Sorry for the delay. I had some time to test the branch that you mentioned.

I can say that the backup job took way less time to finish after you applied those fixes.
In my case it came down from 2 days to 10 hours.

I wanted to compare all of the backup runs of the last weeks and give you some real data but unfortunately my Xen Orchestra seems to not show the backup runs that took place before updating to your branch in backup history.

I still feel like backups take a little bit longer compared to before the merge-refactor but it is only a small difference.

Anyways. It is a big improvement and the jobs finish in reasonable time again. Thank you!

Pilow

@MajorP93 sounds promising
i didn't patch, waiting for official release.

can you tell me if your jobs have lot of concurrency configured ?
for the time being I had to down to 1 the concurrency (was 4 or 6) to mitigate the added time of simultaneous mergings

eager to see if with the patchs, we can re pump concurrency up in the jobs.

simonp

Hi, thanks for the heads-up, we will see about doing some comparison with the backups refactoring on our dev environment to check if we lost some speed and try to fix it if so. Very happy to hear that the issue is mostly resolved.

We will patch this ASAP.