clean-vm (end) is stalling ?

Pilow

Hi,

Did you notice that since latest releases (I guess from 6.3.0 upward) that the merge process in the final phase of backups is stalling ?

57 seconds of transfer phase, followed by 8min of clean VM stuff (probably merging)

2min47sec of transfer / 15min of clean-vm

when you have lots of VMs, this adds up a lot to the backup window...

how could we make this better ?
@florent could you describe what is happening in the two clean-vm phases ? start and end ?

we tried many tricks to make this better (concurrency to 1 to avoid multiple mergings happening simultaneously, add cpu & RAM to the xoproxy VM, to the minio remotes too, to better handle the load)
but quite not there yet... still having quite long merging times in regard to the transfered size (i know this isn't last transfer that is merged it's the oldest, but seeing this behavior on quite similar incremental backup transfers)

I think this is new, due to latest releases and updates to backup code

Pilow

another example, on another client server :

yesterday we were on XOA 6.0.3

today we updated to XOA 6.3.2

clean-vm get from 4min to an hour on this single VM ! for barely 5Gb transfer size
same VMs, same NFS remote on same NAS, nothing changed except XOA version

whole backup window on this 12 VMs job is shifted +4hours (still not finished...)

olivierlambert

Ping @Team-XO-Backend

simonp

@Pilow
Hi, thanks for the report we will look into this and get back to you ASAP.

MajorP93

@Pilow I noticed the same thing.
After upgrading my XO instance my backups jobs started to take waaaay longer to finish.
I also noticed that the clean-vm / merging seems to slow down the backup jobs quite heavily.

@simonp
Maybe it would be a good idea to test these kind of things before releasing. Maybe by adding time thresholds to the unit tests as in "if step a or b takes longer than x seconds it is considered failed"?

simonp

@MajorP93

Hi, yes this has clearly been a blind spot of our testing when we released the backups refactoring and our first step in fixing this issue is to add benchmarks to our testing suite to be sure this can't happen again.

MajorP93

@simonp Awesome. Thanks for working on this.

simonp

@pilow @majorp93

Hi again, we have found the most likely source of the issue and fixed it.

Could you checkout the fix_vhd_directory_merge branch and test that the backups take a reasonable length of time again ?

Thanks.

MajorP93

@simonp I'll have to wait for the currently running backup job to finish.
I will update my XO instance afterwards and re-test.
I can probably give you feedback in this regard tomorrow or the day after at latest.
Thanks and best regards

simonp

@MajorP93 Perfect, if the issue is confirmed resolved, we aim to release a patch early next week so no hurry

MajorP93

@simonp Hello! Sorry for the delay. I had some time to test the branch that you mentioned.

I can say that the backup job took way less time to finish after you applied those fixes.
In my case it came down from 2 days to 10 hours.

I wanted to compare all of the backup runs of the last weeks and give you some real data but unfortunately my Xen Orchestra seems to not show the backup runs that took place before updating to your branch in backup history.

I still feel like backups take a little bit longer compared to before the merge-refactor but it is only a small difference.

Anyways. It is a big improvement and the jobs finish in reasonable time again. Thank you!

Pilow

@MajorP93 sounds promising
i didn't patch, waiting for official release.

can you tell me if your jobs have lot of concurrency configured ?
for the time being I had to down to 1 the concurrency (was 4 or 6) to mitigate the added time of simultaneous mergings

eager to see if with the patchs, we can re pump concurrency up in the jobs.

simonp

Hi, thanks for the heads-up, we will see about doing some comparison with the backups refactoring on our dev environment to check if we lost some speed and try to fix it if so. Very happy to hear that the issue is mostly resolved.

We will patch this ASAP.

Pilow

@simonp patched tonight, a job who took 3 hours yesterday took only 1 tonight.
so, big improvment !

need to re up concurrency to 2 or 4 on some jobs to see if I can squeeze more time on the backup window

simonp

@Pilow Thanks for the heads-up, you should be able to add back concurrency as it was before and get similar performance to before the refactoring.