clean-vm (end) is stalling ?

Pilow

Hi,

Did you notice that since latest releases (I guess from 6.3.0 upward) that the merge process in the final phase of backups is stalling ?

57 seconds of transfer phase, followed by 8min of clean VM stuff (probably merging)

2min47sec of transfer / 15min of clean-vm

when you have lots of VMs, this adds up a lot to the backup window...

how could we make this better ?
@florent could you describe what is happening in the two clean-vm phases ? start and end ?

we tried many tricks to make this better (concurrency to 1 to avoid multiple mergings happening simultaneously, add cpu & RAM to the xoproxy VM, to the minio remotes too, to better handle the load)
but quite not there yet... still having quite long merging times in regard to the transfered size (i know this isn't last transfer that is merged it's the oldest, but seeing this behavior on quite similar incremental backup transfers)

I think this is new, due to latest releases and updates to backup code

Pilow

another example, on another client server :

yesterday we were on XOA 6.0.3

today we updated to XOA 6.3.2

clean-vm get from 4min to an hour on this single VM ! for barely 5Gb transfer size
same VMs, same NFS remote on same NAS, nothing changed except XOA version

whole backup window on this 12 VMs job is shifted +4hours (still not finished...)

olivierlambert

Ping @Team-XO-Backend

simonp

@Pilow
Hi, thanks for the report we will look into this and get back to you ASAP.

MajorP93

@Pilow I noticed the same thing.
After upgrading my XO instance my backups jobs started to take waaaay longer to finish.
I also noticed that the clean-vm / merging seems to slow down the backup jobs quite heavily.

@simonp
Maybe it would be a good idea to test these kind of things before releasing. Maybe by adding time thresholds to the unit tests as in "if step a or b takes longer than x seconds it is considered failed"?

simonp

@MajorP93

Hi, yes this has clearly been a blind spot of our testing when we released the backups refactoring and our first step in fixing this issue is to add benchmarks to our testing suite to be sure this can't happen again.

MajorP93

@simonp Awesome. Thanks for working on this.

simonp

@pilow @majorp93

Hi again, we have found the most likely source of the issue and fixed it.

Could you checkout the fix_vhd_directory_merge branch and test that the backups take a reasonable length of time again ?

Thanks.

MajorP93

@simonp I'll have to wait for the currently running backup job to finish.
I will update my XO instance afterwards and re-test.
I can probably give you feedback in this regard tomorrow or the day after at latest.
Thanks and best regards

simonp

@MajorP93 Perfect, if the issue is confirmed resolved, we aim to release a patch early next week so no hurry