backup mail report says INTERRUPTED but it's not ?

[MajorP93]

@florent thanks for working on this!

[Pilow]

@florent since your early patch of my XOA (xo-server restart at the disk activity peak)

noticing vif0 transmitting a bit on the network and different ram activity on a timespan of 2 hours

something has changed, is it for the better ? let the night decide

[acebmxer]

Updated to XOA 6.2.0 Will monitor memory usage and report back.

Current use before backups after update. All host and pools connected as of yesterday.

[Pilow]

uptime is 2 days for me

but restarted xo-server yesterday

zoom on 2 hours

[florent]

@Pilow we really did only add one small log . we'll need to dive deeper for a real cause ( and fix)

[Pilow]

@florent okay
but at 3 days without reboot i'm like this :

still on XOA 6.1.2

[acebmxer]

I may have fixed the issue myself. I have moved our Windows VM backups to Veeam and leaving XOA to backup linux VMs.

[MajorP93]

@acebmxer That is very interesting! So according to your tests the XOA backup memory leak is only occurring when Windows VMs are being backed up?

[acebmxer]

@MajorP93

No by reducing the number of vms that are being backed up by xoa.

I moved Windows vms to veeam as we need application aware backups.

[MajorP93]

@acebmxer Okay I see. Yeah that correlates with @florent 's observation that the problem was not visible in a small test environment. Only during high backup load the problem really becomes visible. (In my case 106 VMs, backup jobs running in parallel)

[Pilow]

@MajorP93 we have almost same size 30 backup jobs (mixed delta/cr/repl/dr) for 95+ VMs

but these are not operated by XOA but dispatched in 4 XO Proxies

and still XOA get OOM killed

[flakpyro]

Here is the latest after installing 6.2.1 on Friday.

[florent]

@flakpyro said:

Here is the latest after installing 6.2.1 on Friday.

for whose who have a big memory usage , Is a call to xo-cli xo.clean reduce the memory consumption ?

[flakpyro]

@florent I can try running this command next time memory usage is high and will report my findings!

[flakpyro]

Tonight i noticed our XOA memory usage was high today so i tried running xo-cli xo.clean as you suggested. It returned a status of "true" but unfortunately did not seem to make a difference in overall memory usage.

[florent]

@flakpyro thanks for the test
we are preparing a branch with some performance improvement , hopefully this time we found the right cause

[MajorP93]

After leaving my XO CE VM without reboot for 9 days I got hit with increased RAM usage and "interrupted" backup error again.

@florent is there a branch already that I can test?

[florent]

@MajorP93 we merged a lot of changes yesterday , so it's on master

it clean more aggressively the task log, and more cleanup after backup; We are still looking for other possibilities , but for now it's our best bets

[MajorP93]

@florent Thanks for the reply. I will rebuild XO and test with current master.

[MajorP93]

I can confirm that recent changes in Xen Orchestra improved the situation a lot.
I am still monitoring and will keep testing before coming to a conclusion but so far the backups did not crash due to Node heap size being exhausted .