backup mail report says INTERRUPTED but it's not ?

[florent]

@Pilow I think I remember that you have a xoa ? would it be ok to open a support tunnel a give me the number by the chat ( or by a support ticket ) ?

are you querying on of the dashboard or only the individual CRUD api ? rest api or xo cli ?

We could add some debug

[flakpyro]

@florent I am also experiencing this on XOA and can send you a support tunnel number if you'd like. I rebooted it Sunday night after sharing the screenshot of memory usage above in this thread but it is slowly growing again. It used to crash and restart every 3 days but with --max-old-space-size=12288 i get around 7-8 day now before its an issue.

[florent]

@flakpyro yes, with pleasure
do you also use the rest api / xo6 ?
do you use proxies ?

[flakpyro]

@florent Not using the API but we have two Proxies.

[Pilow]

@florent rest and xo-cli
tunnel ID sent by message

[florent]

thanks guy, I will connect in a few hours, and we will investigate tomorrow all day with @mathieura

[acebmxer]

From my XOA

One of my pools have been offline for 24hrs due to power outage.

Standard XOA can open support tunnel if needed.

[florent]

On tomorrow 's release we will:

• reduce the polling for the patches list
• add more debug to know exactly what request go through the xo6 api (rest api)

following this we will add core xo metrics to the openmetrics ( prometheus) export to allow users and support to dig through real world data when needed

[MajorP93]

@florent thanks for working on this!

[Pilow]

@florent since your early patch of my XOA (xo-server restart at the disk activity peak)

noticing vif0 transmitting a bit on the network and different ram activity on a timespan of 2 hours

something has changed, is it for the better ? let the night decide

[acebmxer]

Updated to XOA 6.2.0 Will monitor memory usage and report back.

Current use before backups after update. All host and pools connected as of yesterday.

[Pilow]

uptime is 2 days for me

but restarted xo-server yesterday

zoom on 2 hours

[florent]

@Pilow we really did only add one small log . we'll need to dive deeper for a real cause ( and fix)

[Pilow]

@florent okay
but at 3 days without reboot i'm like this :

still on XOA 6.1.2

[acebmxer]

I may have fixed the issue myself. I have moved our Windows VM backups to Veeam and leaving XOA to backup linux VMs.

[MajorP93]

@acebmxer That is very interesting! So according to your tests the XOA backup memory leak is only occurring when Windows VMs are being backed up?

[acebmxer]

@MajorP93

No by reducing the number of vms that are being backed up by xoa.

I moved Windows vms to veeam as we need application aware backups.

[MajorP93]

@acebmxer Okay I see. Yeah that correlates with @florent 's observation that the problem was not visible in a small test environment. Only during high backup load the problem really becomes visible. (In my case 106 VMs, backup jobs running in parallel)

[Pilow]

@MajorP93 we have almost same size 30 backup jobs (mixed delta/cr/repl/dr) for 95+ VMs

but these are not operated by XOA but dispatched in 4 XO Proxies

and still XOA get OOM killed

[flakpyro]

Here is the latest after installing 6.2.1 on Friday.