backup mail report says INTERRUPTED but it's not ?

[Pilow]

@MajorP93 we have almost same size 30 backup jobs (mixed delta/cr/repl/dr) for 95+ VMs

but these are not operated by XOA but dispatched in 4 XO Proxies

and still XOA get OOM killed

[flakpyro]

Here is the latest after installing 6.2.1 on Friday.

[florent]

@flakpyro said:

Here is the latest after installing 6.2.1 on Friday.

for whose who have a big memory usage , Is a call to xo-cli xo.clean reduce the memory consumption ?

[flakpyro]

@florent I can try running this command next time memory usage is high and will report my findings!

[flakpyro]

Tonight i noticed our XOA memory usage was high today so i tried running xo-cli xo.clean as you suggested. It returned a status of "true" but unfortunately did not seem to make a difference in overall memory usage.

[florent]

@flakpyro thanks for the test
we are preparing a branch with some performance improvement , hopefully this time we found the right cause

[MajorP93]

After leaving my XO CE VM without reboot for 9 days I got hit with increased RAM usage and "interrupted" backup error again.

@florent is there a branch already that I can test?

[florent]

@MajorP93 we merged a lot of changes yesterday , so it's on master

it clean more aggressively the task log, and more cleanup after backup; We are still looking for other possibilities , but for now it's our best bets

[MajorP93]

@florent Thanks for the reply. I will rebuild XO and test with current master.

[MajorP93]

I can confirm that recent changes in Xen Orchestra improved the situation a lot.
I am still monitoring and will keep testing before coming to a conclusion but so far the backups did not crash due to Node heap size being exhausted .

[Pilow]

@MajorP93 what version of XOA are you up to ?
i'm still in 6.1.2 because of other tiny bugs in new versions

[MajorP93]

@Pilow I am using Xen Orchestra from sources. I think the commits that improved RAM / heap usage are not released for XOA. But I suspect a XOA release coming soon as they already pushed "technical release" commit which is mostly an indicator that a XOA release will follow soon.

[florent]

yes, the last changes are released in latest ( 6.3) tomorrow if everything proceed as intended

mostly : https://github.com/vatesfr/xen-orchestra/pull/9622
https://github.com/vatesfr/xen-orchestra/pull/9557

and this one was in 6.2 :
https://github.com/vatesfr/xen-orchestra/commit/e36e1012e20c9678efa15148179941cb284c39a6

that's nice to hear

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed feat(xo-server): use index for getAllUnhealthyVdiChainsLength #9622

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(backups): better handling of generator cleanup #9557

0 fbeauchamp committed to vatesfr/xen-orchestra

fix(xo-web): reducing polling for patches and license

[MathieuRA]

@pilow @majorp93
Hi, we've detected a memory leak at the REST API level.
We pushed a fix on the branch: mra-fix-rest-memory-leak
Can you test the branch?

[MajorP93]

@MathieuRA Sounds promising! Once my backup job finished I will update my XO instance and start testing. Thanks for the heads up and for working on this.
Best regards

[MajorP93]

I updated to branch "mra-fix-rest-memory-leak".
I will look at backup job results tomorrow and report back.

[MajorP93]

Hello @MathieuRA , @florent

Moving the discussion to this thread as it seems to be tackling the same issue: https://xcp-ng.org/forum/topic/11892/xoa-memory-usage/28?_=1776334896974

I reported my findings so far over there

Best regards