XOA - Memory Usage

[acebmxer]

@florent

We have allways been a veeam customer since vwmare. I have moved over to the beta version of veeam for our windows servers.

XOA is only backing up our linux servers. So half the workload was removed from backups.

[acebmxer]

Well i guess i should not have said anything. Backups have been running fine until tonight. Well technically last night. Got email alter that a backup job failed because job already running. This is one of my remote pools. But took a look at the memory usage in XOA..

9 total vms across 3 pools/locations. 3 vms at the remote site get backed up.

The proxy at the remote site.

If you want i can create a support ticket.

[florent]

@acebmxer yes please ping Mathieu R and me ( Florent B ) on the ticket so we can get it as fast as possible

Would it be ok to extract the memory of the xoa with this PR https://github.com/vatesfr/xen-orchestra/pull/9704 ?
it should gives us pointers

fbeauchamp opened this pull request in vatesfr/xen-orchestra

draft feat(xo-server): allow heap memory export #9704

[acebmxer]

@florent @mathieura Support ticket opened - Ticket#7756329

As noted in ticket Yes that is ok. Just note this is production systems.

[acebmxer]

@florent @mathieura

XOA is giving alart about Memory running low.... I have not rebooted xoa or aything since @florent applied patchs to xoa. Support tunnel still open.

[florent]

@acebmxer thanks to you we have discove a new class of candidates for the fix : the TLS session that stayed behind each connection.

the branch is here https://github.com/vatesfr/xen-orchestra/pull/9725 , as soon as we are done on a QA we will patch your xoa

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-server): TLSSocket memory leak — stoppable + http-server-plus incompatibility #9725

[Pilow]

@florent http 401 error on the pull odd

[acebmxer]

@florent

This is from when patch was applied today.

[florent]

@Pilow said:

@florent http 401 error on the pull odd

can you retry ?

[florent]

@acebmxer said:

@florent

This is from when patch was applied today.

I am exactly on the same screen, but I won't jinx it and emit a hope before at least a few more hours

[Pilow]

@florent even in private navigation

[acebmxer]

@Pilow

Page loads for me.

[ph7]

@florent
I get the same result as @pilow
I'm not logged in if that has anything to do with it

[florent]

@ph7

let's call it a day

@acebmxer after 3 hours the consumption was slightly rising, so we also deployed the latest rest api patch which is already on master : https://github.com/vatesfr/xen-orchestra/commit/0fef765ce6b4bc96b28e6af9be3d3dba3fa7dc1e

0 MathieuRA committed to vatesfr/xen-orchestra

fix(rest-api): fix memory-leak on subscription (#9707)

[acebmxer]

Looks good so far. Thanks for all your help.

[Pilow]

@florent was finally able to read the pull
/clap ! the fix seems totally legit and consistant with XOA ram ramping up !

when will this be officially published ?
so we can disable daily reboot of XOA & XO PROXies

[florent]

that's better but it still grew during the night

(and the process did an aout of memory when I asked for a memory dump)

I reduced the memory allocated to the rest api and will continue monitoring

[MajorP93]

Hello @bastien-nollet @florent

moving the discussion from this thread (link), to this one as it seems to be tackling the same issue.

This is the RAM usage of my Xen Orchestra instance before applying "fix(rest-api): fix memory-leak on subscription"

Between 04/07 and 04/10 the system was not rebooted. The RAM usage increased step by step as shown in the screenshot caused by backup runs. Each backup run increased RAM usage and it did not go back to where it was before the backup run.

This is the RAM usage after applying said fix via branch "mra-fix-rest-memory-leak".

Between 04/14 and 04/16 the VM was not restarted.
Backup jobs are scheduled to run every day.
RAM usage still increased a bit by each backup run but it seems to be less.
Looks like an improvement (which is great!) but maybe the issue is not fully fixed yet.

I can not fully tell as I need to keep monitoring this a bit more.

I will keep having an eye on this and report back.

Thanks for working on this and best regards.

[acebmxer]

After backup jobs ran last night...

[florent]

@MajorP93 I think there are multiple issue for memory . Pillow and acebmxer uses proxy , so the backup memory is handled differently

are you using XO from source or a xoa ? would you be ok to export a memory heap ?