XOA - Memory Usage

[florent]

@acebmxer thanks to you we have discove a new class of candidates for the fix : the TLS session that stayed behind each connection.

the branch is here https://github.com/vatesfr/xen-orchestra/pull/9725 , as soon as we are done on a QA we will patch your xoa

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-server): TLSSocket memory leak — stoppable + http-server-plus incompatibility #9725

[Pilow]

@florent http 401 error on the pull odd

[acebmxer]

@florent

This is from when patch was applied today.

[florent]

@Pilow said:

@florent http 401 error on the pull odd

can you retry ?

[florent]

@acebmxer said:

@florent

This is from when patch was applied today.

I am exactly on the same screen, but I won't jinx it and emit a hope before at least a few more hours

[Pilow]

@florent even in private navigation

[acebmxer]

@Pilow

Page loads for me.

[ph7]

@florent
I get the same result as @pilow
I'm not logged in if that has anything to do with it

[florent]

@ph7

let's call it a day

@acebmxer after 3 hours the consumption was slightly rising, so we also deployed the latest rest api patch which is already on master : https://github.com/vatesfr/xen-orchestra/commit/0fef765ce6b4bc96b28e6af9be3d3dba3fa7dc1e

0 MathieuRA committed to vatesfr/xen-orchestra

fix(rest-api): fix memory-leak on subscription (#9707)

[acebmxer]

Looks good so far. Thanks for all your help.

[Pilow]

@florent was finally able to read the pull
/clap ! the fix seems totally legit and consistant with XOA ram ramping up !

when will this be officially published ?
so we can disable daily reboot of XOA & XO PROXies

[florent]

that's better but it still grew during the night

(and the process did an aout of memory when I asked for a memory dump)

I reduced the memory allocated to the rest api and will continue monitoring

[MajorP93]

Hello @bastien-nollet @florent

moving the discussion from this thread (link), to this one as it seems to be tackling the same issue.

This is the RAM usage of my Xen Orchestra instance before applying "fix(rest-api): fix memory-leak on subscription"

Between 04/07 and 04/10 the system was not rebooted. The RAM usage increased step by step as shown in the screenshot caused by backup runs. Each backup run increased RAM usage and it did not go back to where it was before the backup run.

This is the RAM usage after applying said fix via branch "mra-fix-rest-memory-leak".

Between 04/14 and 04/16 the VM was not restarted.
Backup jobs are scheduled to run every day.
RAM usage still increased a bit by each backup run but it seems to be less.
Looks like an improvement (which is great!) but maybe the issue is not fully fixed yet.

I can not fully tell as I need to keep monitoring this a bit more.

I will keep having an eye on this and report back.

Thanks for working on this and best regards.

[acebmxer]

After backup jobs ran last night...

[florent]

@MajorP93 I think there are multiple issue for memory . Pillow and acebmxer uses proxy , so the backup memory is handled differently

are you using XO from source or a xoa ? would you be ok to export a memory heap ?

[florent]

maybe a good news ?

is completely stable since yesterday's patch

[MajorP93]

@florent Yes I am using XO from sources:

Currently on the branch that @bastien-nollet asked me to test.

Yeah sure I can export a memory heap. Can you give me instructions for that?
Also: currently I have one backup job running. Is it better to take the memory heap snapshot while backup is running or afterwards?

Best regards

[florent]

@MajorP93 I am not sure of which branch.
can you switch to https://github.com/vatesfr/xen-orchestra/pull/9725

then

when running this branch :

wait for the memory to fill
find the process running xo ( dist/cli.mjs )
send signal : kill -SIGUSR2
a file is created in `/tmp/xo-server-${process.pid}-${Date.now()}.heapsnapshot` 

Note that these exports can contain sensitive data. This will freeze you xo from a few seconds to 2 minutes, and the memory consumption after will be higher after, but this will be very usefull to identify exactly what is leaking

The file will be a few hundred MB

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-server): TLSSocket memory leak — stoppable + http-server-plus incompatibility #9725

[florent]

we merge the code into master ( including the code to allow a heap memory export )

[MajorP93]

@florent Thanks!
I did not have time (yet) to look into the heap export due to weekend being in between.
I will update to current master and provide you with the heap export in the following days.
Best regards