XOA - Memory Usage

[Pilow]

@flakpyro yup still aggressive on memory consumption
I have a task that reboots XOA & XOPROXIES every two days to mitigate

since 6.3 / 6.3.1 it seems more aggressive on the ramp up (I updated just yesterday... was still on 6.1.2)

[olivierlambert]

Adding @Team-XO-Backend in the loop

[MathieuRA]

@acebmxer
Hi, we've detected a memory leak at the REST API level.
We pushed a fix on the branch: mra-fix-rest-memory-leak
Can you test the branch?

[acebmxer]

@MathieuRA

I have been busy at work. I will try to test this branch out when i get home tonight.

Looking back on this. This was related to XOA at work not XO from sources.

I have also changed up our back up process and this has no longer been an issues for me.

[florent]

@acebmxer do you know what you did change on the backups ?

[acebmxer]

@florent

We have allways been a veeam customer since vwmare. I have moved over to the beta version of veeam for our windows servers.

XOA is only backing up our linux servers. So half the workload was removed from backups.

[acebmxer]

Well i guess i should not have said anything. Backups have been running fine until tonight. Well technically last night. Got email alter that a backup job failed because job already running. This is one of my remote pools. But took a look at the memory usage in XOA..

9 total vms across 3 pools/locations. 3 vms at the remote site get backed up.

The proxy at the remote site.

If you want i can create a support ticket.

[florent]

@acebmxer yes please ping Mathieu R and me ( Florent B ) on the ticket so we can get it as fast as possible

Would it be ok to extract the memory of the xoa with this PR https://github.com/vatesfr/xen-orchestra/pull/9704 ?
it should gives us pointers

fbeauchamp opened this pull request in vatesfr/xen-orchestra

draft feat(xo-server): allow heap memory export #9704

[acebmxer]

@florent @mathieura Support ticket opened - Ticket#7756329

As noted in ticket Yes that is ok. Just note this is production systems.

[acebmxer]

@florent @mathieura

XOA is giving alart about Memory running low.... I have not rebooted xoa or aything since @florent applied patchs to xoa. Support tunnel still open.

[florent]

@acebmxer thanks to you we have discove a new class of candidates for the fix : the TLS session that stayed behind each connection.

the branch is here https://github.com/vatesfr/xen-orchestra/pull/9725 , as soon as we are done on a QA we will patch your xoa

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-server): TLSSocket memory leak — stoppable + http-server-plus incompatibility #9725

[Pilow]

@florent http 401 error on the pull odd

[acebmxer]

@florent

This is from when patch was applied today.

[florent]

@Pilow said:

@florent http 401 error on the pull odd

can you retry ?

[florent]

@acebmxer said:

@florent

This is from when patch was applied today.

I am exactly on the same screen, but I won't jinx it and emit a hope before at least a few more hours

[Pilow]

@florent even in private navigation

[acebmxer]

@Pilow

Page loads for me.

[ph7]

@florent
I get the same result as @pilow
I'm not logged in if that has anything to do with it

[florent]

@ph7

let's call it a day

@acebmxer after 3 hours the consumption was slightly rising, so we also deployed the latest rest api patch which is already on master : https://github.com/vatesfr/xen-orchestra/commit/0fef765ce6b4bc96b28e6af9be3d3dba3fa7dc1e

0 MathieuRA committed to vatesfr/xen-orchestra

fix(rest-api): fix memory-leak on subscription (#9707)

[acebmxer]

Looks good so far. Thanks for all your help.