XOA - Memory Usage
-
Well i guess i should not have said anything. Backups have been running fine until tonight. Well technically last night. Got email alter that a backup job failed because job already running. This is one of my remote pools. But took a look at the memory usage in XOA..
9 total vms across 3 pools/locations. 3 vms at the remote site get backed up.
The proxy at the remote site.
If you want i can create a support ticket.
-
@acebmxer yes please ping Mathieu R and me ( Florent B ) on the ticket so we can get it as fast as possible
Would it be ok to extract the memory of the xoa with this PR https://github.com/vatesfr/xen-orchestra/pull/9704 ?
it should gives us pointersdraft feat(xo-server): allow heap memory export #9704
-
@florent @mathieura Support ticket opened - Ticket#7756329
As noted in ticket Yes that is ok. Just note this is production systems.
-
XOA is giving alart about Memory running low.... I have not rebooted xoa or aything since @florent applied patchs to xoa. Support tunnel still open.
-
@acebmxer thanks to you we have discove a new class of candidates for the fix : the TLS session that stayed behind each connection.
the branch is here https://github.com/vatesfr/xen-orchestra/pull/9725 , as soon as we are done on a QA we will patch your xoa
-
@florent http 401 error on the pull
odd -
-
-
-
@florent even in private navigation
-
-
-
@ph7
let's call it a day@acebmxer after 3 hours the consumption was slightly rising, so we also deployed the latest rest api patch which is already on master : https://github.com/vatesfr/xen-orchestra/commit/0fef765ce6b4bc96b28e6af9be3d3dba3fa7dc1e
-
Looks good so far. Thanks for all your help.
-
@florent was finally able to read the pull
/clap ! the fix seems totally legit and consistant with XOA ram ramping up !when will this be officially published ?
so we can disable daily reboot of XOA & XO PROXies
-
that's better but it still grew during the night
(and the process did an aout of memory when I asked for a memory dump)I reduced the memory allocated to the rest api and will continue monitoring
-
Hello @bastien-nollet @florent
moving the discussion from this thread (link), to this one as it seems to be tackling the same issue.
This is the RAM usage of my Xen Orchestra instance before applying "fix(rest-api): fix memory-leak on subscription"
Between 04/07 and 04/10 the system was not rebooted. The RAM usage increased step by step as shown in the screenshot caused by backup runs. Each backup run increased RAM usage and it did not go back to where it was before the backup run.
This is the RAM usage after applying said fix via branch "mra-fix-rest-memory-leak".
Between 04/14 and 04/16 the VM was not restarted.
Backup jobs are scheduled to run every day.
RAM usage still increased a bit by each backup run but it seems to be less.
Looks like an improvement (which is great!) but maybe the issue is not fully fixed yet.I can not fully tell as I need to keep monitoring this a bit more.
I will keep having an eye on this and report back.
Thanks for working on this and best regards.
-
M MajorP93 referenced this topic
-
After backup jobs ran last night...
-
@MajorP93 I think there are multiple issue for memory . Pillow and acebmxer uses proxy , so the backup memory is handled differently
are you using XO from source or a xoa ? would you be ok to export a memory heap ?
-
maybe a good news ?
is completely stable since yesterday's patch
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login