backup mail report says INTERRUPTED but it's not ?
-
@flakpyro said in backup mail report says INTERRUPTED but it's not ?:
@olivierlambert Using the prebuilt XOA appliance which reports:
[08:39 23] xoa@xoa:~$ node --version v20.18.3@majorp93 @pilow Can you please capture some heap snapshots from during backup runs of XOA via NodeJS?
Then compare them to each other, they need to be in the following order:-
- Snapshot before backup
- Snapshot following first backup
- Snapshot following second backup
- Snapshot following third backup
- Snapshot following subsequent backups to get to Node.js OOM (or as close as you’re willing to risk)
These will require that XOA (or XOCE) is started with Node.js heap snapshots enabled. Then open in a Chromium based browser the following url:-
chrome://inspectThe above URL will require using the browser’s DevTools features!
Another option is to integrate and enable use of Clinic.js (clinic heapprofiler), or configure node to use node-heapdump when it reaches a threshold amount.
Once your got those heap dumps your looking for the following:-
- Object types that grow massively between the snapshots.
- Large arrays or maps of backup-related objects (VMs, snapshots, jobs, tasks, etc.).
- Retained objects whose “retainers” point to long-lived structures (global, caches, singletons).
These will likely help to pin down what and where in the backup code, the memory leak is located.
Once have these a heap snapshot diff showing which object type (or types) growing by a stated size per backup will finally help the Vates developers fix this issue.
@florent I left the above for the original reporters of the memory leak issue, and/or yourselves.
-
@john.c If a Xen Orchestra developer asks for logs / heap snapshots I will be happy to provide them.
Ideally they should tell us what is needed from us to debug this.That being said I am not entirely sure they are currently working on this as XO team has not been very vocal here.
Looking at Github repository commits indicates that other things are currently being prioritized over this crucial backup stability issue.
-
we are trying to reproduce it here, without success for now
It would be good for us too if we can identify what cause this before the upcoming release ( thursday )for example here my xoa at home , seems quite ok running by itself :
to be fair the debug is not helped by the fact that we have one big process, with so much object that changes that exploring and pinpointing exactly what is growing is tricky ( or we may not look at the right place)
-
@florent do you have (heavy?) use of backup jobs on your XOA ?
we all noticed this because of correlation between backup jobs & memory growthI do have all my backup jobs through XO PROXIES though, and XOA still explodes in RAM Usage...
-
Could you try disabling XO6?
In a configuration file, enable XO5 as the default version and ensure that XO6 is inaccessible (for example, by using an invalid path or removing the corresponding entry in the configuration). This will help determine if the issue is related to XO6/REST API.https://docs.xen-orchestra.com/configuration#using-xo-5-as-the-default-interface
-
@Pilow said in backup mail report says INTERRUPTED but it's not ?:
@florent do you have (heavy?) use of backup jobs on your XOA ?
we all noticed this because of correlation between backup jobs & memory growthI do have all my backup jobs through XO PROXIES though, and XOA still explodes in RAM Usage...
I back up a only few VM on this one , it was to illustrate that we try to reproduce it with a scope small enough to improve the situation
is the memory stable on the proxy ?
-
@florent In my case I have 4 backup jobs. 1 full, 2 delta (1 onsite, 1 offsite), 1 mirror.
106 VMs are being backed up by these jobs.
In my case that setup clearly shows the RAM issue. I do not know if it can be replicated in a very small setup.
Sometimes multiple backup jobs run at the same time. It appears that backup jobs running in parallel make the issue occur faster.@mathieura Thanks for the hint! I can imagine that this could possibly change something in this regard as I did not have any issues with this exact backup job configuration some time ago. The issues started to appear somewhere in december so it could be related to the XO6 release. I will try what you suggested and report back.
-
I also have this problem in my XO-CE (ronivay-script) at home
I get the mail report after 4 days
A reboot resets the memorythe XO-CE have 3.1 GB and the Control domain memory have 2 GB
Node v24.13.1
Running Continuous Replication and Delta Backups -
@florent said in backup mail report says INTERRUPTED but it's not ?:
is the memory stable on the proxy ?
can't really say, as i reboot them everyday with my XOA
but when I do not reboot them, they dont seem to be so impacted as I need to reboot them.
I will detach them from the reboot job and will report RAM consumption on proxies -
@Pilow at least this can rule out the disk data and backup archive handling (that changed a lot in 2025 for qcow2 )
so if it's backup related, it can be the task log and the scheduler.
are you using the open metrics plugin ?
-
@florent not really, it is activated and I tried to fiddle a bit with it, why ?
-
@Pilow said in backup mail report says INTERRUPTED but it's not ?:
@florent not really, it is activated and I tried to fiddle a bit with it, why ?
the smaller the changes, the more we can focus on the right component(s). And december release was quite massive ( a good consequence of the team growing)
can you disable it to be sure ?again, thank you for your help here
-
@florent okay disabled.
We have quite an extensive use of API calls too, all over the place
and websockets (to get VM console in in-house web app)I don't know if this could have an impact
-
@Pilow I think I remember that you have a xoa ? would it be ok to open a support tunnel a give me the number by the chat ( or by a support ticket ) ?
are you querying on of the dashboard or only the individual CRUD api ? rest api or xo cli ?
We could add some debug
-
@florent I am also experiencing this on XOA and can send you a support tunnel number if you'd like. I rebooted it Sunday night after sharing the screenshot of memory usage above in this thread but it is slowly growing again. It used to crash and restart every 3 days but with --max-old-space-size=12288 i get around 7-8 day now before its an issue.
-
@flakpyro yes, with pleasure
do you also use the rest api / xo6 ?
do you use proxies ? -
@florent Not using the API but we have two Proxies.
-
@florent rest and xo-cli
tunnel ID sent by message -
thanks guy, I will connect in a few hours, and we will investigate tomorrow all day with @mathieura
-
From my XOA
One of my pools have been offline for 24hrs due to power outage.
Standard XOA can open support tunnel if needed.
