backup mail report says INTERRUPTED but it's not ?
-
@Pilow said in backup mail report says INTERRUPTED but it's not ?:
@florent do you have (heavy?) use of backup jobs on your XOA ?
we all noticed this because of correlation between backup jobs & memory growthI do have all my backup jobs through XO PROXIES though, and XOA still explodes in RAM Usage...
I back up a only few VM on this one , it was to illustrate that we try to reproduce it with a scope small enough to improve the situation
is the memory stable on the proxy ?
-
@florent In my case I have 4 backup jobs. 1 full, 2 delta (1 onsite, 1 offsite), 1 mirror.
106 VMs are being backed up by these jobs.
In my case that setup clearly shows the RAM issue. I do not know if it can be replicated in a very small setup.
Sometimes multiple backup jobs run at the same time. It appears that backup jobs running in parallel make the issue occur faster.@mathieura Thanks for the hint! I can imagine that this could possibly change something in this regard as I did not have any issues with this exact backup job configuration some time ago. The issues started to appear somewhere in december so it could be related to the XO6 release. I will try what you suggested and report back.
-
I also have this problem in my XO-CE (ronivay-script) at home
I get the mail report after 4 days
A reboot resets the memorythe XO-CE have 3.1 GB and the Control domain memory have 2 GB
Node v24.13.1
Running Continuous Replication and Delta Backups -
@florent said in backup mail report says INTERRUPTED but it's not ?:
is the memory stable on the proxy ?
can't really say, as i reboot them everyday with my XOA
but when I do not reboot them, they dont seem to be so impacted as I need to reboot them.
I will detach them from the reboot job and will report RAM consumption on proxies -
@Pilow at least this can rule out the disk data and backup archive handling (that changed a lot in 2025 for qcow2 )
so if it's backup related, it can be the task log and the scheduler.
are you using the open metrics plugin ?
-
@florent not really, it is activated and I tried to fiddle a bit with it, why ?
-
@Pilow said in backup mail report says INTERRUPTED but it's not ?:
@florent not really, it is activated and I tried to fiddle a bit with it, why ?
the smaller the changes, the more we can focus on the right component(s). And december release was quite massive ( a good consequence of the team growing)
can you disable it to be sure ?again, thank you for your help here
-
@florent okay disabled.
We have quite an extensive use of API calls too, all over the place
and websockets (to get VM console in in-house web app)I don't know if this could have an impact
-
@Pilow I think I remember that you have a xoa ? would it be ok to open a support tunnel a give me the number by the chat ( or by a support ticket ) ?
are you querying on of the dashboard or only the individual CRUD api ? rest api or xo cli ?
We could add some debug
-
@florent I am also experiencing this on XOA and can send you a support tunnel number if you'd like. I rebooted it Sunday night after sharing the screenshot of memory usage above in this thread but it is slowly growing again. It used to crash and restart every 3 days but with --max-old-space-size=12288 i get around 7-8 day now before its an issue.
-
@flakpyro yes, with pleasure
do you also use the rest api / xo6 ?
do you use proxies ? -
@florent Not using the API but we have two Proxies.
-
@florent rest and xo-cli
tunnel ID sent by message -
thanks guy, I will connect in a few hours, and we will investigate tomorrow all day with @mathieura
-
From my XOA
One of my pools have been offline for 24hrs due to power outage.
Standard XOA can open support tunnel if needed.
