backup mail report says INTERRUPTED but it's not ?

[MajorP93]

@florent In my case I have 4 backup jobs. 1 full, 2 delta (1 onsite, 1 offsite), 1 mirror.
106 VMs are being backed up by these jobs.
In my case that setup clearly shows the RAM issue. I do not know if it can be replicated in a very small setup.
Sometimes multiple backup jobs run at the same time. It appears that backup jobs running in parallel make the issue occur faster.

@mathieura Thanks for the hint! I can imagine that this could possibly change something in this regard as I did not have any issues with this exact backup job configuration some time ago. The issues started to appear somewhere in december so it could be related to the XO6 release. I will try what you suggested and report back.

[ph7]

I also have this problem in my XO-CE (ronivay-script) at home
I get the mail report after 4 days
A reboot resets the memory

the XO-CE have 3.1 GB and the Control domain memory have 2 GB
Node v24.13.1
Running Continuous Replication and Delta Backups

[Pilow]

@florent said in backup mail report says INTERRUPTED but it's not ?:

is the memory stable on the proxy ?

can't really say, as i reboot them everyday with my XOA

but when I do not reboot them, they dont seem to be so impacted as I need to reboot them.
I will detach them from the reboot job and will report RAM consumption on proxies

[florent]

@Pilow at least this can rule out the disk data and backup archive handling (that changed a lot in 2025 for qcow2 )

so if it's backup related, it can be the task log and the scheduler.

are you using the open metrics plugin ?

[Pilow]

@florent not really, it is activated and I tried to fiddle a bit with it, why ?

[florent]

@Pilow said in backup mail report says INTERRUPTED but it's not ?:

@florent not really, it is activated and I tried to fiddle a bit with it, why ?

the smaller the changes, the more we can focus on the right component(s). And december release was quite massive ( a good consequence of the team growing)
can you disable it to be sure ?

again, thank you for your help here

[Pilow]

@florent okay disabled.
We have quite an extensive use of API calls too, all over the place
and websockets (to get VM console in in-house web app)

I don't know if this could have an impact

[florent]

@Pilow I think I remember that you have a xoa ? would it be ok to open a support tunnel a give me the number by the chat ( or by a support ticket ) ?

are you querying on of the dashboard or only the individual CRUD api ? rest api or xo cli ?

We could add some debug

[flakpyro]

@florent I am also experiencing this on XOA and can send you a support tunnel number if you'd like. I rebooted it Sunday night after sharing the screenshot of memory usage above in this thread but it is slowly growing again. It used to crash and restart every 3 days but with --max-old-space-size=12288 i get around 7-8 day now before its an issue.

[florent]

@flakpyro yes, with pleasure
do you also use the rest api / xo6 ?
do you use proxies ?

[flakpyro]

@florent Not using the API but we have two Proxies.

[Pilow]

@florent rest and xo-cli
tunnel ID sent by message

[florent]

thanks guy, I will connect in a few hours, and we will investigate tomorrow all day with @mathieura

[acebmxer]

From my XOA

One of my pools have been offline for 24hrs due to power outage.

Standard XOA can open support tunnel if needed.

[florent]

On tomorrow 's release we will:

• reduce the polling for the patches list
• add more debug to know exactly what request go through the xo6 api (rest api)

following this we will add core xo metrics to the openmetrics ( prometheus) export to allow users and support to dig through real world data when needed

[MajorP93]

@florent thanks for working on this!

[Pilow]

@florent since your early patch of my XOA (xo-server restart at the disk activity peak)

noticing vif0 transmitting a bit on the network and different ram activity on a timespan of 2 hours

something has changed, is it for the better ? let the night decide

[acebmxer]

Updated to XOA 6.2.0 Will monitor memory usage and report back.

Current use before backups after update. All host and pools connected as of yesterday.

[Pilow]

uptime is 2 days for me

but restarted xo-server yesterday

zoom on 2 hours

[florent]

@Pilow we really did only add one small log . we'll need to dive deeper for a real cause ( and fix)