backup mail report says INTERRUPTED but it's not ?

[olivierlambert]

Okay weird, so having a very different RAM usage with Node 24 should be checked when we'll update XOA Node version.

[Pilow]

@MajorP93 this was found in the troubleshooting section of the documentation when i tried to optimise my xoa/xoproxies deployments

https://docs.xen-orchestra.com/troubleshooting#memory

[Pilow]

not looking better today. still not crashed.

but before the patches at 48h max, i topped the 8Gb and got the OOM killed process.

[Pilow]

before disruption, I prefer to patch/reboot my XOA
We got to the limit

still some memory leak somewhere guys !

[florent]

@Pilow we are still working on it, but for now we didn't find a solution

[MajorP93]

After implementing the --max-old-space-size Node parameter as recommended by @pilow it took longer time for the VM to hit the issue.
Still: backups went into interrupted status.
Memory leak seems to be still there.
With each subsequent backup run the memory usage rises and rises. After backup run the memory usage does not fully go back to "normal".

After adding the node parameter there was no heap size error on Node anymore since the heap size got increased. The system went into various OOM errors in kernel log (dmesg) despite not all RAM (8GB) being used.

This is what htop looks like with 3 backup jobs running:

[acebmxer]

While working last night i noticed one of my backups/pools did this. Got the email that it was interupted but when i looked the tasks were still running and moving data it untill it porcess all vms in that backup job.

Edit - note my backup job was run via proxy on the specific pool/job.

2026-02-19T03_00_00.028Z - backup NG.txt

Edit 2 - homelab same last backup was interupted.

2026-02-19T05_00_00.011Z - backup NG.txt

[MajorP93]

I wonder if this PR https://github.com/vatesfr/xen-orchestra/pull/9506 aims to solve the issue that was discussed in this thread.
To me it looks like it's the case as the issue seems to be related to RAM used by backup jobs not being freed correctly and the PR seems to add some garbage collection to backup jobs.
I hope that it will fix the issue and if needed I can test a branch.

b-Nollet opened this pull request in vatesfr/xen-orchestra

closed Backup tasks gc #9506

[Bastien Nollet]

Hi @MajorP93,

This PR is only about changing the way we delete old logs (linked to a bigger work of making backups use XO tasks instead of their own task system), it won't fix the issue discussed in this topic.

[MajorP93]

Hi @Bastien-Nollet,

oh okay, thanks for clarifying!

[flakpyro]

I have been having the same issue and have been watching it for the last couple weeks. Initially my XOA only had 8GB of ram assigned, i have bumped it up to 16 to try an alleviate the issue. Seems to be some sort of memory leak. This is the official XO Appliance too not XO CE.

I changed the systemd file to make use of the extra memory as per the docs,

ExecStart=/usr/local/bin/node --max-old-space-size=12288 /usr/local/bin/xo-server

It seems that over time it will just consume all of its memory until it crashes and restarts no matter how much i assign.

[MajorP93]

During the past month my backups failed (status interrupted) 1-2 times per week due to this memory leak.
When increasing heap size (node old space size) it takes longer but the backup fails when RAM usage eventually hits 100%.
I guess I’ll go with @Pilow ‘s workaround for now and create a cronjob for rebooting XO VM right before backups start.

[olivierlambert]

All of you are using a Node 24 LTS version? Do you still have the issue with Node 20 or Node 22?

[MajorP93]

@olivierlambert No.
I reverted to Node 20 as previously mentioned.
I was using Node 24 before but reverted to Node 20 as I hoped it would "fix" the issue.
Using Node 20 it takes longer for these issues to arise but in the end they arise.

3 of the users in this thread that encounter the issue said that they are using XOA.
As XOA also uses Node 20, I think most people that reported this issue actually use Node 20.

[olivierlambert]

Thanks for the recap!

[flakpyro]

@olivierlambert Using the prebuilt XOA appliance which reports:

[08:39 23] xoa@xoa:~$ node  --version
v20.18.3

[john.c]

@flakpyro said in backup mail report says INTERRUPTED but it's not ?:

@olivierlambert Using the prebuilt XOA appliance which reports:

[08:39 23] xoa@xoa:~$ node  --version
v20.18.3

@majorp93 @pilow Can you please capture some heap snapshots from during backup runs of XOA via NodeJS?

Then compare them to each other, they need to be in the following order:-

1. Snapshot before backup
2. Snapshot following first backup
3. Snapshot following second backup
4. Snapshot following third backup
5. Snapshot following subsequent backups to get to Node.js OOM (or as close as you’re willing to risk)

These will require that XOA (or XOCE) is started with Node.js heap snapshots enabled. Then open in a Chromium based browser the following url:-
chrome://inspect

The above URL will require using the browser’s DevTools features!

Another option is to integrate and enable use of Clinic.js (clinic heapprofiler), or configure node to use node-heapdump when it reaches a threshold amount.

Once your got those heap dumps your looking for the following:-

• Object types that grow massively between the snapshots.
• Large arrays or maps of backup-related objects (VMs, snapshots, jobs, tasks, etc.).
• Retained objects whose “retainers” point to long-lived structures (global, caches, singletons).

These will likely help to pin down what and where in the backup code, the memory leak is located.

Once have these a heap snapshot diff showing which object type (or types) growing by a stated size per backup will finally help the Vates developers fix this issue.

@florent I left the above for the original reporters of the memory leak issue, and/or yourselves.

[MajorP93]

@john.c If a Xen Orchestra developer asks for logs / heap snapshots I will be happy to provide them.
Ideally they should tell us what is needed from us to debug this.

That being said I am not entirely sure they are currently working on this as XO team has not been very vocal here.

Looking at Github repository commits indicates that other things are currently being prioritized over this crucial backup stability issue.

[florent]

we are trying to reproduce it here, without success for now
It would be good for us too if we can identify what cause this before the upcoming release ( thursday )

for example here my xoa at home , seems quite ok running by itself :

to be fair the debug is not helped by the fact that we have one big process, with so much object that changes that exploring and pinpointing exactly what is growing is tricky ( or we may not look at the right place)

[Pilow]

@florent do you have (heavy?) use of backup jobs on your XOA ?
we all noticed this because of correlation between backup jobs & memory growth

I do have all my backup jobs through XO PROXIES though, and XOA still explodes in RAM Usage...