@MajorP93 this was found in the troubleshooting section of the documentation when i tried to optimise my xoa/xoproxies deployments

https://docs.xen-orchestra.com/troubleshooting#memory

@MajorP93 you say to have 8GB Ram on XO, but it OOMkills at 5Gb Used RAM.

did you do those additionnal steps in your XO Config ?

You can increase the memory allocated to the XOA VM (from 2GB to 4GB or 8GB).
Note that simply increasing the RAM for the VM is not enough.
You must also edit the service file (/etc/systemd/system/xo-server.service) 
to increase the memory allocated to the xo-server process itself.

You should leave ~512MB for the debian OS itself. Meaning if your VM has 4096MB total RAM, you should use 3584 for the memory value below.

- ExecStart=/usr/local/bin/xo-server
+ ExecStart=/usr/local/bin/node --max-old-space-size=3584 /usr/local/bin/xo-server
The last step is to refresh and restart the service:

$ systemctl daemon-reload
$ systemctl restart xo-server

i'm not foreseeing something good happening on this one
still growing

@florent
checked this morning, it took another step to 3.63Gb used RAM.

the slope is really different than before, stay tuned

@nikade you should have a TASK where you can see it downloading patches (triggering host update)
but indeed, there is a floating moment where the web ui cant connect to updater and it is doing its magic, up to an automatic toolstack restart where you host come back with the "need reboot" icon

ramp up and stabilization at 2.9Gb RAM

need some 48H more data to confirm the behavior have changed

XOPROXIES have also been upgraded to latests :

they didnt present any problem of RAM overusage even before
and my 4 proxies are offloading the entire backup jobs from main XOA thats serves only for management

so, I stopped rebooting my XOA everyday

just patched 6.1.1, it restarted xo-server

guess I'll have to let it disabled for 48h to see if with new patch, RAM is still ramping up.
will report back.

@dinhngtu tried on a Windows Server 2025 VM, UEFI BIOS
working well, no reboot needed

@florent I did healthchecks one month or two ago.. I do not remember precisely

but I didn't have this bug.

so either specific to OP installation, or there is a regression somewhere

@markxc could you tell us what endpoint you used ?

@Greg_E i think protection is doing its job. do not delete.
but in case of healthcheck, the process should bypass that for the restored VM

thoughts @bastien-nollet @florent ?

@olivierlambert okay, understood.
it is small minor cosmetic problem as of now.

need to get you an access to our internal developped app, to show you the potential of sub-xoas ^^'
almost out of alpha phase

hi,

We added pools/hosts on main XOA with XO PROXY httpUrl
we try to get all our clients onpremise pools&hosts in our "MSP-like" central XOA

host are seen not licensed

and so are the proxies

but on the remote sites, there is a licence & pro support in the remote XOA

I get it, on the central XOA is not present the licence of the others XOA onprem. so we get this error.

Would it be possible to "fetch" the licence information when attaching these remote pools/hosts ?

@john.c said in backup mail report says INTERRUPTED but it's not ?:

Are you using NodeJS 22 or 24 for your instance of XO?

here is the node version on our problematic XOA

this XOA do NOT manage backup jobs, totally offloaded to XO PROXIES

XOA PROXies :

[06:18 04] xoa@XOA-PROXY01:~$ node -v
v20.18.3

and XO CE :

root@fallback-XOA:~# node -v
v24.13.0

memory problems arise on our XOA

we have a spare XO CE deployed by ronivay script on ubuntu VM that we use only as a spare when main XOA is upgrading/rebooting
same pools/hosts attached, quite a read only XO

totally different behavior

@florent said in backup mail report says INTERRUPTED but it's not ?:

@Pilow We pushed a lot of memory fixes to master, would it be possible to test it ?

how so ? I stop my reboot everyday task and check if RAM is still crawling to 8Gb ?

@randyrue on our side, we do not use custom fields, but TAGs.
working well. depends on the number of custom fields you manage, and perhaps not usable because you need field:value ?

but nested tags could do the trick : field=value

@MajorP93 here are some screenshots of my XOA RAM

(lost before sunday stats since I crashed my host in RPU this weekend...)

you can clearly see RAM crawling and beeing dumped each reboot.

here is one of my XOA Proxies (4 in total, they totally offload backups from my main XOA)

there is also a slope of RAM crawling up... little spikes are overhead when backups are ongoing.

I started to reboot XOA+all 4 proxxies every morning.

@MajorP93 probably irrelevant, but since end of december I noticed a memory-leak behavior on my XOA.

I finally put up a job to restart it everyday 4.15am, otherwise at about 48h it was saturating it's RAM (8Gb...)

no more problem with a reboot everyday but, something is cooking.

@robertblissitt yup, afterward this seems to be a good best practice...
my hosts were up for 4 month, and because of DNS resolution problem had 77 patches to catch up (80 for one with advanced telemetry enabled)

a rolling reboot would have probably put in front the initial migration/evacuation problem (and subsequent zombies VMs)

and no patches applied, and no pool in a semi upgraded state

note to my future self, try a rolling reboot first.