@McHenry could be another cap limit on the SR
can you screenshot homepage GENERAL tab of HST150 SR ?

how many VDIs present ?
what type of SR is it ?

the limit enforced by XOA is 30

https://docs.xen-orchestra.com/backup_troubleshooting?utm_source=chatgpt.com#vdi-chain-protection

@McHenry could you screenshot the SNAPSHOTS tab of a problematic VM ?

the new backup & CR code changed the way it is handled as you understood... no more "one VM per replica point", but "one VM with a snapshot per replica point"

you should see 16 snapshots on the VM ?

EDIT : my bad, its the first screenshot you provided... indeed 16 snapshots

I do not remember the snapshots limit per VM (50?) , but you seem to bee far from it

@McHenry for TOO MANY SNAPSHOTS in this case, better look the RETENTION applied to this CR schedule

how many retention points do you have in the schedule of this job ?

another example, on another client server :

yesterday we were on XOA 6.0.3

today we updated to XOA 6.3.2

clean-vm get from 4min to an hour on this single VM ! for barely 5Gb transfer size
same VMs, same NFS remote on same NAS, nothing changed except XOA version

whole backup window on this 12 VMs job is shifted +4hours (still not finished...)

Hi,

Did you notice that since latest releases (I guess from 6.3.0 upward) that the merge process in the final phase of backups is stalling ?

57 seconds of transfer phase, followed by 8min of clean VM stuff (probably merging)

2min47sec of transfer / 15min of clean-vm

when you have lots of VMs, this adds up a lot to the backup window...

how could we make this better ?
@florent could you describe what is happening in the two clean-vm phases ? start and end ?

we tried many tricks to make this better (concurrency to 1 to avoid multiple mergings happening simultaneously, add cpu & RAM to the xoproxy VM, to the minio remotes too, to better handle the load)
but quite not there yet... still having quite long merging times in regard to the transfered size (i know this isn't last transfer that is merged it's the oldest, but seeing this behavior on quite similar incremental backup transfers)

I think this is new, due to latest releases and updates to backup code

@MajorP93 said:

I was also suprised to see that XOA still uses node 20

its presumably more stable memory-wise, as discussed in another topic about XOA memory leaks

on XO CE with node.js 24.x it installs OK

@olivierlambert because xo-disk-cli reviewed in last updates is not present
and the github tells us to npm install here : https://github.com/vatesfr/xen-orchestra/tree/master/%40xen-orchestra/disk-cli

hi,

we tried to install the npm package but it cries about node.js being too old... we are on official XOA latest commit, node.js is 20.x

will try on an XO CE and report back

@jerry1333 said:

There is nothing else on that host and this is only host in pool but it's using 30% of cpu all the time?

it's not using 30% of CPU, you see a graph of cumulated (switch is on) core consumption of your 32 cores.

never switch this on. it adds up like that : 32x1%=32%, wrongfully letting you think you are at 30%ish CPU usage.

haaaa on the pool stats, it spikes to 50% (but I have 2 hosts in this pool)

my guess ? false positive real monitoring do not add up to what we see in XOA stats.
I think we're down a rabbit hole

@olivierlambert told once that these stat graphs have not been updated for 10 years.

@jerry1333 I have an empty server in a pool, I do see some regulare bumps in the network, probably XOA polling RRDs for the stats from the master.

but no spikes in CPU as you have

followed by a

systemctl disable netdata

it should get rid of it, no easy way by XOA UI to disable this

@jerry1333 on your host

# systemctl status netdata
● netdata.service - Real time performance monitoring
   Loaded: loaded (/usr/lib/systemd/system/netdata.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2026-02-01 13:55:32 +04; 2 months 4 days ago
  Process: 2425 ExecStartPre=/usr/libexec/netdata/xcpng-iptables-restore.sh (code=exited, status=0/SUCCESS)
 Main PID: 2464 (netdata)
   Memory: 355.1M
   CGroup: /system.slice/netdata.service
           ├─   2464 /usr/sbin/netdata -D
           ├─   2466 spawn-plugins
           ├─   2724 /usr/libexec/netdata/plugins.d/network-viewer.plugin 1
           ├─   2739 /usr/libexec/netdata/plugins.d/apps.plugin 1
           ├─   2743 /usr/libexec/netdata/plugins.d/systemd-journal.plugin 1
           ├─   2745 spawn-setns
           ├─2300997 /usr/libexec/netdata/plugins.d/debugfs.plugin 1
           └─3392346 /usr/bin/bash /usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1

try that :

systemctl stop netdata

@jerry1333 you could try to stop & disable auto load plugin in SETTING/PLUGINS

and restart host ?

@jerry1333 nothing obvious.. except for NETDATA popping on your host

did you enable advanced telemetry ? could be it

@tsukraw trial and official licence are on the same email address ?

try to unconfigure email address, and re attach the email address ?

@jerry1333 could you do a TOP in XOA vm to identify what consumes CPU ?

Re: FILE RESTORE / overlapping loop device exists

we still had this annoying problem where we couldn't FLR our backup, source VM could be Windows or linux, with or without LVM... (see error logs in previous thread, overlapping loop...)

latest XOA is installed.

on a secondaray XO CE, I mounted the same remote, and NO PROBLEMS AT ALL... soooo I thought my main XOA is probably beaten up somewhere... but that was not it.

all remotes on main XOA are attached to a XO PROXY (official, not CE) !
on the XO CE that is in another datacenter, I mounted the S3 remotes directly WITHOUT the proxy use, and it worked.

On main XOA, we offload all backup tasks to proxies, so all remotes are attached to a proxy...
I duplicated a remote, same conf, but without proxy, so that XOA would do the restore job, not the proxy, and voila, all is working as intended, FLR is okay on main datacenter.

TLDR : FLR is not working with proxy attached remotes, if you duplicate the remote without the XO PROXY, all good

@florent @bastien-nollet I think there is something to look into.
probably a race condition, the XOA and the XO PROXY both try to mount the loop device ?