Too many snapshots

McHenry

Currently set to 15 on one schedule and 1 on the other schedule

Pilow

@McHenry could you screenshot the SNAPSHOTS tab of a problematic VM ?

the new backup & CR code changed the way it is handled as you understood... no more "one VM per replica point", but "one VM with a snapshot per replica point"

you should see 16 snapshots on the VM ?

EDIT : my bad, its the first screenshot you provided... indeed 16 snapshots

I do not remember the snapshots limit per VM (50?) , but you seem to bee far from it

Pilow

the limit enforced by XOA is 30

https://docs.xen-orchestra.com/backup_troubleshooting?utm_source=chatgpt.com#vdi-chain-protection

McHenry

@Pilow

Thank you. I did not know this change had been made.

So now I have the master VM with a snapshot for each backup schedule.

And a CR created VM with 16 snapshots.

I wonder why the health check shows too many snapshots as I am well under 30 per VM

Pilow

@McHenry could be another cap limit on the SR
can you screenshot homepage GENERAL tab of HST150 SR ?

how many VDIs present ?
what type of SR is it ?

jr-m4

Throwing my two cents into the hat, as well.

I had a similar situation. Snapshots would't be rotated out of retention, regardless of what I set on the retention setting (on a subset of VMs ie not all of the).. The thing that solved it for my, was to recreate the affected schedule.
Do note that this will mark the previous backups taken by that technically deleted schedule, as being abandoned/orphaned.

But after I did this, snapshots would be properly rotated as expected.

McHenry

@Pilow

SR is local storage

McHenry

@jr-m4

Recreated, testing now.

McHenry

@Pilow
Was there an announcement about the change in how these CR backups are done with snapshots now?

I'd love to read up on it.

Pilow

@McHenry screenshot the GENERAL tab of "Disaster Recovery" SR please
just to see how many VDIs it hosts... at least 288

it was announced here
https://xen-orchestra.com/blog/xen-orchestra-6-3/#-backup
but it is not explaing in details, I gathered information in another topix in this forum from @florent

you also have the Changelog of 6.3.0
https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md#630-2026-03-31

points to PR9524
[Replication] Reuse the same VM as an incremental replication target (PR #9524)

McHenry

@Pilow

I am rerunning the backups after recreating the schedule to see if the error clears

tjkreidl

@Pilow Yes; it's been my understanding that this has been the default for many years now.

Pilow

@tjkreidl yeah, but he has 16 snapshots.
but the documentation also talks about vdi chain length

but it seems to me impossible to have only 16 snaps and a vdi chain length >30

thats why I wondered perhaps is it a cap limit of snapshots per SR, but I didn't find relevant info about this possibility

you know a lot about Xen, ever heard of this type of per SR limit of snapshots ?
Only info I found is that more than 100/150 VDIs in production per SR can degrade performance

tjkreidl

@Pilow Yes, you are correct that the chain length is also limited. You might try to manually delete some of the snapshots and though the limit is supposed to be 30, perhaps there are other factors involved? Does that VM have a particularly large amount of storage and a lot of changes between snapshots? Are any other of your VMs experiencing similar issues? Your SR appears to be mostly empty, correct? Are there any related errors showing up in /var/log/SMlog ?

McHenry

@tjkreidl

I wish to maintain 16 restore points using CR, being an hourly restore point over the last two days (8 per day)
I perform a full backup nightly to reset the chain.

It appears that each CR creates a new snapshot and the old snapshot is removed when a new one is crated
The documentation states this error is shown then there are more than 3 snapshots on a VM
https://docs.xen-orchestra.com/manage_infrastructure#too-many-snapshots

Is this a problematic backup strategy?

tjkreidl

@McHenry Are you sure with that frequent running backups that each backup completes successfully before the next one starts? How long does the full backup typically take (less than 7 hours?) as well as the incrementals (under 1 hour?)? Again, I'd suggest looking in /var/log/SMlog for any error conditions that might help identify an issue. Also, how fragmented is your storage, as that can slow things down quite a bit, as can the lack of adequate CPU power as well as memory (run the top or xentop utility to view the load during backups).

Pilow

@tjkreidl @mchenry haaaa I remember how & when I was able to provoke this error
I was trying to purge 12 "replica VM" with the new CR method by forcing CR manually to get 1 VM with 12 replicas

so I ended up clicking START on the CR job as soon as the CR finished, and got this same error.
this was because GC didn't finish the previous job. Just had to wait 2 min for GC to reduce the chain length and I could go manual again on the CR

so I guess @tjkreidl is right, and the error message is misleading
your CR probably finish before the one hour interval BUT Garbage Collector do not

you have two options

space up your CR jobs to give GC some time to finish
find why GC is taking too much time (could be SR performance, nerver ending GC because of high I/O on the VM, ...)

Pilow

@florent @bastien-nollet could it be possible to monitor GC job to pause the job instead of failing with misleading error message ?

instead of TOO MANY SNAPSHOTS juste pause with WAITING PREVIOUS GARBAGE COLLECTOR TO FINISH and resume ASAP ?

this would force the admin of backup to re think his CR RPO/RTO strategy but not fail jobs