<![CDATA[Too many snapshots]]>I have 15 VMs included in a CR backup job.

Recently, I have seen notifications of "too many snapshots".

Whereas previously, each CR job would create a new VM, it appears now that each CR job creates a new snapshot.
dfa9ec67-fd32-4eee-a19c-e54ba10a8ec3-image.jpeg

The job itself appears to be configured correctly
f077259d-af28-4b85-9283-80dd227a313e-image.jpeg

I am sure I have a setting wrong, but I cannot find which one.

]]>https://xcp-ng.org/forum/topic/12067/too-many-snapshotsRSS for NodeMon, 08 Jun 2026 13:52:38 GMTWed, 08 Apr 2026 22:14:46 GMT60<![CDATA[Reply to Too many snapshots on Fri, 05 Jun 2026 14:18:43 GMT]]>

@henri9813 said:

Hello,

I see also this behavior which is "new" since few weeks.

Previously, when a backup start:

  • it stake a snapshot ( if there another one before, it delete it ).
  • it upload the snapshot as a backup
  • it coalesce the backup on the remote.
  • end of the game.

Now, the old snapshots are not deleted anymore which can lead easily to some disk full.

Even with a retention of 1, the problem is present.

I observe this only in Backup job, not DR/CR job.

I just updated my XO to latest version, i will see if the issue is fixed.

Hi @henri9813 ,
Issue should be resolved in 6.5.x, can you confirm on your side ?
Thanks!

]]>https://xcp-ng.org/forum/post/105995https://xcp-ng.org/forum/post/105995Fri, 05 Jun 2026 14:18:43 GMT<![CDATA[Reply to Too many snapshots on Wed, 20 May 2026 17:18:46 GMT]]>@julienXOvates PR is here https://github.com/vatesfr/xen-orchestra/pull/9868

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed fix(xo-web): ignore the number of snapshot of replicated VM in the health dashboard #9868

]]>https://xcp-ng.org/forum/post/105629https://xcp-ng.org/forum/post/105629Wed, 20 May 2026 17:18:46 GMT<![CDATA[Reply to Too many snapshots on Wed, 20 May 2026 15:37:06 GMT]]>@florent I think we discussed this and we thought it's not meaningful to have CR snapshots counted in the Dashboard (otherwise, we should maybe display standalone snapshots and snapshots from backups separately in 2 colmuns in XO6).

]]>https://xcp-ng.org/forum/post/105623https://xcp-ng.org/forum/post/105623Wed, 20 May 2026 15:37:06 GMT<![CDATA[Reply to Too many snapshots on Wed, 20 May 2026 09:38:22 GMT]]>@poddingue this is something that was hidden with the previous system ( same disk chains, but not shown as snapshot )

@julienxovates are you ok to not check the vm tagged as replication from this chech ?

]]>https://xcp-ng.org/forum/post/105598https://xcp-ng.org/forum/post/105598Wed, 20 May 2026 09:38:22 GMT<![CDATA[Reply to Too many snapshots on Wed, 20 May 2026 07:02:47 GMT]]>If the lower retention value gets things stable, that probably confirms Pilow's hypothesis. If it doesn't help, that's the signal that something heavier is going on, and a @Team-XO-Backend ping would make sense. Would you mind dropping the result back here either way? Helps the next person hitting the same wall.

]]>https://xcp-ng.org/forum/post/105591https://xcp-ng.org/forum/post/105591Wed, 20 May 2026 07:02:47 GMT<![CDATA[Reply to Too many snapshots on Sat, 18 Apr 2026 00:28:22 GMT]]>@Pilow

I did check this and it definitely completes within the hour.

I am testing a lesser value for CR retention to see if this resolves it.

]]>https://xcp-ng.org/forum/post/104576https://xcp-ng.org/forum/post/104576Sat, 18 Apr 2026 00:28:22 GMT<![CDATA[Reply to Too many snapshots on Fri, 17 Apr 2026 04:25:31 GMT]]>

@McHenry said:

Thanks.
The old snapshots are being removed as the total never increases beyond 16, so when a new snapshot is added, the old one is removed.

immediatly removed, yes, but then Garbage collection takes place.
and perhaps with 19x16 GC to process it can't be done in one hour, and then next CR is launched, etc etc...

]]>https://xcp-ng.org/forum/post/104529https://xcp-ng.org/forum/post/104529Fri, 17 Apr 2026 04:25:31 GMT<![CDATA[Reply to Too many snapshots on Thu, 16 Apr 2026 21:46:40 GMT]]>@henri9813

Thanks.
The old snapshots are being removed as the total never increases beyond 16, so when a new snapshot is added, the old one is removed.

]]>https://xcp-ng.org/forum/post/104523https://xcp-ng.org/forum/post/104523Thu, 16 Apr 2026 21:46:40 GMT<![CDATA[Reply to Too many snapshots on Thu, 16 Apr 2026 08:46:51 GMT]]>Hello @McHenry .

Yes but no, once the snapshot is exported, the previous one must be cleaned on local.

Best regards,

]]>https://xcp-ng.org/forum/post/104498https://xcp-ng.org/forum/post/104498Thu, 16 Apr 2026 08:46:51 GMT<![CDATA[Reply to Too many snapshots on Wed, 15 Apr 2026 21:40:04 GMT]]>@Pilow
eac3d82b-82a4-46b3-b8e1-1f4b64c57e35-image.jpeg

The number of snapshots shows 16, which makes sense as I have two backup schedules, one with a retention of 15 and one with a retention of 1. The daily backup with a retention of 1 resets the chain, as it is a full backup.
883856d8-222a-4593-a013-3204a340ecbc-image.jpeg

]]>https://xcp-ng.org/forum/post/104486https://xcp-ng.org/forum/post/104486Wed, 15 Apr 2026 21:40:04 GMT<![CDATA[Reply to Too many snapshots on Wed, 15 Apr 2026 15:57:46 GMT]]>Hello,

I see also this behavior which is "new" since few weeks.

Previously, when a backup start:

  • it stake a snapshot ( if there another one before, it delete it ).
  • it upload the snapshot as a backup
  • it coalesce the backup on the remote.
  • end of the game.

Now, the old snapshots are not deleted anymore which can lead easily to some disk full.

Even with a retention of 1, the problem is present.

I observe this only in Backup job, not DR/CR job.

I just updated my XO to latest version, i will see if the issue is fixed.

]]>https://xcp-ng.org/forum/post/104476https://xcp-ng.org/forum/post/104476Wed, 15 Apr 2026 15:57:46 GMT<![CDATA[Reply to Too many snapshots on Wed, 15 Apr 2026 12:48:45 GMT]]>@McHenry could you screen the health page ?
where we could see the chain length

]]>https://xcp-ng.org/forum/post/104468https://xcp-ng.org/forum/post/104468Wed, 15 Apr 2026 12:48:45 GMT<![CDATA[Reply to Too many snapshots on Wed, 15 Apr 2026 11:56:40 GMT]]>@Pilow

CR jobs are not failing just XO reports too many snapshots under:
Dashboard >> Health

All good if I can just ignore this warning but thought best to check in case it was an issue.

I got the value of 3 from here.
https://docs.xen-orchestra.com/manage_infrastructure#too-many-snapshots

]]>https://xcp-ng.org/forum/post/104467https://xcp-ng.org/forum/post/104467Wed, 15 Apr 2026 11:56:40 GMT<![CDATA[Reply to Too many snapshots on Wed, 15 Apr 2026 08:42:08 GMT]]>@McHenry I dont think more than 3 snapshots triggers an error, just tested on one VM a150619d-1010-4efc-80ec-9dc4bcc730b0-image.jpeg

it is not recommended for "in production" VMs, but for a CR destination, it's OK (as you would need to start a copy anyway)

your problem, failing CR jobs is probably due to garbage collection not finishing in the one hour timeframe when chain is long.

]]>https://xcp-ng.org/forum/post/104459https://xcp-ng.org/forum/post/104459Wed, 15 Apr 2026 08:42:08 GMT<![CDATA[Reply to Too many snapshots on Wed, 15 Apr 2026 01:53:59 GMT]]>@tjkreidl

If each CR backup is now created as a snapshot, instead of a new VM, and the alert triggers after a VM has more than three snapshots, this logically means that the alert will trigger if the CR has a retention value greater than 3.

Have I misunderstood how the CR backup process works?

]]>https://xcp-ng.org/forum/post/104456https://xcp-ng.org/forum/post/104456Wed, 15 Apr 2026 01:53:59 GMT<![CDATA[Reply to Too many snapshots on Tue, 14 Apr 2026 03:24:46 GMT]]>@Pilow Agree.... have to be sure that garbage collection is completed or it'll never catch up if backups continue to be run without the coalesce completing.

]]>https://xcp-ng.org/forum/post/104440https://xcp-ng.org/forum/post/104440Tue, 14 Apr 2026 03:24:46 GMT<![CDATA[Reply to Too many snapshots on Mon, 13 Apr 2026 05:45:51 GMT]]>@McHenry 19 VMs is 19 Chains of 16 VDIs
at each hourly run, a new snapshot is created (some minutes) and the oldest one is merged/garbage collected in the first snap (time undetermined)

I guess 19 merge + chain garbage collected seems to not be able to be done in the one hour timeframe before next CR is done

you possibly have a chain growing

can you check in DASHBOARD/HEALTH the unhealthy VDI section at 11 am ?

]]>https://xcp-ng.org/forum/post/104385https://xcp-ng.org/forum/post/104385Mon, 13 Apr 2026 05:45:51 GMT<![CDATA[Reply to Too many snapshots on Sun, 12 Apr 2026 22:19:33 GMT]]>@tjkreidl
The offsite backup runs at 8 pm and takes 6/7 hours, whereas the hourly runs from 7 am to 7 pm and only take a few minutes.
76eca590-f9f4-4f99-8964-23be858c62c0-image.jpeg

The backup job has 19 VMs, suely this is not too many.

]]>https://xcp-ng.org/forum/post/104381https://xcp-ng.org/forum/post/104381Sun, 12 Apr 2026 22:19:33 GMT<![CDATA[Reply to Too many snapshots on Sun, 12 Apr 2026 17:27:03 GMT]]>

@tjkreidl said:

It's better IMO to have a solid backup less frequently than have them fail on a regular basis.

totally agree.

]]>https://xcp-ng.org/forum/post/104379https://xcp-ng.org/forum/post/104379Sun, 12 Apr 2026 17:27:03 GMT<![CDATA[Reply to Too many snapshots on Sun, 12 Apr 2026 17:10:36 GMT]]>@Pilow The other thing to to consider is being cognizant of how long your backups typically take (or even, planning a worst-case condition) and defining the backup intervals accordingly.
In other words, if you know you cannot consistently do your incremental backups in less than an hour, perform them 90 minutes or two hours between backups. It's better IMO to have a solid backup less frequently than have them fail on a regular basis.

]]>https://xcp-ng.org/forum/post/104378https://xcp-ng.org/forum/post/104378Sun, 12 Apr 2026 17:10:36 GMT<![CDATA[Reply to Too many snapshots on Sun, 12 Apr 2026 15:41:00 GMT]]>@tjkreidl either skip or wait until possible
I'm used to veeam backup & recovery that is very resilient to these corner cases, on vmware if it understands that a Datastore has too many snapshots, or some backup ressouce is not ready yet (you can throttle number of active workers on a repository or per proxy), veeam will just wait for availability and keep going.

problem with this way of doing is it can shift in time the schedule where you expect CR or backup to be happening.

but can be a problem to skip altogether, if @mchenry need compliancy of a certain number of replicas happening

waiting vs skipping, in a perfect world the devs give us a switch to choose our destiny 😃

ps : I know XO Backup is not to be 100% mapped on Veeam functionnalities, but some of these functionnalities would really augment the XO Backup experience. just have to take into account Xen environment (no GC in vmware infrastructure)

]]>https://xcp-ng.org/forum/post/104376https://xcp-ng.org/forum/post/104376Sun, 12 Apr 2026 15:41:00 GMT<![CDATA[Reply to Too many snapshots on Sun, 12 Apr 2026 15:24:52 GMT]]>@Pilow Right, just skip the currently planned backup if a coalesce is still in progress and check again the next scheduled backup. This could potentially be implemented in the existing backup code.

]]>https://xcp-ng.org/forum/post/104375https://xcp-ng.org/forum/post/104375Sun, 12 Apr 2026 15:24:52 GMT<![CDATA[Reply to Too many snapshots on Sun, 12 Apr 2026 15:19:55 GMT]]>@tjkreidl yes would be a good way to deal with the original problem

hope backup Devs @florent and/or @bastien-nollet can implement this, would profit to everyone

]]>https://xcp-ng.org/forum/post/104374https://xcp-ng.org/forum/post/104374Sun, 12 Apr 2026 15:19:55 GMT<![CDATA[Reply to Too many snapshots on Sun, 12 Apr 2026 13:27:23 GMT]]>@Pilow Ah, right. You'd have to check the time stamp if you worked on automating this.
So maybe @McHenry could write a script to do the backups and that way, ensure there was no on-going task in progress before kicking off the next backup instance.
It could be run periodically from a cron job and if there's still on-going activity, just exit and try again the next time.

]]>https://xcp-ng.org/forum/post/104373https://xcp-ng.org/forum/post/104373Sun, 12 Apr 2026 13:27:23 GMT