Too many snapshots

Pilow

@tjkreidl said:

It's better IMO to have a solid backup less frequently than have them fail on a regular basis.

totally agree.

McHenry

@tjkreidl
The offsite backup runs at 8 pm and takes 6/7 hours, whereas the hourly runs from 7 am to 7 pm and only take a few minutes.

The backup job has 19 VMs, suely this is not too many.

Pilow

@McHenry 19 VMs is 19 Chains of 16 VDIs
at each hourly run, a new snapshot is created (some minutes) and the oldest one is merged/garbage collected in the first snap (time undetermined)

I guess 19 merge + chain garbage collected seems to not be able to be done in the one hour timeframe before next CR is done

you possibly have a chain growing

can you check in DASHBOARD/HEALTH the unhealthy VDI section at 11 am ?

tjkreidl

@Pilow Agree.... have to be sure that garbage collection is completed or it'll never catch up if backups continue to be run without the coalesce completing.

McHenry

@tjkreidl

If each CR backup is now created as a snapshot, instead of a new VM, and the alert triggers after a VM has more than three snapshots, this logically means that the alert will trigger if the CR has a retention value greater than 3.

Have I misunderstood how the CR backup process works?

Pilow

@McHenry I dont think more than 3 snapshots triggers an error, just tested on one VM

it is not recommended for "in production" VMs, but for a CR destination, it's OK (as you would need to start a copy anyway)

your problem, failing CR jobs is probably due to garbage collection not finishing in the one hour timeframe when chain is long.

McHenry

@Pilow

CR jobs are not failing just XO reports too many snapshots under:
Dashboard >> Health

All good if I can just ignore this warning but thought best to check in case it was an issue.

I got the value of 3 from here.
https://docs.xen-orchestra.com/manage_infrastructure#too-many-snapshots

Pilow

@McHenry could you screen the health page ?
where we could see the chain length

henri9813

Hello,

I see also this behavior which is "new" since few weeks.

Previously, when a backup start:

it stake a snapshot ( if there another one before, it delete it ).
it upload the snapshot as a backup
it coalesce the backup on the remote.
end of the game.

Now, the old snapshots are not deleted anymore which can lead easily to some disk full.

Even with a retention of 1, the problem is present.

I observe this only in Backup job, not DR/CR job.

I just updated my XO to latest version, i will see if the issue is fixed.

McHenry

@Pilow

The number of snapshots shows 16, which makes sense as I have two backup schedules, one with a retention of 15 and one with a retention of 1. The daily backup with a retention of 1 resets the chain, as it is a full backup.