Backup cache getting out of sync when one of multiple remotes runs out of space
-
I write this, not to ask you to solve my problem but to suggest an area for exploration and possible improvement.
I have a couple NFS remotes. One has about 6TB of striped drive space. The other has 6TB of UNRAID space. Because of how UNRAID works, I set it up to keep folders together when they are two folders deep. The intent was to keep the individual VM's backup folder all on one drive so an array failure would leave me with at least some of the backups in a usable state. The backup task was set to keep 6 versions (because I was being dumb and didn't bother to do any math).
What I found is that the first few backups worked fine. One of the backups is about 500GB. When it tried to do the fourth backup, that folder exceeded the capacity of the 2TB drive UNRAID had the folder on so the backup failed. That totally makes sense.
What doesn't make sense is that the cached information about all the backups is now off. For instance, in the backup log it will tell me that a particular VM has 0 backups and it was expecting 3 but I can see that there are 6 or 7 backup files in the folder for that VM on the UNRAID server. Every VM that is backed up in that job now says it has too few actual backups compared to what it expects and when I look at the files, I see there are more backups that what the UI says there are. In fact, when I look at the Restore tab and click the Delete button, only one of the six VMs shows that it has any backups at all on that second remote. This is true for the VMs that aren't on the same UNRAID disk so those VM backups are still working.
One thing that might impact this. I already had the main remote set up and working for this backup, with 6 retained backups. I then added the UNRAID remote to that backup job. Very quickly that UNRAID disk filled up because all the VMs were being backed up to the same disk. I deleted all the backups on just that remote and tweaked the share settings in UNRAID to get it to spread the VM backup folders across the three disks. It's possible that that also caused problems, that the two remotes were never in sync. When adding a new remote, is there an expectation that the backups in existing remotes are deleted so both remotes are empty? Or, I suppose, the files from the existing remote could be copied into the new remote to get them in sync.
Note, this is a secondary backup and I'm still experimenting with how I'm going to do backups so it was ok with me that this caused problems, I'll just scrub the backups and try something different. I will likely try delta backups for that remote since that makes more sense with space restrictions like UNRAID has. Also note that I'm currently testing UNRAID to decide if I want to use it on the server that will host the backups.
Obviously working with remotes that have different free space or restrictions around free space usage (like UNRAID) should not both be targets for the same backup unless you make sure you won't exceed the storage of the more restricted remote, since backup retention is at the schedule level rather than the remote level.