S3 Immutable and Delta Backups (how does this work)
-
Update: After doing some more testing, and thanks to suggestions from @olivierlambert I was able to get this working. Basically, as long as the retention period is longer than the full backup period, in a Delta backup, then the deltas never have to be coalesced into the full backup (since a new full backup takes place before the retention runs out). I tested this with a test VM and it works as expected, with no errors about having issues writing or anything like that.
Hoping to get some info here, I'm using Wasabi as an S3 backup provider (I know this is in beta, however we also have local backups and a remote replication server, so the Wasabi ones are absolute worst case scenario backups, talking total data corruption and a natural disaster knocking out our other backups), so far everything with my full backups of some old VMs that will never have data change (they aren't even on) worked flawlessly, so did recovery, very impressed with the recover speeds too.
I was wondering though, how would this work on Immutable S3 storage with Delta backups? Delta backups require write access to the initial full backup correct? If that is the case, would XOA just create a new full backup each time, or would it fail since it can't write?
Hoping I can get a better understanding of this, I don't think doing nightly fulls to S3 is a good solution for this environment so I may just have a non immutable bucket with versioning enabled for the "ransomware protection" of sorts.
-
Hi @planedrop
First, thanks for your feedback. @nraynaud will be happy to know that S3 is also working correctly (we tend to focus on corner cases when it doesn't
)Regarding your question: you are right, we need to merge the oldest delta in the full, then when it's done, we remove that oldest delta.
It seems that VEEAM is doing immutable backups: https://www.veeam.com/blog/v11-immutable-backup-storage.html
But only for a time, I suppose after that, there's a merge happening to "consolidate".
My first guess of a way to achieve that in XO: when you use Full backup interval, it will create a new chain. So if you have a retention of 14, and full backup interval every 7, it means you'll end up with more than "1 big chain" of delta. So more time to avoid any coalesce (so Read Only on those files will work).
The best way to test:
- set a network share to be used as an XO remote
- play with retention and full backup interval
- put the full in Read only when you think no merge would happen
If you make a mistake, you'll see the behavior anyway
-
Happy to provide feedback, it's an excellent feature!!
This makes a ton of sense actually. Just to make sure I'm totally clear, if the retention is longer than the full backup interval, it shouldn't ever actually need to coalesce the delta into the full backup, since a new full will be created before hand. Correct?
I'll go ahead and test this and see how it goes, thank you!!
-
Yes, coalesce will only happens to respect the retention. But it's been a long time I've not took a look in the details, you should really test and report
In brief: if you use Full backup interval, fulls that aren't the oldest one won't be touched.
-
@olivierlambert OK sounds good, I will give this a shot.
On another unrelated note (I can start another thread if preferred), how does XOA handle simultaneous backups from different backup jobs?
What I mean is, imagine I have VM XYZ and set 2 backup jobs to separate SRs, one runs at 10pm and the next runs at 10:30pm, but the 10pm one hasn't finished yet, will the 10:30pm one still work or will that cause a VDI chain issue?
To be more clear, I noticed that backing up to Wasabi S3 and my local NAS at the same time with a single backup job, the NAS backup is slowed to the speed of Wasabi (upload about 16MB/s), so I want to create 2 backup jobs, one for the NAS and one for Wasabi so that the NAS backup finishes faster.
-
If you use 2 different remotes in the same job, it will be "multiplexed" and will push to both, at the speed of the slowest one (back pressure, there's no local cache in XO)
If you use 2 different jobs, it will rely on 2 different snapshots.
-
@olivierlambert OK perfect, thank you!
I've got a S3 Delta test going, will come back and post my results for others to reference once my testing is done.
