Immutable S3 Backups (Backblaze) and Merging; A Little Confused

planedrop

Been doing some testing recently and hoping to get some feedback or a better understanding of how this works.

I am using Backblaze (S3 compatible) for some backup testing, I have the bucket setup with immutable storage (Object Lock as they call it) with a 30 day retention period for testing.

I created a backup job to test with this remote with the below settings:

2 VMs set to backup
Retention is 30
Full Backup Interval is 15

What I am confused about is, what happens when a merge needs to happen? If it's a locked object, you can't merge data into the full backup.

So once the 30 backup retention period is hit, how does the delta backup merge blocks into the full backup of the chain? That should be impossible if the object is write restricted.

However, I am not getting any errors when running this backup job beyond the 30 backup retention setting. Shouldn't it error out since it can't write to the full backup VHD?

Or there is maybe something I'm not understanding here?

I'm also wondering how this is managed long term, since the objects can't be deleted, XOA will (I presume) try to delete them, that will fail, and then they are just there in the bucket forever since XOA isn't aware of the immutable retention period to go cleanup later.

olivierlambert

Pinging @florent

planedrop

@olivierlambert @florent Just following up on this again, I am going to be doing some more testing this week to see if I can discover any issues. Still very curious about how this should be handled though.

florent

@planedrop
with your setup, you'll have 15 real days of immutability at worst.
here is a little schema with Key backup ( full) and delta . A chain of backup is a key backup and it's delta descendants
mutable backups are noted with a point

K.d.dKdddddKd // in this case the first chain is not protected, because parts of the chain are mutable, but the 2 most recent
K.KdddddKddd // 2 more backups here we have the longest immutables chains
K.dddddKdddd // here is the critical part with the shortest imutable chains : only the last one
K.ddddKdddd //the protetectd chain gros

to ensure a usable immutability of n day, you must have a full backup retention of at least n, and a backup retention of at least 2xn

I am not really sure on the error message that the UI show on backblaze, but the backup will be merged and deleted as soon as possible, when the object lock is lifted.
you can trust backblaze on their object lock, and you cna check the real number of backup stored by looking at the restore tab of XO

planedrop

@florent Finally getting back to this post, I know it's been months, just haven't had a lot of time, sorry!

I think I am still a bit confused, but I will do some additional testing to see if I can confirm my suspicions.

My confusion is that, you can't merge the deltas into the key if the key is locked behind Object Lock, the file isn't writable so you can't do the merge operation, right?

So that being said, it sounds to me like maybe the retention of the object lock/immutability needs to be set to be less than the retention period in XOA, right?

This way the original key is not immutable and can be written to when the merge happens?

Or does XOA just "wait" until the key isn't locked and then do the merge operation?