Feedback on immutability

john.c

@olivierlambert said in Feedback on immutability:

I'm fine to target Synology first, so we need a dedicated resource to know how to build Syno app, because we have no idea.

If you later target TrueNAS bear in mind that TrueNAS Enterprise has immutable Snapshots support.

Also TrueNAS Scale is based on Debian Linux and with the most recent update it's using Docker containers for its apps functionality.

rtjdamen

@olivierlambert i will discuss with my 2 developers to see if this is something we can do internally or if we need to involve someone else. Who is doing this within vates? Can we discuss them directly?

olivierlambert

That's still @florent and you know he's pretty busy ATM However, you should have the immutability script available around to adapt it for Syno, it shouldn't be really hard and since it's fully self-contained, I don't see any risk to work on this Happy to review any contribution!

edit: source code is here: https://github.com/vatesfr/xen-orchestra/tree/master/%40xen-orchestra/immutable-backups

rtjdamen

@olivierlambert haha, let's not disturb @florent with this indeed, i will ask my dev to see what we can do with it and if we can adapt it to one of our syno boxes!

Sam Agnew

Also interested and willing to test on Synology support for this feature.

rtjdamen

@olivierlambert we are starting to develop the synology version this week. I will ask my developer to contact you or support if he has any questions.

rtjdamen

@olivierlambert we have managed to get this working on a synology box. However i have some question regarding the first setup.

the file is locked for x days (let's say 14 days), it will then work for a backup of 14 day retention, what to do if u have different retentions per job? there is no way for the repository to know this. what theorie do you have for this?
someone with root access does still have the option to change the files is this a good idea?
this immutability is also for meta files in the repo, is this not an issues? in other words the repository meta files can's be changed for 14 days as well.

Hope u can give me some answers so we can proceed working on this feature.

olivierlambert

Thanks for the feedback!

Those are questions for @florent

florent

@rtjdamen great work

the immutability duration is per repository, to limit the attack surface to the bare minimum
nothing can really be software protected against the root user. This is where physical device writable only once win
it should ignore the cache.json.gz , but the json file containing the backup metadata are protected along the disk data. Same for the pool metadata/xo config

An additional note : to ensure that an incremental backup is really protected during n days, you must have

a full backup interval smaller than n
a retention greater than 2n - 1
That way an attacker won't be able to modify the base disk used for restore

rtjdamen

@florent thanks for the feedback, so if i understand correct, currenlty there can only be one retention be used, if u use a retention of 14 days and 30 days mixed, u can only set immutablity for 13 days, otherwise you would run into issues with merge?

maybe this is something we can think a better solution for in future versions. we will implement your comments into the script and test it on our lab. If it's working we will share the code.

rtjdamen

@florent Goodmorning,

We have prepared the script here https://github.com/Virtual-Computing-bv/Xen-Orchestra-Immutability-SYNOLOGY

Would it be possible for you to do a check on this script to see if u have any comments?

rtjdamen

We are further testing with immutability and have encountered a question that came to mind. Suppose you run a 14-day backup, merging your oldest delta into the full backup every day. If you're doing forever incremental, the oldest file will keep being modified. This means that either this file becomes immutable for another 14 days after each merge, or it remains editable after 14 days, rendering immutability ineffective. How do you handle this in your setup?

florent

@rtjdamen for the immutability to be useful, the full chain must be immutable and must never be out of immutability

the merge process can't lift/ put back the immutability , and increasing synchronization between process will extend the attack surface.

immutability duration must be longer than or equal to 2 time the full backup interval -1
the retention must be strictly longer than the immutability .

for example, if you have a full backup interval of 7 a retention of 14 and immutability duration of 13 , key backup are K, delta are D. Immutable backup are in bold . unprotected chain are ~~striked~~

K~~DDDDDD~~KDDDDDD worst case, only one full chain protected
K~~DDDDD~~KDDDDDDK
K~~DDDD~~KDDDDDDKD
K~~DDD~~KDDDDDDKDD
KDDKDDDDDDKDDD
KDKDDDDDDKDDDD
KKDDDDDDKDDDDD best case almost 2 full chain protected

rtjdamen

@florent so this does mean it will never work when a forever incremental is used?

florent

@rtjdamen said in Feedback on immutability:

@florent so this does mean it will never work when a forever incremental is used?

you can't have a immutable forever backup without having a infinite length, and an infinite

It may be possible only if we release the constraints.
The immutable script could release the immutability , merge the disks, but that means : the immutability will be lifted from time to time, and the responsibilities of the immutability script will be greater, and we'll need a way to track the vhd to merge and transmit the information to the immutability script

afk

@florent said in Feedback on immutability:

@rtjdamen for the immutability to be useful, the full chain must be immutable and must never be out of immutability

the merge process can't lift/ put back the immutability , and increasing synchronization between process will extend the attack surface.

immutability duration must be longer than or equal to 2 time the full backup interval -1
the retention must be strictly longer than the immutability .

for example, if you have a full backup interval of 7 a retention of 14 and immutability duration of 13 , key backup are K, delta are D. Immutable backup are in bold . unprotected chain are ~~striked~~

K~~DDDDDD~~KDDDDDD worst case, only one full chain protected
K~~DDDDD~~KDDDDDDK
K~~DDDD~~KDDDDDDKD
K~~DDD~~KDDDDDDKDD
KDDKDDDDDDKDDD
KDKDDDDDDKDDDD
KKDDDDDDKDDDDD best case almost 2 full chain protected

I have not tried backups in XO yet but I'm really looking forward to test the immutability as we have it configured on all veeam backups at work.

Just to be sure, the XO immutability "agent" only does its immutability check by date right ?
Would it be possible to consider the entire backup chain related to the oldest immutable restore point instead ? This would prevent misconfigurations from the user that result in insecure backup chains.

florent

@afk the agent is as dumb as possible

also if you encrypt the backup, the agent will need to decrypt the metadata to detect the chains, thus having access to the encryption key, which need getting the encryption key out of XO and transferred to the immutability agent

I think it will be easier to provide more feedback on the immutabiltiy backup, XO has access to the chain , and / or alert when something seems to be strange

vkeven

Where is the doc for these ? edit ok right here https://github.com/vatesfr/xen-orchestra/tree/master/%40xen-orchestra/immutable-backups , with V**m there is an option for immutability when you add the S3 bucket directly so its look like its only a flag sent at bucket creation and using the versionning/compliance feature

florent

@vkeven we don't have ( for now) the feature to create bucket directly from XO. Also I think it is more secure if XO don't know at all the credits of the bucket admin

olivierlambert

Indeed, because if XO is compromised, then it could disable immutability.