Backup from replicas possible?

flakpyro

Hi All

Continuing to experiment with XO / XCP-NG in a lab environment and looking into how backup and replication works. With our current backup provider (Veeam) it is possible to replicate VMs off site then backup from those VMs to longer term off site storage from the replicas, this allows you to say have 5 replicas ready to run but 30 days of backups based off those replicas stored on slower cheaper storage off site as well all while only having to send data to the off site location across the WAN one time. Im wondering if the same is possible with XO backup?

One thought i had would be to configure incremental replications (Continuous replication) and then create a smart backup job to backup all VMs with the tag of "Continuous Replication", it would be nice if we could specify a tag to be applied within the replication job itself though to allow you be more granular.

olivierlambert

Pinging @florent about this

florent

@flakpyro said in Backup from replicas possible?:

Hi All

Continuing to experiment with XO / XCP-NG in a lab environment and looking into how backup and replication works. With our current backup provider (Veeam) it is possible to replicate VMs off site then backup from those VMs to longer term off site storage from the replicas, this allows you to say have 5 replicas ready to run but 30 days of backups based off those replicas stored on slower cheaper storage off site as well all while only having to send data to the off site location across the WAN one time. Im wondering if the same is possible with XO backup?

One thought i had would be to configure incremental replications (Continuous replication) and then create a smart backup job to backup all VMs with the tag of "Continuous Replication", it would be nice if we could specify a tag to be applied within the replication job itself though to allow you be more granular.

this is a great idea, I add it to the list
As of today, replicated VM alread have the "continuous replication" tag (for incremental replication , and "disaster recovery" tag for full replication, but your solution will gives us more flexibility

flakpyro

@florent Happy to be able to help contribute to this project in some way! I think something like this will really help organizations with their DR/offsite backups as they potentially move from other products!

One other thing that would be useful in regards to this would perhaps be an advanced setting for a backup job to "remove deleted VMs from backup after X amount of days"

This would bring a two fold advantage: First allowing your smart backup job to continue to backup the replicated VMs (based on their tag) while removing old backups of old replicas points from the job that no longer exist in XOs pool inventory. Preventing the backup from growing indefinitely.

Secondly it would help keep backup jobs in check in environments where VMs are created and deleted by team members while backup jobs are only perhaps maintained by one team member. Without it you could end up storing a backup of a VM that was deleted months ago from XO until someone notices it in the backup file list.

florent

@flakpyro Yes, we took a look at the settings of restic/borg, veeam also has something simimar !keep n backup, then 1 per week for x week, 1 per month for x month ,...
Plus adding a "delete after xxx ", so a backup of VM that didn't have any backup this year can be deleted
and probably a "out of band backup" . Like this backup must kept until the end ot time and does not count in the workflow

It will probably be implemented this year, and it will probably make use of our new scoped tags

john.c

@florent Yes definitely sounds a very amazing idea. Though this may be a bit odd, but there's businesses that still use tape cartridge based backups.

How do you handle those in regards to backup of VMs via XO on XCP-ng?

flakpyro

@florent Thats great news! We use both Borg and Veeam so that's where most of these ideas are coming from. Having GFS backups like you mention for weekly, monthly, yearly would also be awesome to have and something we make use of in Veeam currently.

john.c

@flakpyro said in Backup from replicas possible?:

@florent Thats great news! We use both Borg and Veeam so that's where most of these ideas are coming from. Having GFS backups like you mention for weekly, monthly, yearly would also be awesome to have and something we make use of in Veeam currently.

I have volunteered at organisations where backup are being with this kind of cadence. Though some times the Friday backup would be queued up to run at midnight on Saturday morning, if there's activity planned or happening which would affect the backup proceeding on late night Friday.

florent

@john-c said in Backup from replicas possible?:

@florent Yes definitely sounds a very amazing idea. Though this may be a bit odd, but there's businesses that still use tape cartridge based backups.

How do you handle those in regards to backup of VMs via XO on XCP-ng?

For now we don't support tape backup. We know it is still used, especially since it gives a hardware guaranty of WORM, but regarding the huge work needed to make it work reliably, we plan to do it later. The challenges comes from the sequential nature of tape reading and writing that will mandate a huge code refactor, but also all the handling tools needed to locate the band containing a specific backup.
We're talking of the same scale as our import tool from vmware.

As always, priority can be adjusted, depending on market and user base needs Vs complexity.

john.c

@florent Would it be okay to open it as an issue on GitHub so the community developers can see it, that way can contribute by developing the functionality?

Though there's already libraries developed for developers to use which handle the grunt work of doing at least some of the work required to perform backups (accessing and working with the tape cartridges).

The only pieces of extra code then required to be developed are the NodeJS plugins which can use these libraries (or command line binary tools) and the Xen Orchestra tie ins.

https://packages.debian.org/bullseye/mt-st
https://packages.debian.org/bullseye/mtx
https://packages.debian.org/bullseye/stenc

florent

@john-c said in Backup from replicas possible?:

@florent Would it be okay to open it as an issue on GitHub so the community developers can see it, that way can contribute by developing the functionality?

Though there's already libraries developed for developers to use which handle the grunt work of doing at least some of the work required to perform backups (accessing and working with the tape cartridges).

The only pieces of extra code then required to be developed are the NodeJS plugins which can use these libraries (or command line binary tools) and the Xen Orchestra tie ins.

https://packages.debian.org/bullseye/mt-st
https://packages.debian.org/bullseye/mtx
https://packages.debian.org/bullseye/stenc

without these we wouldn't even be able to start working on it without building a dedicated team. There is still a lot of work to make it work. Especially the part we have to read and write sequentially.
Still, it's in our backlog, and not in the "one day maybe" part

john.c

@florent said in Backup from replicas possible?:

@john-c said in Backup from replicas possible?:

@florent Would it be okay to open it as an issue on GitHub so the community developers can see it, that way can contribute by developing the functionality?

Though there's already libraries developed for developers to use which handle the grunt work of doing at least some of the work required to perform backups (accessing and working with the tape cartridges).

The only pieces of extra code then required to be developed are the NodeJS plugins which can use these libraries (or command line binary tools) and the Xen Orchestra tie ins.

https://packages.debian.org/bullseye/mt-st
https://packages.debian.org/bullseye/mtx
https://packages.debian.org/bullseye/stenc

without these we wouldn't even be able to start working on it without building a dedicated team. There is still a lot of work to make it work. Especially the part we have to read and write sequentially.
Still, it's in our backlog, and not in the "one day maybe" part

@florent Additionally the tapes get listed like Linux block devices at the location of /dev. Though would require a filter to ensure that only tape drives are listed as option choices.

A good way to do it when you do would be as a plugin for Xen Orchestra backup functions. That way the process for performing the backup can occur in a sequential manner, within the plugin. Thus leaving the normal backup functions unaffected - still non-sequential (or abstracted).

At the moment is it possible to have plugins which extend Xen Orchestra's backup feature? If not may be a helpful addition as then plugins can be developed and used to provide functionality like this as well as other backup destination functionality, such as Microsoft Azure blob file storage functionality.

Also if necessary any client and/or customer specific plugins can then be developed, if their backup procedures are specialised in some way which the standard can't handle.

flakpyro

@florent Any news on this being added? I was not able to find anything about it on the Github. We are moving from XO from source to full XOA soon and our DR backup/replication plan will be one of the first things we need to get straightened out before we can totally move off of VMware and Veeam.

florent

@flakpyro said in Backup from replicas possible?:

@florent Any news on this being added? I was not able to find anything about it on the Github. We are moving from XO from source to full XOA soon and our DR backup/replication plan will be one of the first things we need to get straightened out before we can totally move off of VMware and Veeam.

Could you open a github issue to track this, or , if you have a XOA, open a a feature request ticket ?

In this thread I see 2 features :

adding a tag to a replicated VM, this should be quite easy
the lifecycle of the backup / replication , which is more work, but should be done this year

DustinB

replying to follow, FR created: 22844

flakpyro

Think this will make it into XO5 or is more likely to be an XO6 feature?

olivierlambert

It's kind of unrelated to XO 6 (especially point 2).

flakpyro

@olivierlambert Apologies, i thought adding the ability to add a tag to a replicated backup would be a feature added directly into XO as it handles backups in the Vates stack?

We are eagerly awaiting this functionality as it will allow us to do the same thing with XOA that we now do with Veeam. (Replicate to a DR site, then backup those replicas, keeping multiple restore points, onto long term ZFS backed backup repositories.)

Point 2 was not what i originally made this thread about, it came about afterwords from another forum user

olivierlambert

Adding a tag to a replicated backup should be trivial (we need to discuss the entire thing to be sure there's no obvious catch with @julien-f ).

DR automated fallback to the original site (re-sync) is planned for this year.

flakpyro

@olivierlambert we have our refreshed DR storage on order! (Replacing block only storage with another NFS capable array), as we close in on rebuilding our DR site i wonder if there has been any discussion on adding this feature? Being able to append a custom tag to a replication job would allow us to backup said replicas with the new GFS functionality to cheaper long term storage. This would in practice be pretty similar to what Veeam does with backing up from replicated storage snapshots but instead be totally hardware agnostic