XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup from replicas possible?

    Scheduled Pinned Locked Moved Backup
    21 Posts 5 Posters 1.1k Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F Offline
      flakpyro
      last edited by

      Hi All

      Continuing to experiment with XO / XCP-NG in a lab environment and looking into how backup and replication works. With our current backup provider (Veeam) it is possible to replicate VMs off site then backup from those VMs to longer term off site storage from the replicas, this allows you to say have 5 replicas ready to run but 30 days of backups based off those replicas stored on slower cheaper storage off site as well all while only having to send data to the off site location across the WAN one time. Im wondering if the same is possible with XO backup?

      One thought i had would be to configure incremental replications (Continuous replication) and then create a smart backup job to backup all VMs with the tag of "Continuous Replication", it would be nice if we could specify a tag to be applied within the replication job itself though to allow you be more granular.

      florentF 1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Pinging @florent about this 🙂

        1 Reply Last reply Reply Quote 0
        • florentF Offline
          florent Vates 🪐 XO Team @flakpyro
          last edited by florent

          @flakpyro said in Backup from replicas possible?:

          Hi All

          Continuing to experiment with XO / XCP-NG in a lab environment and looking into how backup and replication works. With our current backup provider (Veeam) it is possible to replicate VMs off site then backup from those VMs to longer term off site storage from the replicas, this allows you to say have 5 replicas ready to run but 30 days of backups based off those replicas stored on slower cheaper storage off site as well all while only having to send data to the off site location across the WAN one time. Im wondering if the same is possible with XO backup?

          One thought i had would be to configure incremental replications (Continuous replication) and then create a smart backup job to backup all VMs with the tag of "Continuous Replication", it would be nice if we could specify a tag to be applied within the replication job itself though to allow you be more granular.

          this is a great idea, I add it to the list
          As of today, replicated VM alread have the "continuous replication" tag (for incremental replication , and "disaster recovery" tag for full replication, but your solution will gives us more flexibility

          F 2 Replies Last reply Reply Quote 0
          • F Offline
            flakpyro @florent
            last edited by

            @florent Happy to be able to help contribute to this project in some way! I think something like this will really help organizations with their DR/offsite backups as they potentially move from other products!

            One other thing that would be useful in regards to this would perhaps be an advanced setting for a backup job to "remove deleted VMs from backup after X amount of days"

            This would bring a two fold advantage: First allowing your smart backup job to continue to backup the replicated VMs (based on their tag) while removing old backups of old replicas points from the job that no longer exist in XOs pool inventory. Preventing the backup from growing indefinitely.

            Secondly it would help keep backup jobs in check in environments where VMs are created and deleted by team members while backup jobs are only perhaps maintained by one team member. Without it you could end up storing a backup of a VM that was deleted months ago from XO until someone notices it in the backup file list.

            florentF 1 Reply Last reply Reply Quote 0
            • florentF Offline
              florent Vates 🪐 XO Team @flakpyro
              last edited by florent

              @flakpyro Yes, we took a look at the settings of restic/borg, veeam also has something simimar !keep n backup, then 1 per week for x week, 1 per month for x month ,...
              Plus adding a "delete after xxx ", so a backup of VM that didn't have any backup this year can be deleted
              and probably a "out of band backup" . Like this backup must kept until the end ot time and does not count in the workflow

              It will probably be implemented this year, and it will probably make use of our new scoped tags

              J F 2 Replies Last reply Reply Quote 1
              • J Offline
                john.c @florent
                last edited by

                @florent Yes definitely sounds a very amazing idea. Though this may be a bit odd, but there's businesses that still use tape cartridge based backups.

                How do you handle those in regards to backup of VMs via XO on XCP-ng?

                florentF 1 Reply Last reply Reply Quote 0
                • F Offline
                  flakpyro @florent
                  last edited by

                  @florent Thats great news! We use both Borg and Veeam so that's where most of these ideas are coming from. Having GFS backups like you mention for weekly, monthly, yearly would also be awesome to have and something we make use of in Veeam currently.

                  J 1 Reply Last reply Reply Quote 1
                  • J Offline
                    john.c @flakpyro
                    last edited by

                    @flakpyro said in Backup from replicas possible?:

                    @florent Thats great news! We use both Borg and Veeam so that's where most of these ideas are coming from. Having GFS backups like you mention for weekly, monthly, yearly would also be awesome to have and something we make use of in Veeam currently.

                    I have volunteered at organisations where backup are being with this kind of cadence. Though some times the Friday backup would be queued up to run at midnight on Saturday morning, if there's activity planned or happening which would affect the backup proceeding on late night Friday.

                    1 Reply Last reply Reply Quote 0
                    • florentF Offline
                      florent Vates 🪐 XO Team @john.c
                      last edited by

                      @john-c said in Backup from replicas possible?:

                      @florent Yes definitely sounds a very amazing idea. Though this may be a bit odd, but there's businesses that still use tape cartridge based backups.

                      How do you handle those in regards to backup of VMs via XO on XCP-ng?

                      For now we don't support tape backup. We know it is still used, especially since it gives a hardware guaranty of WORM, but regarding the huge work needed to make it work reliably, we plan to do it later. The challenges comes from the sequential nature of tape reading and writing that will mandate a huge code refactor, but also all the handling tools needed to locate the band containing a specific backup.
                      We're talking of the same scale as our import tool from vmware.

                      As always, priority can be adjusted, depending on market and user base needs Vs complexity.

                      J 1 Reply Last reply Reply Quote 0
                      • J Offline
                        john.c @florent
                        last edited by john.c

                        @florent Would it be okay to open it as an issue on GitHub so the community developers can see it, that way can contribute by developing the functionality?

                        Though there's already libraries developed for developers to use which handle the grunt work of doing at least some of the work required to perform backups (accessing and working with the tape cartridges).

                        The only pieces of extra code then required to be developed are the NodeJS plugins which can use these libraries (or command line binary tools) and the Xen Orchestra tie ins.

                        https://packages.debian.org/bullseye/mt-st
                        https://packages.debian.org/bullseye/mtx
                        https://packages.debian.org/bullseye/stenc

                        florentF 1 Reply Last reply Reply Quote 0
                        • florentF Offline
                          florent Vates 🪐 XO Team @john.c
                          last edited by

                          @john-c said in Backup from replicas possible?:

                          @florent Would it be okay to open it as an issue on GitHub so the community developers can see it, that way can contribute by developing the functionality?

                          Though there's already libraries developed for developers to use which handle the grunt work of doing at least some of the work required to perform backups (accessing and working with the tape cartridges).

                          The only pieces of extra code then required to be developed are the NodeJS plugins which can use these libraries (or command line binary tools) and the Xen Orchestra tie ins.

                          https://packages.debian.org/bullseye/mt-st
                          https://packages.debian.org/bullseye/mtx
                          https://packages.debian.org/bullseye/stenc

                          without these we wouldn't even be able to start working on it without building a dedicated team. There is still a lot of work to make it work. Especially the part we have to read and write sequentially.
                          Still, it's in our backlog, and not in the "one day maybe" part

                          J 1 Reply Last reply Reply Quote 0
                          • J Offline
                            john.c @florent
                            last edited by john.c

                            @florent said in Backup from replicas possible?:

                            @john-c said in Backup from replicas possible?:

                            @florent Would it be okay to open it as an issue on GitHub so the community developers can see it, that way can contribute by developing the functionality?

                            Though there's already libraries developed for developers to use which handle the grunt work of doing at least some of the work required to perform backups (accessing and working with the tape cartridges).

                            The only pieces of extra code then required to be developed are the NodeJS plugins which can use these libraries (or command line binary tools) and the Xen Orchestra tie ins.

                            https://packages.debian.org/bullseye/mt-st
                            https://packages.debian.org/bullseye/mtx
                            https://packages.debian.org/bullseye/stenc

                            without these we wouldn't even be able to start working on it without building a dedicated team. There is still a lot of work to make it work. Especially the part we have to read and write sequentially.
                            Still, it's in our backlog, and not in the "one day maybe" part

                            @florent Additionally the tapes get listed like Linux block devices at the location of /dev. Though would require a filter to ensure that only tape drives are listed as option choices.

                            • https://www.cyberciti.biz/hardware/unix-linux-basic-tape-management-commands/
                            • https://www.cyberciti.biz/faq/linux-tape-backup-with-mt-and-tar-command-howto/
                            • https://blog.yucas.net/2018/09/12/15-useful-linux-and-unix-tape-managements-commands-for-sysadmins/
                            • https://access.redhat.com/solutions/68115
                            • https://linux.die.net/man/1/mt

                            A good way to do it when you do would be as a plugin for Xen Orchestra backup functions. That way the process for performing the backup can occur in a sequential manner, within the plugin. Thus leaving the normal backup functions unaffected - still non-sequential (or abstracted).

                            At the moment is it possible to have plugins which extend Xen Orchestra's backup feature? If not may be a helpful addition as then plugins can be developed and used to provide functionality like this as well as other backup destination functionality, such as Microsoft Azure blob file storage functionality.

                            Also if necessary any client and/or customer specific plugins can then be developed, if their backup procedures are specialised in some way which the standard can't handle.

                            1 Reply Last reply Reply Quote 0
                            • F Offline
                              flakpyro @florent
                              last edited by

                              @florent Any news on this being added? I was not able to find anything about it on the Github. We are moving from XO from source to full XOA soon and our DR backup/replication plan will be one of the first things we need to get straightened out before we can totally move off of VMware and Veeam.

                              florentF 1 Reply Last reply Reply Quote 0
                              • florentF Offline
                                florent Vates 🪐 XO Team @flakpyro
                                last edited by

                                @flakpyro said in Backup from replicas possible?:

                                @florent Any news on this being added? I was not able to find anything about it on the Github. We are moving from XO from source to full XOA soon and our DR backup/replication plan will be one of the first things we need to get straightened out before we can totally move off of VMware and Veeam.

                                Could you open a github issue to track this, or , if you have a XOA, open a a feature request ticket ?

                                In this thread I see 2 features :

                                • adding a tag to a replicated VM, this should be quite easy
                                • the lifecycle of the backup / replication , which is more work, but should be done this year
                                1 Reply Last reply Reply Quote 0
                                • D Offline
                                  DustinB
                                  last edited by DustinB

                                  replying to follow, FR created: 22844

                                  F 1 Reply Last reply Reply Quote 1
                                  • F Offline
                                    flakpyro @DustinB
                                    last edited by

                                    Think this will make it into XO5 or is more likely to be an XO6 feature?

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      It's kind of unrelated to XO 6 (especially point 2).

                                      F 1 Reply Last reply Reply Quote 0
                                      • F Offline
                                        flakpyro @olivierlambert
                                        last edited by

                                        @olivierlambert Apologies, i thought adding the ability to add a tag to a replicated backup would be a feature added directly into XO as it handles backups in the Vates stack?

                                        We are eagerly awaiting this functionality as it will allow us to do the same thing with XOA that we now do with Veeam. (Replicate to a DR site, then backup those replicas, keeping multiple restore points, onto long term ZFS backed backup repositories.)

                                        Point 2 was not what i originally made this thread about, it came about afterwords from another forum user 🙂

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO Offline
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          Adding a tag to a replicated backup should be trivial (we need to discuss the entire thing to be sure there's no obvious catch with @julien-f ).

                                          DR automated fallback to the original site (re-sync) is planned for this year.

                                          F 1 Reply Last reply Reply Quote 0
                                          • F Offline
                                            flakpyro @olivierlambert
                                            last edited by

                                            @olivierlambert we have our refreshed DR storage on order! (Replacing block only storage with another NFS capable array), as we close in on rebuilding our DR site i wonder if there has been any discussion on adding this feature? Being able to append a custom tag to a replication job would allow us to backup said replicas with the new GFS functionality to cheaper long term storage. This would in practice be pretty similar to what Veeam does with backing up from replicated storage snapshots but instead be totally hardware agnostic 🙂

                                            florentF 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post