XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup solutions for XCP-ng

    Scheduled Pinned Locked Moved News
    34 Posts 5 Posters 14.4k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      cg @florent
      last edited by

      @florent IIRC OpenZFS 2 uses zstd and/or lz4 as efficient algorythms, which do a pretty good job. Yet I only know brotli from webservers.

      How do you connect the tape, if it's virtualized?
      Putting it on bare metal would also target that (aside of performance benefits and falling restrictions on backup size due to VHD limits).

      florentF 1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        The problem of bare metal is to provide the appliance. As you can imagine, it's a very different business to distribute hardware appliances than a virtual one (stock management, spare parts, hardware support, shipping and so on).

        C 1 Reply Last reply Reply Quote 0
        • C Offline
          cg @olivierlambert
          last edited by

          @olivierlambert Sure it is a different thing, that's why I recommended using your connections to HPE to offer a bundle or at least to offer a version, that runs with a (more or less specific) version of one of their servers. As it only makes sense when the environment reaches a certain point, it would make sense to pick a DL380/385 series/generation, which offer a good bandwith of performance and space.
          E.g. we use a DL385 with 10x 10 TB HDD + a few SSDs for cache and database.

          IMHO it's okay to say: We support bare metal on platform X. Lots of configurations options don't matter for your support, as more memory, bigger CPUs or more storage behind the same controller don't touch the needed drivers/evaluations.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            It's more complicated than that. We need then to have a way to install the exact environment we need to have decent QA on it. So it's more like building an installer for it (which is not immensely complex but MORE work, since the installer should be written but also maintained).

            I'm pretty convinced about the perf level of using it on a physical machine, it's just that everything around is more complex to deliver.

            1 Reply Last reply Reply Quote 1
            • M Offline
              mjtbrady @florent
              last edited by

              @florent How can the level of compression and parallelisms be set?

              Thanks

              1 Reply Last reply Reply Quote 0
              • florentF Offline
                florent Vates 🪐 XO Team @cg
                last edited by olivierlambert

                @cg we are envisaging various way, from using iSCSI to access tape from the VM, to using an agent on the tape (but here we'll have to support physical hardware patching , updating ) . There is also a lot of work to ensure we write sequentially without concurrency and to make it work with the futur dedup and to keep a catalog of backups / tapes

                @mjtbrady my bad it is set as zlib.constants.BROTLI_MIN_QUALITY for now. Since I called it , I can add the param if you want to help me test it . Is it ok for you ?

                upload concurrency is in writeBlockConcurrency
                merge concurrency is maxMergedDeltasPerRun
                Compression type can be set in the config.toml, vhdDirectoryCompression

                C M 2 Replies Last reply Reply Quote 0
                • C Offline
                  cg @florent
                  last edited by

                  @florent said in Backup solutions for XCP-ng:

                  @cg we are envisaging various way, from using iSCSI to access tape from the VM, to using an agent on the tape (but here we'll have to support physical hardware patching , updating ) . There is also a lot of work to ensure we write sequentially without concurrency and to make it work with the futur dedup and to keep a catalog of backups / tapes

                  I don't know every product, but yet I've never seen a Tapedrive/Library using iSCSI.
                  iSCSI is usually only used by storage systems, not by devices or libraries.
                  Common interfaces are either SAS or - especially in larger environments - Fiberchannel. So your way to go, probably, is to passthrough an HBA.

                  ForzaF florentF 2 Replies Last reply Reply Quote 0
                  • ForzaF Offline
                    Forza @cg
                    last edited by Forza

                    @cg said in Backup solutions for XCP-ng:

                    I don't know every product, but yet I've never seen a Tapedrive/Library using iSCSI.

                    iSCSI, at least with Linux's LIO subsystem, can pass through SCSI devices (PSCSI, not the block device, though I have seen some warnings about that. I guess the initiator has to understand the target's device model specific commands properly.

                    I actually haven't tested this myself, so I don't know how well it works.

                    http://www.linux-iscsi.org/wiki/LIO#Backstores

                    Edit: found this article of doing something like that https://www.kraftkennedy.com/virtualizing-scsi-tape-drives-with-an-iscsi-bridge/

                    1 Reply Last reply Reply Quote 0
                    • florentF Offline
                      florent Vates 🪐 XO Team @cg
                      last edited by

                      @cg We're still in the early phase, but it seems possible to connect the tape loader to a host, and to let the host expose it. Passthrough will means that Xo's host is directly connected, which is not always the case

                      Backup to tape is quite different than backuping on disk, S3 or even glacier, I am confident that it will be possible, and we will communicate as soon as we have a working copy.

                      1 Reply Last reply Reply Quote 1
                      • M Offline
                        mjtbrady @florent
                        last edited by

                        @florent Happy to do what I can to test this.

                        In my xo-server config.toml I see writeBlockConcurrency, maxMergedDeltasPerRun and vhdDirectoryCompression in a [backups] section.

                        Does this mean that these setting are system wide?

                        Shouldn't these be configurable on a per Remote basis?

                        Also in a previous post you have indicated that writeBlockConcurrency defaults to 8, but my xo-server config.toml it is 16.

                        florentF 1 Reply Last reply Reply Quote 0
                        • florentF Offline
                          florent Vates 🪐 XO Team @mjtbrady
                          last edited by

                          @mjtbrady For now these settings are globals. Yes it's 16 by default , I was misled by my non default test installation

                          C 1 Reply Last reply Reply Quote 0
                          • C Offline
                            cg @florent
                            last edited by cg

                            I've set up a test-environment and indeed it worked. Commvault was (is?) checking for specific version/agent string when connecting to XS/CHV/XCP-ng and in older versions refused to connect to XCP-ng.

                            I successfully installed the agent on a proxy-vm (it uses a proxy to ro-mount the VM-VHDs and backup the content), connected the pool, delivered the VM-inventory and also ran a successful backup:
                            171c0956-e3e2-4fb9-a93b-48a2be6751bb-grafik.png

                            In other words: If you're using Commvault (11.28+) you're not any longer locked on Citrix.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post