Backup solutions for XCP-ng
-
Also it has more problems:
I need an agent to do inside-VM backups, as they are too big to snapshot (means it has to snapshot inside the VM, using VSS).
I don't see any mentioning of deduplicationz, just gzip compression, which would make our storage requirements explode. -
I already answered on dedup that is coming relatively soon. Inside VM backup can be achieve with our new hook we released last month. VSS is already done via the quiesce snapshot mechanism.
-
Also, if you want something more powerful, that would be nice to start contributing by getting XCP-ng Pro support, since all the money given to the product will help us to grow and achieve the level of feature you expect in the end
-
And to answer the question, no, nobody at Commvault ever knocked at our door. I think it's more an "opportunity shot" on their side since we have the same API than Citrix.
That might have been a good idea to do so, since we are open to have a bigger ecosystem
-
@olivierlambert Where did I say XOA isn't powerful at all? Quote me!
I see what you're doing, but I know what I'm doing and let me assure you: No chance at all.
We do have a big backup archive already and realisticly there's no option to migrate that to XOA for many years - depending on your development.
Of course I could go and use whatever to make our enviroment fitting to XOA, but it makes no sense at all. Big overheads.Don't get me wrong: We're talking about 2+ PB of (undeduplicated) data, native Exchange support is mandatory as it's no option to export a 1+ TB exchange DB before XOA is able to grab it.
Of course you might fit for 1.000s of environments, but I've checked with ours and you're far away from that.
-
Quoting you:
anyone is considering XCP-ng for his enterprise environment, but was missing a powerfull backup solution
This implies clearly there's no existing powerful backup solution for XCP-ng.
Also, I was talking about XCP-ng Pro support, not XOA.
-
@olivierlambert said in Backup solutions for XCP-ng:
Also, if you want something more powerful, that would be nice to start contributing by getting XCP-ng Pro support, since all the money given to the product will help us to grow and achieve the level of feature you expect in the end
If I won't consider the Pro support, I won't make this thread. Running that environment unsupported is not an option.
I'm just struggling with the price for the enterprise support, as the citrix one is pretty cheap and I have to justify the budget.
A bit of that would be: I would help developing long-due features to XCP-ng and OSS in general, of course.@olivierlambert said in Backup solutions for XCP-ng:
And to answer the question, no, nobody at Commvault ever knocked at our door. I think it's more an "opportunity shot" on their side since we have the same API than Citrix.
That might have been a good idea to do so, since we are open to have a bigger ecosystem
Can't say if my poke made them go for or something else triggered it, but as I just replaced one of the servers, I have something to test that out.
Oh and: XOA Premium is closes to Commvault, but multiple times as expensive (for our environment, others may differ) - I cannot ever justify that.
-
@olivierlambert said in Backup solutions for XCP-ng:
Quoting you:
anyone is considering XCP-ng for his enterprise environment, but was missing a powerfull backup solution
This implies clearly there's no existing powerful backup solution for XCP-ng.
Okay, you somehow got me there: I implied someone is using XCP-ng to host a bigger Windows environment, which usually includes MS SQL DBs, Exchange and, of course, an Active Directory. And for that, the application awareness is mandatory at some point.
-
But we might change to private conversion about the details, if you're interested.
Generally: This is the XCP-ng forum and should not be focussed on XOA sales, so IMHO it would be good for XCP-ng to give an overview with a feature matrix or something like that.There are enough people that just don't even give solutions like this a chance, when they can't (easily) see their options.
And by that you might miss XCP-ng pro customers. If someone pays for (enterprise) backup software, he most likely won't run the hypervisors unsupported. -
@cg said in Backup solutions for XCP-ng:
@olivierlambert said in Backup solutions for XCP-ng:
Quoting you:
anyone is considering XCP-ng for his enterprise environment, but was missing a powerfull backup solution
This implies clearly there's no existing powerful backup solution for XCP-ng.
Okay, you somehow got me there: I implied someone is using XCP-ng to host a bigger Windows environment, which usually includes MS SQL DBs, Exchange and, of course, an Active Directory. And for that, the application awareness is mandatory at some point.
We have customers in that position, and it's not mutually exclusive: they use also application aware solution (ie agent inside VMs) at the same time than XOA (to manage and backup their VMs).
-
@cg This forum is not focused on XOA sales, it's a community forum where people (at Vates but not exclusively) help others for free when they can on any XO version (source or paid). Same for XCP-ng. And my proof is all other topics this years here.
Our official doc is also listing alternative backup solutions. So no, don't try to imply this too, this is simply false
You can't just pop in there and tell there's no powerful solution to backup XCP-ng in general, which is dishonest. If it doesn't fit your use case it's something very different, and we are always happy to spend some time to learn about what feature is missing, that's how we built XOA and XCP-ng from the start.
-
@olivierlambert said in Backup solutions for XCP-ng:
@cg This forum is not focused on XOA sales, it's a community forum where people (at Vates but not exclusively) help others for free when they can on any XO version (source or paid). Same for XCP-ng. And my proof is all other topics this years here.
Our official doc is also listing alternative backup solutions. So no, don't try to imply this too, this is simply false
It's more a "partner page". It's missing a feature matrix, something like you did with VMware. Also there are more solutions like Unitrends, Alike and SEP - probably a few more, I don't know of. SEP also supports dedup and is application aware of (MS) SQL and Exchange - at least at the point I was evaluating a new solution. Though I had a massive performance issue and didn't get support to find the reason.
You can't just pop in there and tell there's no powerful solution to backup XCP-ng in general, which is dishonest. If it doesn't fit your use case it's something very different, and we are always happy to spend some time to learn about what feature is missing, that's how we built XOA and XCP-ng from the start.
I would say: Of course I can and I'm right, but I'm fine if we agree on: We have different levels of expectations about the features a solutions needs.
If something isn't very powerful, it doesn't mean it's bad or doesn't fit for anyone.
My usecase is a common one, but there are many you don't hear from, because they don't even seriously consider moving away from VMware, which is pretty much the standard in my type of environment.I'll move to PM as this pretty much didn't go to a "discuss backup solutons for XCP-ng", instead a very specific thread about my (and collegues) expectations and your POV.
-
It's fine to discuss your requirements here, this way others could interact and also upvote your needed features. That's how open source is working
If you want to discuss financing features specifically, this is also possible. To do that, you can contact us directly via a contact form on https://vates.fr
-
@olivierlambert
Okay, so bascially what I (and other people, I know, who are in a similar environment) need:- Tape backup is mandatory, as we're talking about (deduplicated, compressed) dozens of TB
- Deduplication and good compression: duplicated, uncompressed ends in PB
- All of that needs to scale! E.g. Commvault scales on 16 threads here even with only 1 - 2 tasks. (Can't say if even more without upgrade of CPU)
(We need chains, of course, to go B2D2T -> dumping the dedup-store on tape as disaster recovery) - Applicationawareness (which can be done via agent - agentless is not always king or very important):
-- MS Exchange, recovering datastores, mailboxes and even single mail items
-- SQL Servers: MS SQL, MySQL (MariaDB)...
-- Windows Active Directory Items
Over here I also need to have the option to do backups only inside the VM via agent, as I can't snapshot them.
You might have gotton me wrong: AFAIK quiesced backups always made a VM snapshot and submitted the request also to the VSS writer and then the VM/VHDs have been "read". In this case it will always fail, as there's no space on storage for such. Also Citrix discontinued that in their VM tools! XCP-ng yet lacks a proper tool maintenance with corresponding releases, which would be the only chance to keep that.As an idea for XOA:
- Make it run on physical hardware, as VMs are too small for such
- You already cooperate with HPE - let them run bare metal on ProLiants!
- ZFS(oL) Could be your way to go to implement dedup and compression
...feel free to add, comment... whatever.
-
Thanks for your suggestions. Most of them are already taken into account
-
@cg hi, I am working on the backups , and I can add some precisions
- compression is already on (first on S3, but now for all backups remotes) with the new vhd storage system, using brotli compression by default ( gzip possible , but it's slower and less efficient) . Compression level can be configured
- deduplicaton is in our backlog
- backups can be processed in parallel, and with the new vhd storage vhd blocks are uploaded in parallel ( default 8, can be configured)
- delta backup merging is done in parallel ( 2 by default, can be configured)
- we 're working on using network block device server to be able to read the data in parallel from the host, with the goal of speeding up backups and offloading DOM0
-
@florent great info. Is there some metrics on how long merging operations take? This would be helpful to avoid bottlenecks.
-
@Forza
We're currently creating a lab that will allow us to make benchmark more easily. The latest version of the merging operation only do a block renaming + deletion of unused block (blocks are 2MB , not configurable for now), no data copy, thus no data transfer between the remote and XO.
It is faster on performant backend, but we still have some fine tuning to do to ensure we don't overload slower backend. In the meantime we put some very conservatives settings.backup tiering and backup immutability are also in the work. I don't have a precise ETA, but it's in a few weeks to a few month at most.
theses features will pave the way for backup to tape
-
@florent IIRC OpenZFS 2 uses zstd and/or lz4 as efficient algorythms, which do a pretty good job. Yet I only know brotli from webservers.
How do you connect the tape, if it's virtualized?
Putting it on bare metal would also target that (aside of performance benefits and falling restrictions on backup size due to VHD limits). -
The problem of bare metal is to provide the appliance. As you can imagine, it's a very different business to distribute hardware appliances than a virtual one (stock management, spare parts, hardware support, shipping and so on).
