Switching to XCP-NG, want to hear your problems
-
You sound a bit grumpy today @rfx77
-
@rfx77 there are several other options, alike a3 is one good agentless solution that runs xcp-ng cbt backups. XOA is allready having in beta and is focussing on getting 100%, we run our backups on cbt and it seems to work allrwsdy very good. Coalesce with cbt is almost instant.
I know Nakivo is also planning on adding xcp-ng to there solution kn the near future. -
Sorry for that but i am fighting to tune and get backup right for nearly a week now and it is hard to find a real 100% solution.
the migration from vmware gives us lots of headaces because of tight storage space and the snapshot problematic -
CBT will be helpful for you, as @rtjdamen said
-
@olivierlambert
Yes. Unfortunately i dont think that CommVault will add this feature if i request it. And from a bigger perspective where Xen is only on part of many there is no other option for us where we use it today.For smaller customers we are already evaluating XO with CBT for the VM backups + CommVault Agents in the VM (like others do with veeam)
But we are not entirely convinced that Xen will be our one-fits-all solution for our customers. maybe it will be a combination of hyperv and xenserver/xcp-ng
-
@rfx77 said in Switching to XCP-NG, want to hear your problems:
@olivierlambert
Yes. Unfortunately i dont think that CommVault will add this feature if i request it. And from a bigger perspective where Xen is only on part of many there is no other option for us where we use it today.For smaller customers we are already evaluating XO with CBT for the VM backups + CommVault Agents in the VM (like others do with veeam)
But we are not entirely convinced that Xen will be our one-fits-all solution for our customers. maybe it will be a combination of hyperv and xenserver/xcp-ng
If you don't try asking (or requesting) CBT support of XCP-ng being added from CommVault then you won't know for sure (let alone get). They have support for CBT on other hypervisors, so thus they do have CBT support generally. Thus they just need to add support for CBT on XCP-ng (via XO) in CommVault products. Xen Orchestra has recently on XOA latest channel added CBT.
-
@john-c
okay. you convinced me
i opened a support ticket with a feature request. I will keep you updated on this forum
-
@rfx77 said in Switching to XCP-NG, want to hear your problems:
@nikade No our current storage does not support nfs. but which enterprise grade-storage beside top end Netapp and Dell PowerStor really do??
We're using Dell Powerstore 500T and 1000T and they both support NFS, tbh everything is beond expectation. Dedup, performance and reliability is great.
Before that we were using Nexentastor which were ZFS based and they supported both iSCSI and NFS.
I guess it all comes down to your infrastructure and what you have at hand - Not everyone can afford a Powerstore. -
@nikade
We were also evaluating Dell PowerStore, ME5 (We are Dell and IBM Partners) and Infortrend but all NFS options are simply much to expensive for 90% of our cutomers. We openly discussed it with our major customers which could efford it but they opted for iSCSI and bought just more SSDs. So on new projects we do not really have that much of a problem if we are careful.On existing customer installations where we have FibreChannel and iSCSI it is different Story.
But i am glad to hear some feedback what others are using and that they are using xen in mission critical environments. often it is not clear what usecase people have and if they are talking about homelab, small non critical systems or something like Oracle VMs, ERP VMs, so serious work.
Thanks
-
That's the opposite. We have far more critical deployments than homelabbers. You can see us as an "opposite Proxmox" on this
-
@olivierlambert @john-c So i have feedback from commvault. There is an existing Request for Enhancement but no timeline or if it ever will be implemented. The CMR is 346970, for reference
To speed things up there are many more customers needed that reference to this CMR. Thats a chicken-egg problem and i dont think that thats the way to go.
But maybe vates could commmunicate with commvault directly and reference the CMR. For commvault it would be clearly heplful if they could have better numbers about how many customers use XCP-NG and what the potential is.
-
@olivierlambert thanks for this info. i also cannot not understand where all the love for proxmox comes from. and thats from someone who has his office in their neighborhood.
-
Have you tried CBT with XO before?
-
@rfx77 Proxmox exists from 2008 and built a big home labber community (which is great!). All those people are ambassadors and love using it for their lab. Having a big and vocal community doesn't mean it's deployed everywhere in the corporate world.
-
@olivierlambert i have running a XO installation which does backup a small part of our infrastructure, which works well. it is a single host with local thin-provisioned storage.
we also tried it on our main pool with iscsi storage and had mostly the same problems like others above.
we decided to wait a litte longer before we give it another try on our main production system mainly due to some hanging snapshots,...
in the future we are planning to use it as main vm-level backup solution for standalone hosts in combination with commvault agents in the vm.
our current test status i that we try to use XO as a docker container which would fit better in out existing backup infrastructure as we could easily integrate it in our linux backup appliances (see it like a linux NAS). The docker tests are running very well and we dont have any issues at the moment. even file-level-recovery is working well in this scenario.
-
So I advise to test CBT, this might solve your coalesce issues, since the snapshot will be removed a lot faster
-
@olivierlambert i do agree, from our experience cbt did solve this issues. Snapshots are removed in seconds, this was 10 to 15 minutes prior to using cbt. There is still some work to be done with cbt but it is usable at this time.
-
@rfx77 How has your experience been with Commvault and XCP-NG for backups? We are bumping into similar issues migrating away from VMware/Veeam. XO backups work but the snapshots that remain on larger VMs can grow quite large between nightly backup runs.
Trying CBT has been a mixed bag with snapshot coalesces taking anywhere from seconds up to 2-3 hours on larger VMs (with looping timeouts in SMlog) and backup failures if a VM has moved between hosts in the same pool.,and in the worst cases causing our NFS based storage (Pure storage) to expire all NFS leases to the pool master trying to coalesce causing NFS timeouts and storage outages. We have Pure looking into why this is happening for us now but until that's figured out we are back to CBT-less backups that leave a snapshots behind.
From what i have read Commvault uses their proxy VM to track changed blocks without leaving the snapshot in place? Sounds like it may be worth checking out.
-
@flakpyro are u using ssd storage? Of so i would recommend increasing the leaf coalesce parameters. This will prevent this loops from occuring. We had issues with this this prior to changing it. Now we do not see this anymore.
-
@rtjdamen The Pure storage array is all flash. I did see your comments about increasing those values, i did do that and while it did help it still can be hit or miss for us. For example our sharepoint server took 2-3 hours the other night to coalesce when using full CBT with snapshot delete enabled. Running it without snapshot delete the traditional snapshot coalesce only takes maybe 10-15 mins.
With that said we are also having issues with Pure's NFS implementation and how it interacts with XCP-NG, causing storage timeouts for us. According to them the array, when under load is disconnecting hosts due to "expired NFS leases" we are currently working with them to stabilize that, perhaps then i can revisit full CBT backups. Weirdly enough the disconnects do not appear to happen under regular VM operations even under high load, we mostly run into these during backup runs with CBT enabled runs increasing the chance of it happening.
What we ideally will end up with is local backups, offsite backups and archival backups. This is what we had with Veeam prior. Doing this with traditional XOA backups would result in 3 snapshots per VM (1 for each job) which lead me to looking into commvault since it does things a bit differently. CBT would also solve this as well i assume.
