Why does the backup use snapshots and not CBT
-
@olivierlambert, so it's different from VMware, where a snapshot is created for the backup job and then removed upon completion? In other words, it does not retain the snapshot for an entire day until the next job runs?
-
In CBT, a snapshot is created for the backup and then almost immediately removed. But still, the snapshot needed twice the space of the original disk to be made.
-
@olivierlambert, yes, but the snapshot doesn't need to remain till the next job is running. This distinction significantly impacts space requirements. You don't back up all your VMs simultaneously.
-
Yes, but the space reclaim won't be instant anyway (coalesce must happen first to save space). So the bonus isn't that great. It's better but not that a game changer.
-
@olivierlambert we are further in the process of using xcp, cbt would make a big difference, if u create a backup now on a large and heavy machine u will create a large snapshot during the day, coalesce this snapshot takes hours. If u create a snapshot at ghe start of a backup, copy only changed blocks and then delete the backup u only need to coalesce the small changes created during the backup. For large backups this is a game changer, we currently cannot move large vms to xcp as its backup is not suitable. When will this change? Is this planned?
-
@florent started some experiments with CBT and it didn't go very well. We'll re-discuss that internally.
-
@olivierlambert the current backup method is not suitable for larger vms, if i understand correct on xenserver cbt is allready the standard.
-
CBT is working on XCP-ng too. It's a question of implementation in Xen Orchestra.
-
@olivierlambert ok would be great if we can get this working, will be a huge improvement.
-
@rtjdamen said in Why does the backup use snapshots and not CBT:
@olivierlambert ok would be great if we can get this working, will be a huge improvement.
I concur, I've gotten my backup window down to around an hour, but still the amount of delta in each backup nightly is like 50GB, which doesn't seem feasible given the amount of delta is actually occurring.
-
Are there any 3th party backup solutions on xcp that support this?
-
Reading the doc is a good first step: https://docs.xcp-ng.org/project/ecosystem/
-
@olivierlambert indeed, what about xenserver compatible backups solutions, can they work on xcp as well? I see we could maybe use agent based backups inside the vms for the larger ones, i will test this with one vm, but need a proper solution in the future, i believe if we can get this to work in xoa that would be the best solution.
-
We are using CommVault to backup our XenServer infrastructure. it also does not use CBT but it does not keep the snapshots after the backup. they do some internal tracking. Works very well.
But clearly a CBT solution would be the best way. there is a github repo where you can see how it can be implemented with some simple python scripts.
For now keeping snapshots of all our VMs the whole day to do delta backups is not an option. we are using a iSCSI Storage and dont want to buy that extra SSDs.
Greetings, Franz
-
I suppose there's an agent to install in the Dom0 with CommVault for the tracking? I don't see another solution outside CBT to do it.
-
@rtjdamen said in Why does the backup use snapshots and not CBT:
@olivierlambert indeed, what about xenserver compatible backups solutions, can they work on xcp as well? I see we could maybe use agent based backups inside the vms for the larger ones, i will test this with one vm, but need a proper solution in the future, i believe if we can get this to work in xoa that would be the best solution.
You should look into something that is Application Aware (SQL etc) for these systems as they are likely sensitive to interruptions.
I don't know of any opensource agent based Application Aware backup solutions that would work, (edit) easily enough and not cause other headaches. Of course you would likely only be backing up your database with one of these separate tools and then use XO to restore the guest as a whole and restore whatever database there-after.
Off the top of my head a few backup solutions I've seen and used
- Bacula
- UrBackup
- Duplicati
- BackupPC
I know Bacula is/can be application aware in their paid version but I'm not sure if the opensource edition offers it.
-
@olivierlambert
No. A Proxy-VM (Windows) has to be deployed in the Pool. Nothing in Dom0.To my understanding, Commvault mounts the Snapshot-VDIs to the Proxy-VM and does the backup this way. They keep track of the block changes on the Proxy-VM.
In the simplest Setup the CommVault Server-VM itself can be the Proxy-VM. The only requirement is that it has access to all SRs where the VMs to backup reside on.
You can read the commvault docs on the internet. it is described very well.
-
@rfx77 can you provide a link please? Might worth a look
-
@olivierlambert i am currently looking at Quadric Software and storware, Quadric Software is doing the cbt backup exactly as needed, i got there description on the workflow yesterday
*When you run an ABD/hypervisor backup, the A3 will snapshot the VM for the duration of the backup data acquisition phase. After the datea is collected, the snapshot is immediately deleted.
In the case of CBT backups, the snapshot is deleted with the special XAPI command vdi-data-destroy, which frees data associated with the snapshot, but retains metadata, including the block allocation for the disk. This allows us to perform a delta backup the next time, with extremely negligible storage overhead.*
This is what we need
-
