Why does the backup use snapshots and not CBT

rtjdamen

Hi all,

Just an update on our progress, i have been doing backups for around 10 larger vms now on CBT, i have seen this is working pretty well but it is not the wonderfull solution we had in mind.

First of all, the Alike backup software we now use to make CBT backups does have issues with multi disk vms, for some reason now and then the CBT got disabled by XEN on one of the disks resulting in a new baseline that needs to be created the next day, this is causing higher and longer backup times, still workable but it seems there are still issues with CBT in XCP-NG, i can determine if this is an issue of the alike software or that this is being caused by XCP-Ng itself. I have currently no other backup solution using the CBT so we can’t compare the results there.

Also a new issue can occur, when u create a CBT snapshot during peak hours it can cause issues with coalesce as well, the snapshot is deleted and it has to be coalesced into the base disk, but when the vm is under higher i/o load it can cause an endless process of coalesce. This is basically a limitation of the XEN garbarge collection process but it is giving issues with backups from now and then.

From what i understand XOA is working on CBT as well, i believe some fundamental changes are needed to get this all working in the same way as it currently is in vmware and other products. I think it would be much better and much more reliable if this kind of coalesces are done direct at the vm level and not limited to 1 task per SR (as i understand there currenlty is).

I am curious if anybody else allready is doing tests with this and does have some feedback that could help us on this problems.

rfx77

@rtjdamen
All our coalescing issues were solved by changing some constants in this file:

/opt/xensource/sm/cleanup.py

    LIVE_LEAF_COALESCE_MAX_SIZE = 1024 * 1024 * 1024  # bytes
    LIVE_LEAF_COALESCE_TIMEOUT = 300  # seconds

We did not experience any side-effects

rtjdamen

Hi all,

i have an update on this item, we are running CBT based backups for some larger vms now for a few weeks. We found that in our setup we had issues with NFS based storage, snapshots created there on multiple disk vms do give issues with the cbt snapshot, Xen is disabling CBT on the snapshot. we believe maybe because of latency in the tapdisk that is not ready yet for the specific vdi. We have migrated these vms to iscsi based storage and do not have issues there. That works like a charm!

@olivierlambert is it possible to add quadricsoftware to the backup solutions overview as well?

And in the end the best would be when CBT is enabled on XOA based backups as well, that will help improve the backups in general.

olivierlambert

We have not been contacted by Quadric yet. We are also working on testing CBT on our side, it's possible to be available in July's release.

rtjdamen

@olivierlambert in then end that will be the best, but i am inpressed with the experience of there team on Xen and XCP as well, i think they are a good addition to the allready complete list, to be honest so far there backup tool is doing better then the ones allready on the list.

olivierlambert

Thanks, it will be useful to test CBT so it might not be enabled by default first, but your feedback will be very useful when it will land!

rtjdamen

@olivierlambert we will help testing it, and as we are using it for some vms allready i can provide u with results and experiences from there as well. We will discuss that over a call later this week.

vkeven

Same issue there , we have around 125VMs and this is becoming a real problem , sometime a REP should used no more than 10GB and this is causing big transfer and rep delay, we just tried the xo-cr-seed and once we restart the backup ( 2 days from initial seed snapshot ) the transfer initiated a 12 hours backup delay so this is almost useless for remote DR site

rtjdamen

@vkeven vates is currenlty working hard on making the CBT backups available, we are currenlty using a product called Alike A3 for CBT enabled backups and they rely on the xapi for cbt as well. in general XCP-NG does allready support it and it works as it should. I hope we will see some progress here soon!

olivierlambert

We have already a branch started with CBT (we had first to store the backup metadata outside the snapshot itself because we'll remove it with CBT). This work is now done and now Florent is directly working on CBT