Disater recovery backups crazy slow
-
Hello Comdirect and Olivier,
I think I have about the same problem, what do you mean by CR and DR?
I have four hosts and two storage servers with Truenas core 13 10gig network for migration and backup.
I have 12 crucial 2tb ssd's and 6 8bt wd red pro hdd's .
Whether I use stripe vdev pools or mirror with or without cache, I always have slow performance.
But when I do a Crystaldiskmark test on a machine these look good
This is with a 1vdev 2x8tb mirror and ssd cache.
We would love to use it in production but until this problem gets resolved it is a no go.
I am probably missing the elephant in the room.kind regards,
Ludo -
@ludovic78
Try and set the Sync-setting to Disabled, on the target dataset that you're sharing. See if that does any difference.
This makes a GIANT difference for me.
-
@probain
I know this can give better performance, but I cannot use this in a production environment since in case of power outage or server failure there is loss of data or even corruption of vm's.
Would there be a guide available somewhere on how to get the most out of a setup?Thanks you for your reply!
-
DR is using full everytime and XAPI will generate the XVA (compression can even make it slower). CR will just expose the disks, and export is faster. So it's very likely the bottleneck you have on your setup is due to XVA generation from XAPI.
-
@olivierlambert said in Disater recovery backups crazy slow:
Thanks for the reply!
For my tests I use xen from source
Xen Orchestra e5608 and master a0c39
I also have slow disk migration it is so not only when I do backup.
Just did another migration test from a
4 x vdev mirror ssd to a 1 vdev 2x hdd with ssd cache averaged 30MB/s speed
Two days ago I had 150MB/s in the same test nothing was changed on the installationHow can I know if I am in DR or CR?
-
It is visible in your backup job, edit and check the type
-
@ludovic78
In the case of such a power outage, the backup would fail anyway.
And I'm also assuming that you have a separate share/dataset for those specific backups.
Coble it together with the Health Checks, and it is semi-production at least.
If it is for such a highly critical environment, that it wouldn't tolerate more than that. Then obviously you should open a support ticket -
@probain On the receiving remote (smb) the sync is off I also perform a health check on a stripe ssd dedicated to that where it is also off.
This is a snip of the machine the XOA is running on, what you see is the activity during the health check, I have no idea why this is behaving this way.
This also happens while taking the backup itself, this is not always the case, if it remains stable I have about 80MB/s speed.
Considering I have in production vm's that are 700GB in size this is way too slow.
-
With CR you will only replicate the new blocks, so after the initial "sync", future runs will be very fast.
-
@olivierlambert it first run so it's a full copy and 300 gigs took 1.5 hrs. With dr first run take 3-4. I do backups inside to iscsi very fast also. Just DR seems to take twice as long and I'm fully updated. It is from sources but fully updated. I even downloaded the official and tried trial and dr is still slow.
-
Please switch to CR to get rid of the XVA generation bottleneck first.
-
@olivierlambert Just to make sure we are talking about the same thing cr is continous replication and dr disaster recovery right?
-
Yes. Similar function, just done differently (XVA export vs VDI export).
-
@olivierlambert that's what I did and performance is fine. Soon as I choose dr (disaster recovery) it takes 2 5 times longer.
-
So don't use it. When you use DR, are you using compression or not?
-
It behaves differently and puts much less stress on the cpu
Unfortunately, the speed is still a problem.
-
I will put my xoa on another host maybe this will help
-
Your Dom0 might be not powerful enough to generate XVAs at a reasonable speed, so switch to CR and the issue is solved
-
I set the dom0 ram to 16gig on all hosts a while ago, the hosts (4) run on a dell r530 with two 2690 cpu's and 192gig ram.
What can I do to improve dom0?
-
Rather old CPUs (launched in 2012, so 12y old CPU), I'm not really surprised that XVA generation is slow. You can check if you are in Max Power mode in the BIOS though, maybe you are not and Xen can't boost to turbo.
But in any case, switch to CR.
