How fast are your backups

KPS

Hi!

I am trying to optimize my backup-speeds and I am interested in how fast you can to.

My backups and deltas are not faster than 250-280 MiB/s although my NFS-remote is much faster, everything is on 10 GbE, etc.

Before spending lots of time in trying to optimize further: Are you able to backup "faster"?

Thank you and best wishes
KPS

olivierlambert

not faster than 250-280 MiB/s

That sounds already more than a XAPI can handle. Are you doing backups on multiple pools at once?

KPS

@olivierlambert
Currently, I am only testing backups of one VM (did not parallelize, yet)

...but my hosts are quite beefy

olivierlambert

What matters (in terms of bottleneck) is the XAPI itself. If you backup from one single pool at this speed, I can only assume you are already doing NBD backups, right? (because VHD export can't go at this speed)

KPS

@olivierlambert
Yes, I am using NBD.
The big problem with NBD is, that there is currently no visibility about the backup-progress, as there is no "task". I know, that this is a big point for XO7, but the same is the case for "mirror-jobs".

olivierlambert

XO tasks are currently here to solve this. It's still early and not entirely displayed, but this will come soon

Also, 250-280MiB/s is already a great speed. What's your concurrency? Are you using NBD insecure or secure?

We might also have something else for you to test at some point, using a "plugin" inside the host.

KPS

@olivierlambert
"Concurrency" is my next step. I first wanted to find a good setup for "singe-stream-performance", but I am always happy to test new approaches!
Plugin sounds good. Other backup-providers are taking that way, too.

250MiB/s is good for small VMs, but not for big ones. My "non-XCP-setup" can do backups at 8-9 GiB/s

olivierlambert

This might be dangerous for your SR in real life scenario: what if a backup is taking all the SR resources? We have to find a balance But anyway, you can always restrict the backup speed in XO, so we'll continue to find ways to speed things up!

planedrop

I'm using SMB not NFS but the max I see is around 100MiB/s, so this is REALLY good speeds (I'm also on 10GbE across the board and on a NAS that can easily take in 2000MiB/s), very impressive speeds here.

Do you have any concurrency configured though? I believe the default is 2 so that may be why you're seeing close to 2x what I am (I have it set to 1, don't really need more than that so I like to decrease overall load).

Also on pretty beefy hosts.

rjt

@planedrop what concurrency configuration settings are you referring to?

XOA config?
host network card config?
SMB concurrency?

rjt

@olivierlambert Amazon must have patched their Xen / XAPI to have much higher speeds?

planedrop

@rjt The setting for backup concurrency within the XOA backup settings.

olivierlambert

@rjt AWS isn't using XAPI. Only XenServer and XCP-ng are using XAPI. It's mostly unrelated to Xen performance (at least not at this point).

danoontje

I'm backing up delta's from 5 pools to a lustre filesystem with a concurrency of 10.

olivierlambert

Not bad More than 2Gbit/s!

florent

@KPS Hi,
that is some nice performance, I guess the hardware under is beefy++

Did you try if NBD increase the speed ?
You need to enable NBD on the backup network, and then enable it in the delta backup :

We have a few tasks in the pipeline to increase performance, leveraging our knowledge of both XCP and XO

Florent