Want to compare insecure_nbd but doesn't seem to work (secure does)

sluflyer06

I created a fresh Debian 11.7 VM and installed XO from sources from the offical docs and made really no other change and re-ran NBD (secure) vs standard backup to NFS just to get another data point and see a significant performance advantage without NBD which I think is not supposed to be the case?

olivierlambert

Are you using a block based backup repository? (not single VHD files but 2MiB blocks). Maybe the bottleneck is on your NFS side, because on our various tests, the usual bottleneck is the XCP-ng export speed, not the backup repository.

sluflyer06

@olivierlambert I am using block based, I can make a test repo on a flash array and see if the spinners are choking on all the small files if that's what you're getting at. The NAS is a dedicated box but very modest by enterprise standard, it's a striped pool with 2 sets of 14TB WD Red's in ZFS (8 drives), typically it does quite well but every use case is different

Although I guess if the bottleneck was the storage, the NBD backup would be at worst, as fast as non?

olivierlambert

I would love to have a way to know where the back pressure is happening in the stream How many VM are you backuping and how many disks?

Also, maybe the small blocks are putting too much pressure on your disks, and therefore putting backpressure on the export speed.

@florent is there a way to raise the number of parallel downloads?

sluflyer06

@olivierlambert I'm willing to dig in and provide any data. right now it's 10 VM's all with 1 disk each, i'm currently running a backup without NBD using my flash array as a NFS destination, it's a handful of fast samsung SAS 12 SSD's that are striped (they're 3.2TB each). When that finishes I'll flip NBD back up and re-run so at least I think that will take storage out of the equation I believe.

sluflyer06

Total backup performance was still slower with NBD backing up to a flash pool so my disks do not appear to be the choke point.

olivierlambert

How many vCPUs and memory do you have in your XO VM?

sluflyer06

@olivierlambert 6 and 6gb ram

olivierlambert

In XO, are you in http in Settings/server URL when connected, or default? (nothing, meaning https)

sluflyer06

@olivierlambert http. I'd changed that long time ago for performance

olivierlambert

Okay so compare secure NBD vs HTTPS perf then I'm pretty sure NBD wins.

Alternatively, you might try to raise the number of parallel NBD chunks to be download at the same time (@florent knows the setting I suppose?)

sluflyer06

@olivierlambert okay yeah I can compare to HTTPS. totally game to try whatever setting he's got to increase the parallel work.

Insecure NBD would good comparison to http but I don't know how to turn that on, I know the purpose to set in host but there must be a config line to make it work in xo.

olivierlambert

It's only relevant to compare secure vs secure or insecure vs insecure. @florent will guide you when he's around

sluflyer06

@olivierlambert agree

florent

hi,

in theory, the insecure nbd should work when adding the insecure_nbd flag to the purpose of the network . Have you got any error message in the xo logs ?

you can add in the xapiOptions part

preferNbd = true
nbdOptions.readAhead = 20 # default 10

also do you remote store disks as multiple blocks ?

regards

sluflyer06

@florent Yes to multiple blocks. I did see this in host>networks page.

I do (and did) have the preferNbd set.

florent

@sluflyer06 we did a rework on NBD, to be able to use it everywhere, paving the way for some nice features like retry on network error

but it may have an impact on block based backup. I will make a branch which should alleviate this today

florent

@sluflyer06 can you test the speed with this branch based on master https://github.com/vatesfr/xen-orchestra/pull/6840

The backup job must have only one remote , which must be block based, and without a continuous replication

Warning : this is not production ready, this branch is only to find where the potential gains are.

fbeauchamp opened this pull request in vatesfr/xen-orchestra

draft feat(@xen-orchestra/backups): use parallel reading whith NBD + blocks to only one remote #6840

sluflyer06

@florent How would I verify for testing if its insecure NBD or not? I switched to your branch and rebuilt, I see the same message about not allowed but dunno if that matters.

florent

@sluflyer06 For now , I get back to the first problem : NBD should be faster than xapi export ( or at least as fast)

now I 'm working on enabling insecure

edit : the not allowed is only to inform the user that it can't be set through XO, but it should be used for backup if present