Want to compare insecure_nbd but doesn't seem to work (secure does)
-
Are you using a block based backup repository? (not single VHD files but 2MiB blocks). Maybe the bottleneck is on your NFS side, because on our various tests, the usual bottleneck is the XCP-ng export speed, not the backup repository.
-
@olivierlambert I am using block based, I can make a test repo on a flash array and see if the spinners are choking on all the small files if that's what you're getting at. The NAS is a dedicated box but very modest by enterprise standard, it's a striped pool with 2 sets of 14TB WD Red's in ZFS (8 drives), typically it does quite well but every use case is different
Although I guess if the bottleneck was the storage, the NBD backup would be at worst, as fast as non?
-
I would love to have a way to know where the back pressure is happening in the stream
How many VM are you backuping and how many disks?Also, maybe the small blocks are putting too much pressure on your disks, and therefore putting backpressure on the export speed.
@florent is there a way to raise the number of parallel downloads?
-
@olivierlambert I'm willing to dig in and provide any data. right now it's 10 VM's all with 1 disk each, i'm currently running a backup without NBD using my flash array as a NFS destination, it's a handful of fast samsung SAS 12 SSD's that are striped (they're 3.2TB each). When that finishes I'll flip NBD back up and re-run so at least I think that will take storage out of the equation I believe.
-
Total backup performance was still slower with NBD backing up to a flash pool so my disks do not appear to be the choke point.
-
How many vCPUs and memory do you have in your XO VM?
-
@olivierlambert 6 and 6gb ram
-
In XO, are you in
httpin Settings/server URL when connected, or default? (nothing, meaning https) -
@olivierlambert http. I'd changed that long time ago for performance
-
Okay so compare secure NBD vs HTTPS perf then
I'm pretty sure NBD wins.Alternatively, you might try to raise the number of parallel NBD chunks to be download at the same time (@florent knows the setting I suppose?)
-
@olivierlambert okay yeah I can compare to HTTPS. totally game to try whatever setting he's got to increase the parallel work.
Insecure NBD would good comparison to http but I don't know how to turn that on, I know the purpose to set in host but there must be a config line to make it work in xo.
-
It's only relevant to compare secure vs secure or insecure vs insecure. @florent will guide you when he's around
-
@olivierlambert agree
-
hi,
in theory, the insecure nbd should work when adding the insecure_nbd flag to the purpose of the network . Have you got any error message in the xo logs ?
you can add in the xapiOptions part
preferNbd = true nbdOptions.readAhead = 20 # default 10also do you remote store disks as multiple blocks ?
regards
-
@florent Yes to multiple blocks. I did see this in host>networks page.
I do (and did) have the preferNbd set.
-
@sluflyer06 we did a rework on NBD, to be able to use it everywhere, paving the way for some nice features like retry on network error
but it may have an impact on block based backup. I will make a branch which should alleviate this today
-
@sluflyer06 can you test the speed with this branch based on master https://github.com/vatesfr/xen-orchestra/pull/6840
The backup job must have only one remote , which must be block based, and without a continuous replication
Warning : this is not production ready, this branch is only to find where the potential gains are.
-
@florent How would I verify for testing if its insecure NBD or not? I switched to your branch and rebuilt, I see the same message about not allowed but dunno if that matters.
-
@sluflyer06 For now , I get back to the first problem : NBD should be faster than xapi export ( or at least as fast)
now I 'm working on enabling insecure
edit : the not allowed is only to inform the user that it can't be set through XO, but it should be used for backup if present
-
@florent like Oliver was telling me, to compare secure to secure, it is faster than https,.still curious if anyway to try insecure.
Also. Is there a way to update XO (from sources) without losing the config.yaml or just a minor inconvenience until NBD is fully rolled into XO? (I assume we'll get told when we don't have to run custom config for NBD whenever it's ready?)
