Remove backup bottleneck: please report


  • Admin

    For people who want to test a potential XO backup speed bottleneck (in fact, XCP-ng bottlneck):

    1. in XO, disconnect your pool, edit the master URL by adding "http://" before the IP/hostname
    2. try again a backup

    Note: it won't be encrypted anymore, but could be interesting to see that stunnel is a potential bottleneck on the dom0

    Someone reported that it doubled the backup speed (in delta backup). It's due to how HTTPS is created in XCP-ng/XenServer (XAPI HTTP lib is very old and crappy, without HTTPS support, so they added stunnel on top, which is often a bottleneck).

    We can search for a solution if we are sure it's one of the bottlneck 🙂 Please test and report!



  • For history purpose (as forums will be merged), I copy here my post on XOA Forums:


    Hello all,

    We are currently evaluating XOA as a management interface to XCP/XS and espacially as a backup solution.
    As delta backup is a really good solution, the main drawback we saw is the "export bottleneck".
    Using XAPI to export VM/VHD is a good thing (much more reliable than other backup solutions) there are somme huge bottlenecks.

    The one that we disscussed with XOA support this morning is the stunnel thing.
    By default all connections to XAPI is sent trough HTTPS and stunnel is the one in charge of encryption.
    In our env, we mostly get between 60/90MB per sec as backup speed with XOA (Network is 10G and disks are SSD).
    So we tested connection (using tricky conf) in HTTP to Xen pool in order to get some improvments.

    As soon as we switched to HTTP, backup speed increased (+++). We switched to more than 200MB per sec.
    So (this is already known) stunnel is a huge bootleneck.
    Removing it reduces security (sic) but increases perfs. Security can be mitigated by the overall security of the relying network (like internal or isolated).

    Some discussions have been pointed on Citrix forums about the fact that stunnel is an old version and can be updated.
    "Recent" versions are known to have "parallel" encryption to increase performance.
    As a "stupid" test we updated stunnel (to (5.44) on a fresh install of XCP to test if it changes anything.
    But we do not noticed any improvments.

    We will make some tests later to see if a new version of stunnel can help to speedup things.


    Original post was here:
    https://xen-orchestra.com/forum/topic/1349/backup-performances-stunnel-bottleneck



  • Last update before one week holydays 🙂

    I found that in stunnel by default on XCP/XC compression is set to "zlib" which will compress all data flow.
    After some investigation it seems that XOA HTTP client to XAPI do not implement compression.
    So disabling it as no effect.
    Some tests have shown no effective difference in disabling zlib in stunnel.


  • XCP-ng Center Team

    @olivierlambert I do the http thing since we started at work with xenorchestra backup. Works fine for some years now.



  • I just made the switch in the lab now before calling it a week. I'm testing backups to the SAN over the weekend so I should have some results on Monday.

    Is it just XO that uses https by default when talking xapi or do pool members default to https as well when talking with each other? If it makes as much as a difference as is expected here, wouldn't http by default be the answer for xcp-ng with the option to turn on ssl if needed? Assuming xcpng hosts are connected to well protected management and SAN networks only then http shouldn't be an issue.

    Maybe run it on both http and https and let the admin decide which interfaces to bind to which ports based on need during install or at least via config file changes?

    If hosts are talking https by default, can they be switched to http using something similar to this? It just seems like the perf gains are pretty significant and if you're running your pool on a secured management network then this seems like a potential big win for perf.


  • XCP-ng Center Team

    @Slugger said in Remove backup bottleneck: please report:

    If hosts are talking https by default, can they be switched to http using something similar to this?

    This would be a huge improvement for disk migrations. I did and do many migration in the next days. This is what I found on CLI:

    # ps aux | cat | grep sparse_dd
    root     32352  2.3  0.1 158188 28520 ?        Ssl  23:07   0:59 /usr/libexec/xapi/sparse_dd -machine -src /dev/sm/backend/24236a41-7f7d-8edb-99df-6eb8ba26b715/cf04f6a0-1d79-4846-918e-97184cd68d16 -dest http://XX.XX.XX.XX/services/SM/nbd/187a0b1a-d927-f06f-0a3c-ec5e69414660/481306d1-3057-4ceb-9b7b-19b6e4f3dbe2/063a4c43-8abf-b2ee-dd78-29f46df34084?session_id=OpaqueRef%3addc8eaf8-aaaa-43a6-bff1-20c4a91ef371 -size 214748364800 -good-ciphersuites !EXPORT:RSA+AES128-SHA256 -legacy-ciphersuites RSA+AES256-SHA:RSA+AES128-SHA:RSA+RC4-SHA:RSA+DES-CBC3-SHA -ssl-legacy -prezeroed
    

    It seems to connect using http but has -ssl-legacy in the command, I assume is uses SSL under the hood.

    Would be nice to avoid these additional layers at all.


Log in to reply