Extrlemy slow backup speec = about few MB/s



  • @rjt said in Extrlemy slow backup speec = about few MB/s:

    How much RAM do you allocate to Dom0 on your hosts

    16GB to dom0, but the backup speed is still dramatic - in Disaster Recovery its a few mB/s. Nothing has changed from since then. I think this is dead end, and will have to make some business desicions regarding the hypervizor when jumping next year.


  • XCP-ng Team

    And what about CR?

    Obviously, you have a problem on DR/VM export mechanism (XVA). Also 8.1 will bring an improved XVA generation system, that will solve your issue.



  • @olivierlambert Is there a github issue for this? Does the xcp-ng or xen project have something in mind?


  • XCP-ng Team

    For what issue?



  • @olivierlambert Would the ”dom0 userland process/tools that must be improved” actually mean functionality that does not actually require Dom0? Wondering if dom0less Xen would speed things up.


  • XCP-ng Team

    And how/where would you attach the disk to be exported? You must have a dom0 for some operation (or at least an extra domain for export).

    I was more talking about improving tools that are exposing the disk itself.



  • @olivierlambert said in Extrlemy slow backup speec = about few MB/s:

    And what about CR?

    Obviously, you have a problem on DR/VM export mechanism (XVA). Also 8.1 will bring an improved XVA generation system, that will solve your issue.

    I'm planning the upgrade from 7.6 to 8.x in about 3-4 months, also I'll build the test environement. I'll see how it goes.



  • Hi all,

    i'm faced with the same issue:

     TestVM (redfour) 
     Snapshot 
    Start: Jan 13, 2020, 10:13:51 PM
    End: Jan 13, 2020, 10:13:56 PM
     Local iSCSI 
     transfer 
    Start: Jan 13, 2020, 10:13:56 PM
    End: Jan 13, 2020, 10:17:18 PM
    Duration: 3 minutes
    Size: 813.17 MiB
    Speed: 4.03 MiB/s
    Start: Jan 13, 2020, 10:13:56 PM
    End: Jan 13, 2020, 10:17:18 PM
    Duration: 3 minutes
    Start: Jan 13, 2020, 10:13:51 PM
    End: Jan 13, 2020, 10:17:19 PM
    Duration: 3 minutes
    

    where the backup volume is served on iscsi by a machine having raid5 over 1 Gbps back to back connection to the hypervisor.

    Mouting that volume in the hypervisor and doing some transfers, traffic easily goes up and beyond 800 Mbps whereas the backups takes forever to execute.

    I am uzing zstd as compression and xcp-ng 8.0 on the hypervisor

    One thing i noticed is that xo-server takes 100-130% cpu when the backup is executing.
    The vm above is about 2gb uncompressed, however larger VMs take over 8 hours to complete a normal backup, nothing fancy.

    Thanks,
    Viorel


  • XCP-ng Team

    1. Are you using XOA?
    2. What's the CPU model/freq running in XO VM?


  • Hi Olivier,

    yeap, xo from sources installed in mid december, so fairly new.

    It's running on a VM with 4 vCPUS of xeon x5355 @ 2.66 Ghz



  • Any ETA on an updated xen orchestra that will support the 8.1 (beta).
    I just upgraded a spare host to see if the backup speed has improved, but no luck, backups won't run so i'm guessing we need an updated version 🙂

    Merci,
    Viorel


  • XCP-ng Team

    This is the first report of failing backups with 8.1 but that's what a beta is for! Can you tell us more about backups that won't run?


  • XCP-ng Team

    We don't have any reports from our customers of XOA backup failing with CH 8.1, so it should be pretty similar with XCP-ng 8.1.

    I bet on a configuration problem on maybe your installation. Please try with XOA and not XO from the sources.



  • Hi All,

    after re-creating the backup task, i get the following message in the failed task:

    RevDNS (redthree)
    Snapshot
    Start: Feb 7, 2020, 12:00:00 AM
    End: Feb 7, 2020, 12:00:00 AM
    Error: MESSAGE_REMOVED()
    Start: Feb 7, 2020, 12:00:00 AM
    End: Feb 7, 2020, 12:00:00 AM
    Duration: a few seconds
    Error: MESSAGE_REMOVED()

    If there's any other log sources that may provide more insight, i'd be happy to take a look

    Thanks


  • XCP-ng Team

    @synologic Your XO is outdated. Please stay up to date.



  • Will update right away to the latest version and test.

    Thanks



  • Ok, upgrading to the latest version worked. Thank you.

    I see some improvements in the backup speed like:

    • from xcp 8.1 - no compression - about 33 MiB/s
    • from xcp 8.1 - compression (zstd) about 15 MiB/s

    both are up from about 4 MiB/s

    However, i am also seeing vast improvements when backing up from XCP 8.0.0 with transfer speeds (as reported by iptraf-ng) going up to 700 Mbps, which was not happening before.

    Viorel



  • @synologic said in Extrlemy slow backup speec = about few MB/s:

    Ok, upgrading to the latest version worked. Thank you.

    I see some improvements in the backup speed like:

    • from xcp 8.1 - no compression - about 33 MiB/s
    • from xcp 8.1 - compression (zstd) about 15 MiB/s

    both are up from about 4 MiB/s

    However, i am also seeing vast improvements when backing up from XCP 8.0.0 with transfer speeds (as reported by iptraf-ng) going up to 700 Mbps, which was not happening before.

    Viorel

    Maybe you saw the same thing I did - I also had major issues with backups. Then in the summer of 2019 I upgraded our XOA (Yes, we have the paid version) and everything was going 3-4x times faster. Must have been some major improvement in the XOA code.



  • @nikade said in Extrlemy slow backup speec = about few MB/s:

    Maybe you saw the same thing I did - I also had major issues with backups. Then in the summer of 2019 I upgraded our XOA (Yes, we have the paid version) and everything was going 3-4x times faster. Must have been some major improvement in the XOA code.

    So maybe some official info that if you have XO from source Your backup speeds will be as low as 3-4 MB/s? Or what?


  • XCP-ng Team

    @akurzawa XOA is not supposed to be faster than XO from sources.


Log in to reply
 

XCP-ng Pro Support

XCP-ng Pro Support