Extrlemy slow backup speec = about few MB/s



  • So why it is so slow? the XenOrchestra in on the same host where source VMs are. When I export the vm from command line the export speed is about 210MB/s within the server's local storage. For the test I've setup this DR backup:

    • the test is within one dell r510 server with perc h700
    • local storage (8x8TB wd reds)
    • two raids - raid10 and raid5
    • xen orchestra sits on raid10 volume
    • DR backup test - from raid10 to raid5 volume
    • Two Intel X5675 CPUS, 128 GB RAM

    The results:

    Top on idle (11 vm running on test r510 host):

    Last login: Fri Oct  4 13:07:07 2019 from 10.0.0.102
    [root@XCP01 ~]# top
    top - 09:57:08 up 19 days,  6:05,  1 user,  load average: 0.12, 0.16, 0.29
    Tasks: 400 total,   1 running, 399 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  0.2 us,  0.5 sy,  0.0 ni, 99.0 id,  0.1 wa,  0.0 hi,  0.0 si,  0.0 st
    KiB Mem : 16338996 total,  4881352 free,  1815788 used,  9641856 buff/cache
    KiB Swap:  1048572 total,  1024492 free,    24080 used. 14357448 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     2916 root      10 -10 1304628 156508   9808 S   1.6  1.0 395:05.17 ovs-vswitchd
     4093 65545     20   0  234732  15512   9212 S   1.6  0.1 382:26.90 qemu-system-i38
    26488 65594     20   0  225516  15692   9368 S   1.6  0.1 128:18.99 qemu-system-i38
    27402 65595     20   0  208108  15644   9356 S   1.6  0.1 129:11.60 qemu-system-i38
    22063 65542     20   0  237804  14724   9428 S   1.3  0.1 405:49.63 qemu-system-i38
    22834 65544     20   0  207084  14328   9248 S   1.3  0.1 312:12.40 qemu-system-i38
      582 65601     20   0  215276  14716   9388 S   1.0  0.1  50:00.18 qemu-system-i38
     1873 root      20   0   63156  10240   5092 S   1.0  0.1 348:53.94 forkexecd
    12452 65589     20   0  260332  14856   9524 S   1.0  0.1 114:15.44 qemu-system-i38
    20715 65540     20   0  251116  14492   9304 S   1.0  0.1 311:07.99 qemu-system-i38
    30977 root      20   0  157952   4680   3640 R   1.0  0.0   0:00.14 top
    31518 65551     20   0  251116  14628   9448 S   1.0  0.1 278:13.51 qemu-system-i38
     2463 root      20   0 1067864  64084   4508 S   0.7  0.4 168:41.81 xcp-rrdd
    20150 65569     20   0  255212  15400   9288 S   0.7  0.1 117:21.64 qemu-system-i38
        7 root      20   0       0      0      0 S   0.3  0.0  56:30.75 rcu_sched
     1930 root      20   0  295760   9124   1424 S   0.3  0.1  51:41.59 xapi-storage-sc
     2733 root      20   0   64744  13248   3444 S   0.3  0.1  23:31.58 xcp-rrdd-xenpm
     2880 root      10 -10   46960   5084   3536 S   0.3  0.0  32:57.58 ovsdb-server
     3041 root      20   0  528088  21560   4556 S   0.3  0.1  52:53.53 xcp-networkd
     4495 root      20   0  195716  10584   1112 S   0.3  0.1  19:33.88 mpathalert
    20196 root      20   0   42428   6204   2832 S   0.3  0.0  21:35.79 tapdisk
        1 root      20   0   41828   4856   3072 S   0.0  0.0  20:19.98 systemd
        2 root      20   0       0      0      0 S   0.0  0.0   0:02.00 kthreadd
        3 root      20   0       0      0      0 S   0.0  0.0   0:20.00 ksoftirqd/0
        5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
    

    The top during DR test:

    top - 10:00:03 up 19 days,  6:08,  1 user,  load average: 1.85, 0.59, 0.41
    Tasks: 407 total,   1 running, 406 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  7.6 us,  4.8 sy,  0.0 ni, 78.1 id,  7.8 wa,  0.0 hi,  1.1 si,  0.5 st
    KiB Mem : 16338996 total,  2498564 free,  1834260 used, 12006172 buff/cache
    KiB Swap:  1048572 total,  1024492 free,    24080 used. 14336504 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
     4421 root      20   0 3589768   9664   3924 S  81.2  0.1 628:48.72 stunnel
     3510 root      20   0 5924260 1.010g  12020 S  64.5  6.5   1847:36 xapi
    20408 root      20   0       0      0      0 S  17.1  0.0 162:52.33 vif34.0-q0-gues
    32544 root      20   0   46716  10520   2800 S  16.4  0.1   0:04.61 tapdisk
    20415 root      20   0       0      0      0 S  10.2  0.0  21:33.92 vif34.0-q3-deal
     3723 root      20   0   45164   8716   3620 S   6.2  0.1  23:42.15 tapdisk
    32664 root      20   0   40800   4604   2772 D   5.9  0.0   0:02.07 tapdisk
     2916 root      10 -10 1304628 156508   9808 S   2.3  1.0 395:08.61 ovs-vswitchd
     4093 65545     20   0  234732  15512   9212 S   2.0  0.1 382:29.57 qemu-system-i38
    22063 65542     20   0  237804  14724   9428 S   2.0  0.1 405:52.32 qemu-system-i38
      582 65601     20   0  215276  14716   9388 S   1.6  0.1  50:02.25 qemu-system-i38
    20715 65540     20   0  251116  14492   9304 S   1.6  0.1 311:10.05 qemu-system-i38
    26488 65594     20   0  225516  15692   9368 S   1.6  0.1 128:21.63 qemu-system-i38
    27402 65595     20   0  208108  15644   9356 S   1.6  0.1 129:14.28 qemu-system-i38
    31518 65551     20   0  251116  14628   9448 S   1.6  0.1 278:15.57 qemu-system-i38
    12452 65589     20   0  260332  14856   9524 S   1.3  0.1 114:17.52 qemu-system-i38
    20150 65569     20   0  255212  15400   9288 S   1.3  0.1 117:23.76 qemu-system-i38
    22834 65544     20   0  207084  14328   9248 S   1.3  0.1 312:14.46 qemu-system-i38
    32754 root      20   0  157952   4640   3596 R   1.3  0.0   0:00.30 top
        1 root      20   0   41828   4856   3072 S   1.0  0.0  20:20.14 systemd
     1791 root      20   0  272072   3388   2904 S   1.0  0.0  39:36.45 rsyslogd
    

    and little later TAPDISK causes 100% disk usage:

    top - 10:00:45 up 19 days,  6:08,  1 user,  load average: 2.69, 0.98, 0.55
    Tasks: 407 total,   2 running, 405 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  1.7 us,  8.1 sy,  0.0 ni, 86.3 id,  3.8 wa,  0.0 hi,  0.0 si,  0.1 st
    KiB Mem : 16338996 total,    94932 free,  1828464 used, 14415600 buff/cache
    KiB Swap:  1048572 total,  1024492 free,    24080 used. 14343268 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    32544 root      20   0   46716  10560   2840 R  95.4  0.1   0:41.81 tapdisk
     3510 root      20   0 5932456 1.010g  12020 S  40.9  6.5   1847:56 xapi
      105 root      20   0       0      0      0 S   4.6  0.0 204:37.94 kswapd0
     4093 65545     20   0  234732  15512   9212 S   2.0  0.1 382:30.26 qemu-system-i38
     2916 root      10 -10 1304628 156508   9808 S   1.7  1.0 395:09.41 ovs-vswitchd
      582 65601     20   0  215276  14716   9388 S   1.3  0.1  50:02.78 qemu-system-i38
      668 root      20   0  157952   4744   3696 R   1.3  0.0   0:00.16 top
     1950 root      20   0   70392   8136   3548 S   1.3  0.0  64:25.71 oxenstored
    20715 65540     20   0  251116  14492   9304 S   1.3  0.1 311:10.58 qemu-system-i38
    22063 65542     20   0  237804  14724   9428 S   1.3  0.1 405:52.99 qemu-system-i38
    22834 65544     20   0  207084  14328   9248 S   1.3  0.1 312:15.00 qemu-system-i38
    26488 65594     20   0  225516  15692   9368 S   1.3  0.1 128:22.28 qemu-system-i38
    27402 65595     20   0  208108  15644   9356 S   1.3  0.1 129:14.96 qemu-system-i38
    31292 root      20   0  270420  55960  13336 S   1.3  0.3   3:40.25 xcp-rrdd-iostat
     1873 root      20   0   63156  10292   5144 S   1.0  0.1 348:56.48 forkexecd
    12452 65589     20   0  260332  14856   9524 S   1.0  0.1 114:18.05 qemu-system-i38
    20150 65569     20   0  255212  15400   9288 S   1.0  0.1 117:24.31 qemu-system-i38
    31518 65551     20   0  251116  14628   9448 S   1.0  0.1 278:16.10 qemu-system-i38
     1022 root      20   0   28616   2460   2240 S   0.7  0.0  29:25.36 systemd-journal
     1791 root      20   0  272072   3388   2904 S   0.3  0.0  39:36.65 rsyslogd
     1926 root      20   0  522448  11260   3244 S   0.3  0.1  21:13.46 squeezed
     1930 root      20   0  295760   9124   1424 S   0.3  0.1  51:42.04 xapi-storage-sc
    20408 root      20   0       0      0      0 S   0.3  0.0 162:53.10 vif34.0-q0-gues
    26200 root      20   0   42336   6112   2832 S   0.3  0.0  15:24.10 tapdisk
    30924 root      20   0  139916   9036   7740 S   0.3  0.1   0:00.15 sshd
        1 root      20   0   41828   4856   3072 S   0.0  0.0  20:20.17 systemd
        2 root      20   0       0      0      0 S   0.0  0.0   0:02.00 kthreadd
        3 root      20   0       0      0      0 S   0.0  0.0   0:20.08 ksoftirqd/0
        5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
        7 root      20   0       0      0      0 S   0.0  0.0  56:31.34 rcu_sched
        8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
        9 root      rt   0       0      0      0 S   0.0  0.0   0:03.30 migration/0
       10 root      rt   0       0      0      0 S   0.0  0.0   0:09.76 watchdog/0
       11 root      rt   0       0      0      0 S   0.0  0.0   0:10.25 watchdog/1
    
    

    So it looks that the TAPDISK is the issue? The export speed in XenOrchestra is huge problem.


  • XCP-ng Team

    1. Do you have compression enabled?
    2. It's unclear, you are talking about DR, you are using DR backup then right?

    You have something wrong somewhere, but it's hard to spot right now "as is".



  • @olivierlambert
    no compression, this is from backup pane:
    a81e2625-8421-43aa-9911-7dbfdee8ced0-image.png


  • XCP-ng Team

    1. Try to disable HTTPS (in Settings server, add the host with the URL http://<IP ADDR>, disconnect/reconnect and check the DR speed after
    2. I don't understand, are you DRing to the same host?


  • @olivierlambert
    yes for the test I'm doing the DR test backup within one server to eliminate any network related issues, as I wrote before - raw file copy from raid10 to raid5 is about 210MB/s on this host so I expected that the DR backup will be done with similar speed.

    Olivier do You know anything about TAPDISK high CPU usage in xcp-ng ? this could be the caouse of slow backups?


  • XCP-ng Team

    This is related to VM storage. It's likely the VDI attached to the dom0 for the export.



  • dell r510 connected to xenorchestra thru http://ip

    top - 10:39:43 up 19 days,  6:47,  1 user,  load average: 2.02, 1.25, 1.48
    Tasks: 409 total,   3 running, 406 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  2.7 us,  9.0 sy,  0.0 ni, 84.5 id,  3.7 wa,  0.0 hi,  0.0 si,  0.1 st
    KiB Mem : 16338996 total,    91748 free,  1847752 used, 14399496 buff/cache
    KiB Swap:  1048572 total,  1024492 free,    24080 used. 14326732 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    19196 root      20   0   46716  10520   2800 R  95.7  0.1   1:06.16 tapdisk
     3510 root      20   0 7906732 1.025g  13808 S  45.1  6.6   1862:07 xapi
    19913 root      20   0  127984  17948   7296 R  16.1  0.1   0:00.49 ISOSR
      105 root      20   0       0      0      0 S   4.9  0.0 206:12.14 kswapd0
     1873 root      20   0   63156  10292   5144 S   2.3  0.1 349:22.90 forkexecd
     2916 root      10 -10 1304628 156508   9808 S   2.3  1.0 395:58.63 ovs-vswitc+
     4093 65545     20   0  234732  15512   9212 S   1.6  0.1 383:06.39 qemu-syste+
    22063 65542     20   0  237804  14724   9428 S   1.6  0.1 406:29.39 qemu-syste+
    27402 65595     20   0  208108  15644   9356 S   1.6  0.1 129:51.15 qemu-syste+
      582 65601     20   0  215276  14716   9388 S   1.3  0.1  50:30.96 qemu-syste+
    12452 65589     20   0  260332  14856   9524 S   1.3  0.1 114:46.29 qemu-syste+
    16530 65602     20   0  215276  15340   9256 S   1.3  0.1   0:16.89 qemu-syste+
    22834 65544     20   0  207084  14328   9248 S   1.3  0.1 312:43.08 qemu-syste+
    26488 65594     20   0  225516  15692   9368 S   1.3  0.1 128:58.17 qemu-syste+
    31292 root      20   0  258436  46792  13336 S   1.3  0.3   3:59.62 xcp-rrdd-i+
    31518 65551     20   0  251116  14628   9448 S   1.3  0.1 278:44.21 qemu-syste+
     2463 root      20   0 1067872  64076   4500 S   1.0  0.4 168:56.56 xcp-rrdd
    18264 root      20   0  157952   4688   3644 R   1.0  0.0   0:01.58 top
    20715 65540     20   0  251116  14492   9304 S   1.0  0.1 311:38.49 qemu-syste+
     1950 root      20   0   70392   8136   3548 S   0.7  0.0  64:41.95 oxenstored
        7 root      20   0       0      0      0 S   0.3  0.0  56:39.08 rcu_sched
     1022 root      20   0   28616   2460   2240 S   0.3  0.0  29:31.77 systemd-jo+
     1791 root      20   0  272072   3388   2904 S   0.3  0.0  39:45.22 rsyslogd
     1915 root      20   0  296096   7024   2572 S   0.3  0.0  18:17.79 v6d
     1930 root      20   0  295760   9124   1424 S   0.3  0.1  51:47.13 xapi-stora+
     3021 root      20   0   16272   2424   2192 S   0.3  0.0   5:38.92 lldpad
     3041 root      20   0  528088  22684   4556 S   0.3  0.1  52:58.77 xcp-networ+
     8374 root      20   0       0      0      0 S   0.3  0.0   0:24.18 cifsd
    13065 root      20   0       0      0      0 S   0.3  0.0   0:00.04 kworker/14+
    20196 root      20   0   42672   6308   2824 S   0.3  0.0  21:38.64 tapdisk
    22399 root      20   0   43792   7568   2832 S   0.3  0.0  33:32.07 tapdisk
    23958 root      20   0       0      0      0 S   0.3  0.0   1:42.64 vif7.0-q1-+
    25141 root      20   0       0      0      0 S   0.3  0.0   0:19.15 kworker/7:2
    26200 root      20   0   42336   6112   2832 S   0.3  0.0  15:30.53 tapdisk
    31278 root      20   0   44192   7968   2832 S   0.3  0.0   3:37.06 tapdisk
        1 root      20   0   41828   4856   3072 S   0.0  0.0  20:22.00 systemd
        2 root      20   0       0      0      0 S   0.0  0.0   0:02.00 kthreadd
    

    5d244f79-e690-447f-8794-2724d55e0d10-image.png


  • XCP-ng Team

    Can you give me number in plain text so I haven't to find the value in screenshots? Thanks 🙂



  • Eveything except the nmon is in text, here is nmon:

    These are the values during the DR backup problem:

    lnmonq16eqqqqqq[H for help]qqqHostname=XCP01qqqqqqqqRefresh= 2secs qqq11:59.13qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
    x CPU Utilisation qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq                                                                 x
    x---------------------------+-------------------------------------------------+                                                                                                                                         x
    xCPU User%  Sys% Wait% Steal|0          |25         |50          |75       100|                                                                                                                                         x
    x  1   1.5   5.1   8.1   0.5|ssWWWW       >                                   |                                                                                                                                         x
    x  2   0.5   0.0   0.0  99.5|   >                                             |                                                                                                                                         x
    x  3   2.5   7.1   6.6  83.8|UsssWWW>                                         |                                                                                                                                         x
    x  4   1.5   2.0   0.0  96.5|s>                                               |                                                                                                                                         x
    x  5   5.1  51.8   1.0  42.1|UUsssssssssssssssssssssssss           >          |                                                                                                                                         x
    x  6   0.5   1.0   0.0  98.5|>                                                |                                                                                                                                         x
    x  7   4.5  17.0   7.0   0.5|UUssssssssWWW>                                   |                                                                                                                                         x
    x  8   5.0   6.5  12.4   0.5|UUsssWWWWWW>                                     |                                                                                                                                         x
    x  9   6.2  29.2  12.3  52.3|UUUssssssssssssssWWWWWW>                         |                                                                                                                                         x
    x 10   0.0   1.0   0.0  99.0|>                                                |                                                                                                                                         x
    x 11   1.5   3.5   4.0   0.5|sWW>                                             |                                                                                                                                         x
    x 12   0.0   0.5   0.0  99.5|>                                                |                                                                                                                                         x
    x 13   0.5   4.0   0.5  95.0|ss       >                                       |                                                                                                                                         x
    x 14   0.5   2.0   0.0  97.5|>                                                |                                                                                                                                         x
    x 15   2.0   4.5   5.0   0.5|UssWW>                                           |                                                                                                                                         x
    x 16   0.5   0.5   0.0  99.0|      >                                          |                                                                                                                                         x
    x---------------------------+-------------------------------------------------+                                                                                                                                         x
    xAvg   2.0   8.5   3.6   0.1|UssssW>                                          |                                                                                                                                         x
    x---------------------------+-------------------------------------------------+                                                                                                                                         x
    x Network I/O qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq                                                                 x
    xI/F Name Recv=KB/s Trans=KB/s packin packout insize outsize Peak->Recv Trans                                                                                                                                           x
    x vif54.0      0.0       2.3       0.0   25.4     0.0   93.6       13.4      8.6                                                                                                                                        x
    x      lo      0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0                                                                                                                                        x
    x    eth3      0.9       0.1       8.0    1.5   115.0   66.0        4.6      1.6                                                                                                                                        x
    x vif10.0      0.0       2.3       0.0   25.4     0.0   93.6        0.2      5.8                                                                                                                                        x
    x vif60.0      0.0       2.3       0.0   25.4     0.0   93.6        0.0      5.5                                                                                                                                        x
    x    eth2      1.4       0.1       7.5    1.0   186.3  137.0       13.1      1.0                                                                                                                                        x
    x  vif9.0      0.0       2.3       0.0   25.4     0.0   93.6        0.6      9.0                                                                                                                                        x
    x  vif5.0      0.0       2.3       0.0   25.4     0.0   93.6        0.0      5.5                                                                                                                                        x
    x vif67.0      0.4       3.3       2.0   27.9   204.8  122.2     1038.5   1050.0                                                                                                                                        x
    x vif59.0      0.2       2.5       2.0   27.4   118.5   92.3        1.0      6.2                                                                                                                                        x
    x vif16.0      0.0       2.3       0.0   25.4     0.0   93.6        0.0      5.5                                                                                                                                        x
    xovs-syste     0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0                                                                                                                                        x
    x  xenbr0      2.4       9.6      30.9    8.0    78.7 1229.2     1051.3   1064.4                                                                                                                                        x
    x    eth5      0.9       0.0      10.0    0.0    91.0    0.0        2.3   1085.7                                                                                                                                        x
    x vif66.0      0.0       2.3       0.0   25.4     0.0   93.6        0.0      5.5                                                                                                                                        x
    x    eth1      0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0                                                                                                                                        x
    x  xenbr1      0.0       0.0       0.0    0.0     0.0    0.0        0.0      0.0                                                                                                                                        x
    x   xapi0      2.0       0.0      25.4    0.0    79.6    0.0        4.6      0.0                                                                                                                                        x
    x    eth4      0.4       0.5       5.5    1.5    74.6  311.0     1080.7     13.0                                                                                                                                        x
    x    eth0      2.7       8.7      27.9    8.0    97.6 1122.5     1101.5   1119.1                                                                                                                                        x
    x  vif7.0      0.0       2.3       0.0   25.4     0.0   93.6        0.0      5.5                                                                                                                                        x
    x Network Error Counters qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq                                                                 x
    xI/F Name iErrors iDrop iOverrun iFrame oErrors   oDrop oOverrun oCarrier oColls                                                                                                                                        x
    x vif54.0       0       0       0       0       0       0       0       0       0                                                                                                                                       x
    x      lo       0       0       0       0       0       0       0       0       0                                                                                                                                       x
    x    eth3      33      32      32      16       0       0       0       0       0                                                                                                                                       x
    x vif10.0       0       0       0       0       0       0       0       0       0                                                                                                                                       x
    x vif60.0       0       0       0       0       0       0       0       0       0                                                                                                                                       x
    x    eth2       4       0       0       2       0       0       0       0       0                                                                                                                                       x
    x  vif9.0       0       0       0       0       0       0       0       0       0                                        
    

  • XCP-ng Team

    That's not what I'm telling 😉 Everybody isn't used to those software, try to just sum up with the value you want to put in light. And why this value isn't what you expect.



  • For me the 100% cpu of TAPDISK is the problem, IDK what else to check, below result of DR test (backup within the same server) - backup speed: 3,61MB/s

     28_Venus (XCP01) 
     Snapshot 
    Start: Oct 7, 2019, 11:56:48 AM
    End: Oct 7, 2019, 11:56:54 AM
     x1-ext-sr02 (14.43 TiB free) - XCP01 
     transfer 
    Start: Oct 7, 2019, 11:57:11 AM
    End: Oct 7, 2019, 12:30:24 PM
    Duration: 33 minutes
    Size: 7.02 GiB
    Speed: 3.61 MiB/s
    Start: Oct 7, 2019, 11:57:11 AM
    End: Oct 7, 2019, 12:30:45 PM
    Duration: 34 minutes
    Start: Oct 7, 2019, 11:56:48 AM
    End: Oct 7, 2019, 12:30:45 PM
    Duration: 34 minutes
    

    Olivier that do you tkink about that:
    https://portal.hiveio.com/kb/KBA-001506-URaAAO.aspx

    Root Cause:
    When executing locally, XenServer's VIF uses up too much CPU, causing CPU contention with tapdisk. As a result, I/O throughput is much lower.

    Solution:
    Enable TSO/GSO on the Volume VM via rc.local. This will result in the VIF using up much less CPU and giving tapdisk more resources to process I/O.
    Instructions on how to change TSO/GSO settings can be found here: Performance tuning your USX environment.


  • XCP-ng Team

    Anything in dmesg?



  • with 'errors' only that:

    [   28.275518] Could not initialize VPMU for cpu 0, error -95
    [   28.275569] Performance Events: running under Xen, no PMU driver, software events only.
    [   28.276154] NMI watchdog: disabled (cpu0): hardware events not enabled
    [   28.276155] NMI watchdog: Shutting down hard lockup detector on all cpus
    

    xen top: Domain-0 210% cpu usage?

    
    [root@XCP01 ~]# xentop
    xentop - 14:22:32   Xen 4.7.6-6.6.xcpng
    11 domains: 1 running, 10 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown
    Mem: 134204508k total, 78431792k used, 55772716k free    CPUs: 24 @ 3066MHz
          NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO   VBD_RD   VBD_WR  VBD_RSECT  VBD_WSECT SSID
       221_rc01 --b---      16765    3.9    8388600    6.3    8389632       6.3     4    0        0        0    2        0   610009  1090329   29411455   34377572    0
      223_rfs01 --b---      53419    9.0    8388600    6.3    8389632       6.3     4    0        0        0    2        0   662551  1308614   35627468   50098513    0
      225_gtlas --b---      44368    2.4    8388600    6.3    8389632       6.3     4    0        0        0    2        0  4857299  1985548  374377550   77974490    0
      228_henfs --b---      20814    2.9    4194296    3.1    4195328       3.1     2    0        0        0    2        0    16850  2694099     668534   42791106    0
    320_dsaomete --b---       8001    0.7    4194296    3.1    4195328       3.1     2    0        0        0    2        0   405243  1103612   33437483   27560602    0
    333_judsater --b---        551    0.3    4194292    3.1    4195328       3.1     2    0        0        0    1        0     4590     4065     169846      53784    0
    414_wd4310_s --b---      52670    2.5    4194296    3.1    4195328       3.1     2    0        0        0    1        0   986147  2221354   39669481   69579321    0
    445_dd3en9 --b---       3430    0.3    4194292    3.1    4195328       3.1     2    0        0        0    1        0     6911   130167     313526    6857488    0
    619_und13i_c --b---      41818    3.1    4194292    3.1    4195328       3.1     2    0        0        0    1        0    11867  3998049     424982   69774736    0
     18_xoa_x1 --b---       1724   94.6    8388596    6.3    8389632       6.3     4    0        0        0    1        0    28555     2058     511456      52712    0
      Domain-0 -----r    1022492  189.0   16777216   12.5   16777216      12.5    16    0        0        0    0        0        0        0          0          0    0
    
    
      Delay  Networks  vBds  Tmem  VCPUs  Repeat header  Sort order  Quit
    
    

    also I've got a lot off errors on network level but in abovementioned example the data nevel leaves the xcp-ng host during the DR backup I presume?

    tcpdump -i eth0 -v -nn| grep -i incorrect
    

    222099 packets received by filter
    1966 packets dropped by kernel

    But I don't see any transfers drops to and from VMs within out netowrk - everything works over 100MB/s - IDK - if this is the way to find the root cause of the slow backup issue, but the switch does not report major problems on network:
    084798e5-db3d-4bc5-a58a-1f27743edfa2-image.png

    simple hdparm (raid10 and raid5 volume):

    /dev/sda:
     Timing cached reads:   10966 MB in  1.95 seconds = 5625.90 MB/sec
     Timing buffered disk reads: 1194 MB in  3.00 seconds = 397.78 MB/sec
    [root@XCP01 ~]# hdparm -tT /dev/sdc
    /dev/sdc:
     Timing cached reads:   11722 MB in  1.97 seconds = 5941.87 MB/sec
     Timing buffered disk reads: 1210 MB in  3.00 seconds = 402.90 MB/sec
    
    


  • @olivierlambert

    can You take a look on this: when I copy the VM within the one server it runs as hell, there is no TAPDISK problem with high CPU, so why when doing Didaster Recovery backup the TAPDISK hit 100% of CPU?

    90da95e4-3ce2-4db7-a6b7-4d8173a5355e-image.png


  • XCP-ng Team

    VDI copy and DR aren't using the same mechanism. It's probably XVA creation that got a problem on your setup. Try Continuous Replication to see if you have a better perf.



  • for sure something is wrong, but what? how the DR backup copy process is different from VM Copy? Can You give me more info? I don't know how to troubleshoot that, this is major problem for us...


  • XCP-ng Team

    If it's blocking in production, I would suggest to open a support ticket, it's more efficient for us to assist than here 🙂

    Regarding copy, you said you used VM copy: you mean the copy button in the VM view? Or fast clone? (last one isn't the same). Because indeed, VM copy is very similar to DR.

    Please try with CR to see if there's a perf change (that might be helpful to pinpoint the problem)



  • @olivierlambert said in Extrlemy slow backup speec = about few MB/s:

    Regarding copy, you said you used VM copy: you mean the copy button in the VM view? Or fast clone? (last one isn't the same). Because indeed, VM copy is very similar to DR.

    Please:) of course full clone.


  • XCP-ng Team

    So you did use "Clone" button or "Copy" button?



  • I don't have clone button, only Copy
    5f5e0309-adbf-4d81-92ae-e1120934ac8c-image.png


Log in to reply
 

XCP-ng Pro Support

XCP-ng Pro Support