XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    VM migration time

    Scheduled Pinned Locked Moved Management
    16 Posts 4 Posters 370 Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • cairotiC Offline
      cairoti @nikade
      last edited by cairoti

      @nikade Yes, the VM is using shared storage. My storage network is 10G with the MTU of 9000. I don't use multipathing for interfaces. I'm thinking there's something wrong with the settings.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        The network throughput is really weird. What NIC are you using? Also, double check you have MTU 9000 everywhere, this is usually a can of worm when not correctly configured.

        cairotiC 1 Reply Last reply Reply Quote 0
        • cairotiC Offline
          cairoti @olivierlambert
          last edited by cairoti

          @olivierlambert See the settings:

          • Network for pool:

          Network_pool.png

          • Network for host 1 of Pool:

          host1.png

          • Network for host 2 of Pool:

          host2.png

          • Switch Dell: MTU 9216

          • MTU on all Storage SCSI interfaces:

          interfaceStorage.png

          cairotiC 1 Reply Last reply Reply Quote 0
          • cairotiC Offline
            cairoti @cairoti
            last edited by cairoti

            I will try to list more information.

            G 1 Reply Last reply Reply Quote 0
            • nikadeN Offline
              nikade Top contributor
              last edited by

              What NIC do you have in your servers @cairoti ?

              cairotiC 2 Replies Last reply Reply Quote 0
              • G Offline
                Greg_E @cairoti
                last edited by

                @cairoti

                For me, it is normal to only see 400mbps when migrating from one NFS storage server to another NFS storage server. This is also on a 10gbe network and the drives are fast enough to benchmark a Windows VM up to 6gbps. MTU only 1500.

                Under the same storage servers and ESXi8.02, I get faster speeds and I think this is because they use nconnect=4 as the default for NFS connections. I need to do more work with ESXi and the whole vSphere system before rendering firm conclusions, but this might be a thing.

                Truenas Scale 24.10.x on both storage servers, both with spinning SATA drives for the array.

                1 Reply Last reply Reply Quote 0
                • cairotiC Offline
                  cairoti @nikade
                  last edited by

                  @nikade Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet

                  G 1 Reply Last reply Reply Quote 0
                  • G Offline
                    Greg_E @cairoti
                    last edited by

                    @cairoti

                    The Broadcom cards can be a problem, is it possible to swap out for Intel cards?

                    cairotiC 1 Reply Last reply Reply Quote 0
                    • nikadeN Offline
                      nikade Top contributor
                      last edited by

                      +1 for swapping the Broadcom NIC's - We're using only Intel for a numerous of reasons, one being performance issues in the past.
                      Havent tried Broadcom in like 10 years but I remember a lot of issues with dropped packets and driver problems.

                      1 Reply Last reply Reply Quote 0
                      • cairotiC Offline
                        cairoti @nikade
                        last edited by

                        @nikade @olivierlambert I created two Linux VMs and ran the below bandwidth and disk tests:

                        Testing using Dell Storage Network and local server volumes:

                        # iperf -c IPServer -r -t 600 -i 60
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to IPServer, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local IPClient port 54762 connected with IPServer port 5001 (icwnd/mss/irtt=14/1448/1321)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-60.0000 sec  51.6 GBytes  7.39 Gbits/sec
[  1] 60.0000-120.0000 sec  50.9 GBytes  7.28 Gbits/sec
[  1] 120.0000-180.0000 sec  51.2 GBytes  7.33 Gbits/sec
[  1] 180.0000-240.0000 sec  49.1 GBytes  7.03 Gbits/sec
[  1] 240.0000-300.0000 sec  50.8 GBytes  7.27 Gbits/sec
[  1] 300.0000-360.0000 sec  48.8 GBytes  6.99 Gbits/sec
[  1] 360.0000-420.0000 sec  51.7 GBytes  7.41 Gbits/sec
[  1] 420.0000-480.0000 sec  49.2 GBytes  7.05 Gbits/sec
[  1] 480.0000-540.0000 sec  50.1 GBytes  7.17 Gbits/sec
[  1] 540.0000-600.0000 sec  50.0 GBytes  7.16 Gbits/sec
[  1] 0.0000-600.0027 sec   503 GBytes  7.21 Gbits/sec
[  2] local IPClient port 5001 connected with IPServer port 33924
[ ID] Interval       Transfer     Bandwidth
[  2] 0.0000-60.0000 sec  50.8 GBytes  7.28 Gbits/sec
[  2] 60.0000-120.0000 sec  51.4 GBytes  7.36 Gbits/sec
[  2] 120.0000-180.0000 sec  52.4 GBytes  7.51 Gbits/sec
[  2] 180.0000-240.0000 sec  50.3 GBytes  7.21 Gbits/sec
[  2] 240.0000-300.0000 sec  50.4 GBytes  7.22 Gbits/sec
[  2] 300.0000-360.0000 sec  51.0 GBytes  7.30 Gbits/sec
[  2] 360.0000-420.0000 sec  50.6 GBytes  7.24 Gbits/sec
[  2] 420.0000-480.0000 sec  50.4 GBytes  7.22 Gbits/sec
[  2] 480.0000-540.0000 sec  50.1 GBytes  7.18 Gbits/sec
[  2] 540.0000-600.0000 sec  50.9 GBytes  7.29 Gbits/sec
[  2] 0.0000-600.0125 sec   508 GBytes  7.28 Gbits/sec

                        Test using Dell Storage volumes and networking:

                        # iperf -c IPServer -r -t 600 -i 60
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to IPServer, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local IPClient port 50006 connected with IPServer port 5001 (icwnd/mss/irtt=14/1448/4104)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-60.0000 sec  33.7 GBytes  4.82 Gbits/sec
[  1] 60.0000-120.0000 sec  33.3 GBytes  4.77 Gbits/sec
[  1] 120.0000-180.0000 sec  33.4 GBytes  4.78 Gbits/sec
[  1] 180.0000-240.0000 sec  36.1 GBytes  5.16 Gbits/sec
[  1] 240.0000-300.0000 sec  36.7 GBytes  5.25 Gbits/sec
[  1] 300.0000-360.0000 sec  32.8 GBytes  4.69 Gbits/sec
[  1] 360.0000-420.0000 sec  33.4 GBytes  4.78 Gbits/sec
[  1] 420.0000-480.0000 sec  34.5 GBytes  4.93 Gbits/sec
[  1] 480.0000-540.0000 sec  35.3 GBytes  5.05 Gbits/sec
[  1] 540.0000-600.0000 sec  34.3 GBytes  4.91 Gbits/sec
[  1] 0.0000-600.0239 sec   343 GBytes  4.92 Gbits/sec
[  2] local IPClient port 5001 connected with IPServer port 52714
[ ID] Interval       Transfer     Bandwidth
[  2] 0.0000-60.0000 sec  35.7 GBytes  5.12 Gbits/sec
[  2] 60.0000-120.0000 sec  31.6 GBytes  4.53 Gbits/sec
[  2] 120.0000-180.0000 sec  30.3 GBytes  4.34 Gbits/sec
[  2] 180.0000-240.0000 sec  35.1 GBytes  5.02 Gbits/sec
[  2] 240.0000-300.0000 sec  37.9 GBytes  5.42 Gbits/sec
[  2] 300.0000-360.0000 sec  37.5 GBytes  5.37 Gbits/sec
[  2] 360.0000-420.0000 sec  37.5 GBytes  5.37 Gbits/sec
[  2] 420.0000-480.0000 sec  37.1 GBytes  5.31 Gbits/sec
[  2] 480.0000-540.0000 sec  33.9 GBytes  4.86 Gbits/sec
[  2] 540.0000-600.0000 sec  35.0 GBytes  5.00 Gbits/sec
[  2] 0.0000-600.0036 sec   352 GBytes  5.03 Gbits/sec

                        Dell storage disk testing for each VM:

                        dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 15,3566 s, 69,9 MB/s

                        dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 19,0043 s, 56,5 MB/s

                        Server local disk test for each VM:

                        dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 6,88148 s, 156 MB/s

                        dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 5,83594 s, 184 MB/s
                        1 Reply Last reply Reply Quote 0
                        • cairotiC Offline
                          cairoti @Greg_E
                          last edited by

                          @Greg_E At this time we do not have the financial resources to purchase new boards.

                          G 1 Reply Last reply Reply Quote 0
                          • G Offline
                            Greg_E @cairoti
                            last edited by

                            @cairoti

                            I think this may be like my benchmarks, the benchmarks show decent speed to disk, but migration from server to server to local to server are just SLOW.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post