XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    VM migration time

    Scheduled Pinned Locked Moved Management
    16 Posts 4 Posters 371 Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • cairotiC Offline
      cairoti @olivierlambert
      last edited by

      @olivierlambert The interface used for direct communication with storage is eth5.

      fig1.png

      fig2.png

      1 Reply Last reply Reply Quote 0
      • nikadeN Offline
        nikade Top contributor
        last edited by

        Is the VM on shared storage?
        Storage migration is quite slow, but if the VM is on shared storage the migration is usually pretty quick, we've seen about 7Gbit/s on a 10G network with MTU 9000.

        cairotiC 1 Reply Last reply Reply Quote 0
        • cairotiC Offline
          cairoti @nikade
          last edited by cairoti

          @nikade Yes, the VM is using shared storage. My storage network is 10G with the MTU of 9000. I don't use multipathing for interfaces. I'm thinking there's something wrong with the settings.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            The network throughput is really weird. What NIC are you using? Also, double check you have MTU 9000 everywhere, this is usually a can of worm when not correctly configured.

            cairotiC 1 Reply Last reply Reply Quote 0
            • cairotiC Offline
              cairoti @olivierlambert
              last edited by cairoti

              @olivierlambert See the settings:

              • Network for pool:

              Network_pool.png

              • Network for host 1 of Pool:

              host1.png

              • Network for host 2 of Pool:

              host2.png

              • Switch Dell: MTU 9216

              • MTU on all Storage SCSI interfaces:

              interfaceStorage.png

              cairotiC 1 Reply Last reply Reply Quote 0
              • cairotiC Offline
                cairoti @cairoti
                last edited by cairoti

                I will try to list more information.

                G 1 Reply Last reply Reply Quote 0
                • nikadeN Offline
                  nikade Top contributor
                  last edited by

                  What NIC do you have in your servers @cairoti ?

                  cairotiC 2 Replies Last reply Reply Quote 0
                  • G Offline
                    Greg_E @cairoti
                    last edited by

                    @cairoti

                    For me, it is normal to only see 400mbps when migrating from one NFS storage server to another NFS storage server. This is also on a 10gbe network and the drives are fast enough to benchmark a Windows VM up to 6gbps. MTU only 1500.

                    Under the same storage servers and ESXi8.02, I get faster speeds and I think this is because they use nconnect=4 as the default for NFS connections. I need to do more work with ESXi and the whole vSphere system before rendering firm conclusions, but this might be a thing.

                    Truenas Scale 24.10.x on both storage servers, both with spinning SATA drives for the array.

                    1 Reply Last reply Reply Quote 0
                    • cairotiC Offline
                      cairoti @nikade
                      last edited by

                      @nikade Broadcom NetXtreme II BCM57810 10 Gigabit Ethernet

                      G 1 Reply Last reply Reply Quote 0
                      • G Offline
                        Greg_E @cairoti
                        last edited by

                        @cairoti

                        The Broadcom cards can be a problem, is it possible to swap out for Intel cards?

                        cairotiC 1 Reply Last reply Reply Quote 0
                        • nikadeN Offline
                          nikade Top contributor
                          last edited by

                          +1 for swapping the Broadcom NIC's - We're using only Intel for a numerous of reasons, one being performance issues in the past.
                          Havent tried Broadcom in like 10 years but I remember a lot of issues with dropped packets and driver problems.

                          1 Reply Last reply Reply Quote 0
                          • cairotiC Offline
                            cairoti @nikade
                            last edited by

                            @nikade @olivierlambert I created two Linux VMs and ran the below bandwidth and disk tests:

                            Testing using Dell Storage Network and local server volumes:

                            # iperf -c IPServer -r -t 600 -i 60
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to IPServer, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local IPClient port 54762 connected with IPServer port 5001 (icwnd/mss/irtt=14/1448/1321)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-60.0000 sec  51.6 GBytes  7.39 Gbits/sec
[  1] 60.0000-120.0000 sec  50.9 GBytes  7.28 Gbits/sec
[  1] 120.0000-180.0000 sec  51.2 GBytes  7.33 Gbits/sec
[  1] 180.0000-240.0000 sec  49.1 GBytes  7.03 Gbits/sec
[  1] 240.0000-300.0000 sec  50.8 GBytes  7.27 Gbits/sec
[  1] 300.0000-360.0000 sec  48.8 GBytes  6.99 Gbits/sec
[  1] 360.0000-420.0000 sec  51.7 GBytes  7.41 Gbits/sec
[  1] 420.0000-480.0000 sec  49.2 GBytes  7.05 Gbits/sec
[  1] 480.0000-540.0000 sec  50.1 GBytes  7.17 Gbits/sec
[  1] 540.0000-600.0000 sec  50.0 GBytes  7.16 Gbits/sec
[  1] 0.0000-600.0027 sec   503 GBytes  7.21 Gbits/sec
[  2] local IPClient port 5001 connected with IPServer port 33924
[ ID] Interval       Transfer     Bandwidth
[  2] 0.0000-60.0000 sec  50.8 GBytes  7.28 Gbits/sec
[  2] 60.0000-120.0000 sec  51.4 GBytes  7.36 Gbits/sec
[  2] 120.0000-180.0000 sec  52.4 GBytes  7.51 Gbits/sec
[  2] 180.0000-240.0000 sec  50.3 GBytes  7.21 Gbits/sec
[  2] 240.0000-300.0000 sec  50.4 GBytes  7.22 Gbits/sec
[  2] 300.0000-360.0000 sec  51.0 GBytes  7.30 Gbits/sec
[  2] 360.0000-420.0000 sec  50.6 GBytes  7.24 Gbits/sec
[  2] 420.0000-480.0000 sec  50.4 GBytes  7.22 Gbits/sec
[  2] 480.0000-540.0000 sec  50.1 GBytes  7.18 Gbits/sec
[  2] 540.0000-600.0000 sec  50.9 GBytes  7.29 Gbits/sec
[  2] 0.0000-600.0125 sec   508 GBytes  7.28 Gbits/sec

                            Test using Dell Storage volumes and networking:

                            # iperf -c IPServer -r -t 600 -i 60
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to IPServer, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local IPClient port 50006 connected with IPServer port 5001 (icwnd/mss/irtt=14/1448/4104)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-60.0000 sec  33.7 GBytes  4.82 Gbits/sec
[  1] 60.0000-120.0000 sec  33.3 GBytes  4.77 Gbits/sec
[  1] 120.0000-180.0000 sec  33.4 GBytes  4.78 Gbits/sec
[  1] 180.0000-240.0000 sec  36.1 GBytes  5.16 Gbits/sec
[  1] 240.0000-300.0000 sec  36.7 GBytes  5.25 Gbits/sec
[  1] 300.0000-360.0000 sec  32.8 GBytes  4.69 Gbits/sec
[  1] 360.0000-420.0000 sec  33.4 GBytes  4.78 Gbits/sec
[  1] 420.0000-480.0000 sec  34.5 GBytes  4.93 Gbits/sec
[  1] 480.0000-540.0000 sec  35.3 GBytes  5.05 Gbits/sec
[  1] 540.0000-600.0000 sec  34.3 GBytes  4.91 Gbits/sec
[  1] 0.0000-600.0239 sec   343 GBytes  4.92 Gbits/sec
[  2] local IPClient port 5001 connected with IPServer port 52714
[ ID] Interval       Transfer     Bandwidth
[  2] 0.0000-60.0000 sec  35.7 GBytes  5.12 Gbits/sec
[  2] 60.0000-120.0000 sec  31.6 GBytes  4.53 Gbits/sec
[  2] 120.0000-180.0000 sec  30.3 GBytes  4.34 Gbits/sec
[  2] 180.0000-240.0000 sec  35.1 GBytes  5.02 Gbits/sec
[  2] 240.0000-300.0000 sec  37.9 GBytes  5.42 Gbits/sec
[  2] 300.0000-360.0000 sec  37.5 GBytes  5.37 Gbits/sec
[  2] 360.0000-420.0000 sec  37.5 GBytes  5.37 Gbits/sec
[  2] 420.0000-480.0000 sec  37.1 GBytes  5.31 Gbits/sec
[  2] 480.0000-540.0000 sec  33.9 GBytes  4.86 Gbits/sec
[  2] 540.0000-600.0000 sec  35.0 GBytes  5.00 Gbits/sec
[  2] 0.0000-600.0036 sec   352 GBytes  5.03 Gbits/sec

                            Dell storage disk testing for each VM:

                            dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 15,3566 s, 69,9 MB/s

                            dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 19,0043 s, 56,5 MB/s

                            Server local disk test for each VM:

                            dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 6,88148 s, 156 MB/s

                            dd if=/dev/zero of=/tmp/teste1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1,1 GB, 1,0 GiB) copied, 5,83594 s, 184 MB/s
                            1 Reply Last reply Reply Quote 0
                            • cairotiC Offline
                              cairoti @Greg_E
                              last edited by

                              @Greg_E At this time we do not have the financial resources to purchase new boards.

                              G 1 Reply Last reply Reply Quote 0
                              • G Offline
                                Greg_E @cairoti
                                last edited by

                                @cairoti

                                I think this may be like my benchmarks, the benchmarks show decent speed to disk, but migration from server to server to local to server are just SLOW.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post