XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Poor VM performance after migrating from VMWare to XCP-NG

    Scheduled Pinned Locked Moved Migrate to XCP-ng
    15 Posts 5 Posters 1.6k Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M Offline
      markds
      last edited by

      HI,
      We are currently testing the suitability of XCP-NG for being a replacement of VMWare pool. I am sure I are not alone here, given VMWare recent policy changes.

      I have converted one of the VMWare hosts to XCP-NG 8.2 and have notice the test Linux VM is performing dramatically slower.

      • Loading our large iptables rule set takes 15sec on VMWare vs 140 sec on XCP-NG

      • The average system load (reported via munin) is roughly 2x higher under XCP-NG

      • Via terminal I have notice large spikes in load (as high as 9) (using uptime)

      • The average CPU usage seems 2x higher (increase is in system not user space)

      • Noticed a few instances of packet loss might this might be due to the spikes in system load.

      • The kernel of the guest vm is 3.2.102

      • The block/network modules are loaded (xen_netfront / xen_blkfront) in the guest vm

      • Management agent 7.30.0-12 is loaded in the guest vm

      • XOA correctly reports that the guest is virtualised via PVHVM

      • There doesn't appear to be any vmware modules / agents running

      • The host is only running this single VM (previously under VMWare there were 4)

      • The host reports a load average of 0.4 and 18% cpu usage (via XOA)

      • Hypertheading is disabled

      • No hardware faults reported via iDrac

      So far I have attempted to:

      • Increasing the vCPU from 4 -> 8
      • Enabling / disabling TX offload
      • Running a different kernel version on the guest
      • Running trim on the SSDs to ensure they are running efficiently

      Any ideas why under XCP-NG the system is running so poorly?

      1 Reply Last reply Reply Quote 0
      • DanpD Offline
        Danp Pro Support Team
        last edited by

        A few questions for you --

        • Are you booting in BIOS or UEFI mode? FWIW, I've read where others found the migrated VM ran better in UEFI mode.

        • How did you migrate the VM (XO's V2V tool vs manual export / import)?

        • Have you tried creating a brand new VM using the appropriate template and then attaching the existing disk in place of the default one?

        M 2 Replies Last reply Reply Quote 0
        • M Offline
          markds @Danp
          last edited by

          1. Booting via BIOS mode (both in VMWare and XCP-NG)

          2. Used XO's vmware import tool
            (from memory I believe the VM was powered off and then imported vs taking a snapshot then importing, but can not be sure)

          3. Have not tried creating a brand new VM and attaching the disk. Do you suspect the import has copied over some strange config impacting performance?

          BTW thanks for the prompt response @Danp

          DanpD R 2 Replies Last reply Reply Quote 0
          • DanpD Offline
            Danp Pro Support Team @markds
            last edited by

            @markds said in Poor VM performance after migrating from VMWare to XCP-NG:

            Have not tried creating a brand new VM and attaching the disk

            You should definitely try this. It was reported elsewhere on this forum that this improved performance, but I don't recall if this was for all VMs or specific to Windows.

            1 Reply Last reply Reply Quote 0
            • M Offline
              markds @Danp
              last edited by

              @Danp

              No luck... still

              • I created a full clone of the migrated VM
              • Ensured the clone was shutdown.
              • Create a new VM via the template (no disks, attached a cdrom, same mac as clone)
              • Edited the new VM
                - removed the reference the cdrom
                - attached the cloned disk using the uuid reference (which is a cool feature)
                - changed the boot order so that harddrive was first
              • ran our ip tables and the same issue 2mins 7 secs so I suspect the underlaying issue still exists...

              how do I confirm that the xen network drivers are in use:
              lsmod | grep xen
              xenfs 17639 1
              xen_netfront 22032 0
              xen_blkfront 17478 5

              does that not indicate that whilst xen_netfront module is loaded, no device is using that module?

              DanpD 1 Reply Last reply Reply Quote 0
              • DanpD Offline
                Danp Pro Support Team @markds
                last edited by

                @markds Looks that way. What output do you get for sudo dmesg | grep xen?

                M 3 Replies Last reply Reply Quote 0
                • M Offline
                  markds @Danp
                  last edited by

                  @Danp

                  [ 0.824078] xen/balloon: Initialising balloon driver.
                  [ 0.825817] xen-balloon: Initialising balloon driver.
                  [ 0.848060] Switching to clocksource xen
                  [ 0.861416] xen: --> pirq=16 -> irq=8 (gsi=8)
                  [ 0.861486] xen: --> pirq=17 -> irq=12 (gsi=12)
                  [ 0.861531] xen: --> pirq=18 -> irq=1 (gsi=1)
                  [ 0.861580] xen: --> pirq=19 -> irq=6 (gsi=6)
                  [ 0.861632] xen: --> pirq=20 -> irq=4 (gsi=4)
                  [ 0.861692] xen: --> pirq=21 -> irq=7 (gsi=7)
                  [ 0.916200] xen: --> pirq=22 -> irq=23 (gsi=23)
                  [ 1.135771] xen: --> pirq=23 -> irq=28 (gsi=28)
                  [ 1.936447] vbd vbd-5696: 19 xenbus_dev_probe on device/vbd/5696

                  1 Reply Last reply Reply Quote 0
                  • M Offline
                    markds @Danp
                    last edited by

                    @Danp

                    Actually maybe its a red herring...

                    ethtool -i eth3
                    driver: vif
                    version:
                    firmware-version:
                    bus-info: vif-2

                    Also I tested using iperf:
                    vm <-> vm: 9.78 Gbits/sec (on the same xcp-ng host)
                    vm <-> exsi: 0.97 Gbits/sec (network is 1GB)

                    As a final test I ran:
                    rmmod xen_netfront
                    and as expected the vms lost network access

                    So maybe the issue is elsewhere

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by olivierlambert

                      Can you reproduce the issue by creating a new Linux VM from scratch on XCP-ng? (ie installing the OS etc.) This will help to understand if it's a setup-wise or VM wise issue

                      M 1 Reply Last reply Reply Quote 0
                      • M Offline
                        markds @Danp
                        last edited by

                        @Danp
                        I should also correct my previous answer...

                        Looks like some of the dmesg are in a different case...

                        dmesg | grep -i xen
                        [ 0.000000] DMI: Xen HVM domU, BIOS 4.13 04/11/2024
                        [ 0.000000] Hypervisor detected: Xen HVM
                        [ 0.000000] Xen version 4.13.
                        [ 0.000000] Xen Platform PCI: I/O protocol version 1
                        [ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
                        [ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
                        [ 0.000000] ACPI: RSDP 0x00000000000EA020 00024 (v02 Xen)
                        [ 0.000000] ACPI: XSDT 0x00000000FC00A7C0 00044 (v01 Xen HVM 00000000 HVML 00000000)
                        [ 0.000000] ACPI: FACP 0x00000000FC00A370 000F4 (v04 Xen HVM 00000000 HVML 00000000)
                        [ 0.000000] ACPI: DSDT 0x00000000FC001040 092A3 (v02 Xen HVM 00000000 INTL 20160527)
                        [ 0.000000] ACPI: APIC 0x00000000FC00A470 00260 (v02 Xen HVM 00000000 HVML 00000000)
                        [ 0.000000] ACPI: HPET 0x00000000FC00A750 00038 (v01 Xen HVM 00000000 HVML 00000000)
                        [ 0.000000] ACPI: WAET 0x00000000FC00A790 00028 (v01 Xen HVM 00000000 HVML 00000000)
                        [ 0.000000] Booting paravirtualized kernel on Xen HVM
                        [ 0.000000] Xen HVM callback vector for event delivery is enabled
                        [ 0.113732] Xen: using vcpuop timer interface
                        [ 0.113741] installing Xen timer for CPU 0
                        [ 0.390521] installing Xen timer for CPU 1
                        [ 0.492037] installing Xen timer for CPU 2
                        [ 0.590329] installing Xen timer for CPU 3
                        [ 0.918283] xen/balloon: Initialising balloon driver.
                        [ 0.920100] xen-balloon: Initialising balloon driver.
                        [ 0.952068] Switching to clocksource xen
                        [ 0.967516] xen: --> pirq=16 -> irq=8 (gsi=8)
                        [ 0.967593] xen: --> pirq=17 -> irq=12 (gsi=12)
                        [ 0.967645] xen: --> pirq=18 -> irq=1 (gsi=1)
                        [ 0.967696] xen: --> pirq=19 -> irq=6 (gsi=6)
                        [ 0.967755] xen: --> pirq=20 -> irq=4 (gsi=4)
                        [ 0.967818] xen: --> pirq=21 -> irq=7 (gsi=7)
                        [ 1.025049] xen: --> pirq=22 -> irq=23 (gsi=23)
                        [ 1.228952] xen: --> pirq=23 -> irq=28 (gsi=28)
                        [ 1.571668] XENBUS: Device with no driver: device/vbd/5696
                        [ 1.573958] XENBUS: Device with no driver: device/vbd/768
                        [ 1.577690] XENBUS: Device with no driver: device/vif/0
                        [ 1.579893] XENBUS: Device with no driver: device/vif/1
                        [ 1.584074] XENBUS: Device with no driver: device/vif/2
                        [ 1.588143] XENBUS: Device with no driver: device/vif/3
                        [ 2.040651] vbd vbd-5696: 19 xenbus_dev_probe on device/vbd/5696
                        [ 2.152854] Initialising Xen virtual ethernet driver.

                        1 Reply Last reply Reply Quote 0
                        • M Offline
                          markds @olivierlambert
                          last edited by

                          @olivierlambert
                          So I copied the firewall rules script over to another linux vm (running the current debian version) and that work properly (finished in 5 seconds).

                          It would seem like all the vms that are not performing well are based on legacy versions of Debian.

                          I would have thought a kernel version of 3.2.102 would have been recent enough to get decent performance.

                          Do you know of any issues where Xen performs badly for older Debian versions?

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            I would have thought a kernel version of 3.2.102 would have been recent enough to get decent performance.

                            😬 Kernel 3.2 was out in 2012 and EOL since 2018. That's very old and not secure (I doubt anyone would have backported any security patches since then). That's 6 years worth of CVEs.

                            I don't know if PV mode would be better. It's a Debian 32 bit or 64 bit?

                            1 Reply Last reply Reply Quote 0
                            • R Offline
                              rtjdamen @markds
                              last edited by

                              @markds we have seen major improvement when booting a vm in uefi, for some reason vms created on vmware with bios are terrible slow on xcp-ng, switching to uefi made them running normal again.

                              Tristis OrisT 1 Reply Last reply Reply Quote 0
                              • Tristis OrisT Online
                                Tristis Oris Top contributor @rtjdamen
                                last edited by

                                @rtjdamen switch from bios to uefi is possible? VM can't startup after that.

                                R 1 Reply Last reply Reply Quote 0
                                • R Offline
                                  rtjdamen @Tristis Oris
                                  last edited by

                                  @Tristis-Oris u need to change the boot records for this, on windows we use a specific tool for this that converts the MBR disk to a GPT disk, i am not shure how the proces work in linux but i know that can be done there as well.

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post