@olivierlambert
So I copied the firewall rules script over to another linux vm (running the current debian version) and that work properly (finished in 5 seconds).
It would seem like all the vms that are not performing well are based on legacy versions of Debian.
I would have thought a kernel version of 3.2.102 would have been recent enough to get decent performance.
Do you know of any issues where Xen performs badly for older Debian versions?
@Danp
I should also correct my previous answer...
Looks like some of the dmesg are in a different case...
dmesg | grep -i xen
[ 0.000000] DMI: Xen HVM domU, BIOS 4.13 04/11/2024
[ 0.000000] Hypervisor detected: Xen HVM
[ 0.000000] Xen version 4.13.
[ 0.000000] Xen Platform PCI: I/O protocol version 1
[ 0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[ 0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
[ 0.000000] ACPI: RSDP 0x00000000000EA020 00024 (v02 Xen)
[ 0.000000] ACPI: XSDT 0x00000000FC00A7C0 00044 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: FACP 0x00000000FC00A370 000F4 (v04 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: DSDT 0x00000000FC001040 092A3 (v02 Xen HVM 00000000 INTL 20160527)
[ 0.000000] ACPI: APIC 0x00000000FC00A470 00260 (v02 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: HPET 0x00000000FC00A750 00038 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] ACPI: WAET 0x00000000FC00A790 00028 (v01 Xen HVM 00000000 HVML 00000000)
[ 0.000000] Booting paravirtualized kernel on Xen HVM
[ 0.000000] Xen HVM callback vector for event delivery is enabled
[ 0.113732] Xen: using vcpuop timer interface
[ 0.113741] installing Xen timer for CPU 0
[ 0.390521] installing Xen timer for CPU 1
[ 0.492037] installing Xen timer for CPU 2
[ 0.590329] installing Xen timer for CPU 3
[ 0.918283] xen/balloon: Initialising balloon driver.
[ 0.920100] xen-balloon: Initialising balloon driver.
[ 0.952068] Switching to clocksource xen
[ 0.967516] xen: --> pirq=16 -> irq=8 (gsi=8)
[ 0.967593] xen: --> pirq=17 -> irq=12 (gsi=12)
[ 0.967645] xen: --> pirq=18 -> irq=1 (gsi=1)
[ 0.967696] xen: --> pirq=19 -> irq=6 (gsi=6)
[ 0.967755] xen: --> pirq=20 -> irq=4 (gsi=4)
[ 0.967818] xen: --> pirq=21 -> irq=7 (gsi=7)
[ 1.025049] xen: --> pirq=22 -> irq=23 (gsi=23)
[ 1.228952] xen: --> pirq=23 -> irq=28 (gsi=28)
[ 1.571668] XENBUS: Device with no driver: device/vbd/5696
[ 1.573958] XENBUS: Device with no driver: device/vbd/768
[ 1.577690] XENBUS: Device with no driver: device/vif/0
[ 1.579893] XENBUS: Device with no driver: device/vif/1
[ 1.584074] XENBUS: Device with no driver: device/vif/2
[ 1.588143] XENBUS: Device with no driver: device/vif/3
[ 2.040651] vbd vbd-5696: 19 xenbus_dev_probe on device/vbd/5696
[ 2.152854] Initialising Xen virtual ethernet driver.
Actually maybe its a red herring...
ethtool -i eth3
driver: vif
version:
firmware-version:
bus-info: vif-2
Also I tested using iperf:
vm <-> vm: 9.78 Gbits/sec (on the same xcp-ng host)
vm <-> exsi: 0.97 Gbits/sec (network is 1GB)
As a final test I ran:
rmmod xen_netfront
and as expected the vms lost network access
So maybe the issue is elsewhere
[ 0.824078] xen/balloon: Initialising balloon driver.
[ 0.825817] xen-balloon: Initialising balloon driver.
[ 0.848060] Switching to clocksource xen
[ 0.861416] xen: --> pirq=16 -> irq=8 (gsi=8)
[ 0.861486] xen: --> pirq=17 -> irq=12 (gsi=12)
[ 0.861531] xen: --> pirq=18 -> irq=1 (gsi=1)
[ 0.861580] xen: --> pirq=19 -> irq=6 (gsi=6)
[ 0.861632] xen: --> pirq=20 -> irq=4 (gsi=4)
[ 0.861692] xen: --> pirq=21 -> irq=7 (gsi=7)
[ 0.916200] xen: --> pirq=22 -> irq=23 (gsi=23)
[ 1.135771] xen: --> pirq=23 -> irq=28 (gsi=28)
[ 1.936447] vbd vbd-5696: 19 xenbus_dev_probe on device/vbd/5696
No luck... still
how do I confirm that the xen network drivers are in use:
lsmod | grep xen
xenfs 17639 1
xen_netfront 22032 0
xen_blkfront 17478 5
does that not indicate that whilst xen_netfront module is loaded, no device is using that module?
Booting via BIOS mode (both in VMWare and XCP-NG)
Used XO's vmware import tool
(from memory I believe the VM was powered off and then imported vs taking a snapshot then importing, but can not be sure)
Have not tried creating a brand new VM and attaching the disk. Do you suspect the import has copied over some strange config impacting performance?
BTW thanks for the prompt response @Danp
HI,
We are currently testing the suitability of XCP-NG for being a replacement of VMWare pool. I am sure I are not alone here, given VMWare recent policy changes.
I have converted one of the VMWare hosts to XCP-NG 8.2 and have notice the test Linux VM is performing dramatically slower.
Loading our large iptables rule set takes 15sec on VMWare vs 140 sec on XCP-NG
The average system load (reported via munin) is roughly 2x higher under XCP-NG
Via terminal I have notice large spikes in load (as high as 9) (using uptime)
The average CPU usage seems 2x higher (increase is in system not user space)
Noticed a few instances of packet loss might this might be due to the spikes in system load.
The kernel of the guest vm is 3.2.102
The block/network modules are loaded (xen_netfront / xen_blkfront) in the guest vm
Management agent 7.30.0-12 is loaded in the guest vm
XOA correctly reports that the guest is virtualised via PVHVM
There doesn't appear to be any vmware modules / agents running
The host is only running this single VM (previously under VMWare there were 4)
The host reports a load average of 0.4 and 18% cpu usage (via XOA)
Hypertheading is disabled
No hardware faults reported via iDrac
So far I have attempted to:
Any ideas why under XCP-NG the system is running so poorly?