What should i expect from VM migration performance from Xen-ng ?
-
Hi Everyone ,
When live migrate vms between Xen-ng nodes i saw that couldn't push more then 1 Gbit traffic.
We have 3 nodes and all of them are connected 10Gbit to the switch (bond is configured but only on physical port up for now ) (bond is switch independent configured )
Then we back to Xen-ng hosts install iperf3 and made a test between nodes and get almost 8 Gbit traffic but do not care what happened to other 2 Gbit (maybe because no dpdk enabled default -->
[00:04 xcp-ng-zjiihqws ~]# ovs-vsctl get Open_vSwitch . iface_types
[geneve, gre, internal, ipsec_gre, lisp, patch, stt, system, tap, vxlan])Using MTU 9000 on switch and Xen-ng side (actually 1500 mtu should be enough but we try to apple best practice) bonding 9000 and sub interface migration also same 9000 too.
Then tried 2-3 VM together to understand how did something in paralel and up to 1.24GB traffic we succeeded.
Actually i do not want push 10G or something just want to understand something wrong with me ? or really live migration traffic could not saturate my nic bandwith ? Should check some config to allow high traffic for migration ?
Thanks, Regards
VM -
Are you doing storage/vdi migration?
The speed of that is "limited" some how, I dont know why but migrating VDI's is always slow.If we migrate 3-4 VM's at the same time we could see it spike to a total of 166MB/s but that's about it.
I've even created a report on the Citrix Xenserver bugtracker but they never bothered to do anything about it. -
@nikade hi , no storage migration , only move one vm from one node to another
this is what exactly i saw, almost seeing 1.24 or 1.3 Gbit always which equal 166MB/s -
Live memory migration (without storage) can reach far higher speed. We had some numbers I remember about close to 8Gbits.
-
@olivierlambert hi,
This iperf3 performance between two nodes 141 and 143 via migration network which is 10.111.178 network
I'm sure MTUs are 9000 on switch , nics and related sub interface for migration network work ...
3 VM migrations are kicked
No luck to catch high throughputBetween two vms mtu 1500, i got this
Means couldn't imagine what is the issue here ? Maybe Dom0 limiter for VMs but for migration process nothing should be , isn't it ?
Thanks
VM -
When I live migrate a VM without disk (VDI on shared SR) we're seeing ~4Gbit/s with 2x10G in a bond0 with MTU 9000, within the same broadcast domain/subnet:
We've seen around 9Gbit/s max but thats when we migrate a large VM with a lot of memory.
-
increasing memory of Dom0 did not effect ...
increasing vCPU of Dom0 did not effect (actually not all vcpu already used for it but i just want to try )
I run stress*ng for load vms memory but did not effect
No pinning or numa config need because single cpu and shared L3 cache for all cores
Also MTU size is not effecting its working same with 1500 and 9000 MTU
I saw and change tcp_limit_output_bytes but did not help meOnly what effect is changing the hardware
My Intel servers are Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz --> 0.9 Gbit/s per migration
My AMD servers are AMD EPYC 7502P 32-Core Processor --> 1.76 Gbit/s per migrationDo you have any advise ?