Short VM freeze when migrating to another host
-
@arc1 we're using both Windows Server 2012 R2 (I know its EOL), Windows Server 2016 and Windows Server 2022 with SQL Server.
We're using Debian 11/12 for mysql and Ubuntu for PostgreSQL so quite the mix
Do you lose ping when pinging the VM while migrating?
-
@nikade Yes, we loose 4 (+-1) ping usually.
The freeze occurs on Xen Orchestra VM too which is Debian 11 (only Debian wbased VM in our enviroment). -
@arc1 Think you can give it a try with a Windows VM just to see if the problem goes away (not SQL but just pinging)? Would help diagnose if it's your infrastracture somehow or an XCP-ng specific thing with just certain Linux VMs.
I so far haven't seen behavior like this though.
-
@arc1 said in Short VM freeze when migrating to another host:
@nikade Yes, we loose 4 (+-1) ping usually.
The freeze occurs on Xen Orchestra VM too which is Debian 11 (only Debian wbased VM in our enviroment).Thats very strange, I've played around a bit at work and our VM's do not freeze when migrating between the hosts. I dont even loose a single ping.
In our setup each host has 2x10G in a LACP bond with the mgmt and all the rest of the VLAN's on top of that bond, dom0 has 16Gb ram but I dont think that matters a whole lot.What about MAC aging or similar? I mean since the VM's MAC is moved from another switch port to another the learning ttl has to be reached before the switch will know where to send the new traffic.
-
There is always short-term freeze when migrating to another host.
How short is this short-term freeze?
If it is so short that no one notices it, then no one notices it... -
I cant even notice it, tried moving the mouse around, having task manager up in a windows vm, top in a linux vm and i cant really notice a freeze when i migrated my vm's around.
Maybe it depends on how much ram the vm has? My test VM's only have 2-8Gb ram.
-
It depends on how much RAM has not yet been copied to the new VM-server at the time of the freeze.
If a test virtual machine does virtually nothing, then there are not many changes in its memory.
-
@nikade @planedrop @zmk Thank you all for answering.
We did the test with RockyLinux, Centos 7, Ubuntu 22.04 and Windows Server 2022.
On the Windows Server we only loose a few pings (10 pings in testing enviroment) on Linux we see logs about VM freeze too.
Windows VM isn't busy at all, only test VM but we loose about 10 pings.Vates support said that "depending on the load and the Ram size you can have some freeze of the VM during migration, unfortunately at the moment there is not a lot that can be done about that".
I'm just curious why @nikade and @planedrop don't get any freeze.
-
@arc1 same situation here. we also had dmesg entries when doing live-migration. but the vm did not have any issues beside that.
-
What could be the algorithm for copying the RAM of a running virtual machine to another host?
- Copy the RAM of the running VM to another host.
- While the copying was in progress, the RAM of the running VM has already changed.
- Copy the changes.
- While the copying was in progress, the RAM of the running VM has already changed.
- Copy the changes.
Finally, we understand that this is an infinite loop.
Freeze the running virtual machine.
The RAM of the non-running virtual machine no longer changes.
Copy the changes RAM of the non-running virtual machine.
After copying the changes, the RAM of the non-running VM on the old host matches the RAM of the VM on the new host.
Unfreeze the VM on the new host.The more uncopied changes at the time of freezing, the longer the freezing time.
Copying of uncopied changes after freezing cannot happen instantly.
-
@zmk We only had the dmesg entris on Xen, not on VMWare and not on HyperV
-
@arc1 how much ram/cpu/disk does your VM's have?
Seems like something is taking too long in the last phase of the migration, when the original source and destination VM are syncronized. -
The problem may be in the transfer speed between hosts.
-
@zmk yeah maybe, we're connected with 2x10G on each host to the network and while doing a migration (without storage migration) between 2 hosts in the pool I can see it spike at 6-7Gbit/s.
-
@nikade 4cpu, 16ram and roughly 200gb disk.
10ping downtime was on test enviroment with slower speeds between hosts, so this explains longer freeze.
But on production 2x25gb lacp is still noticable freeze on VMs with more sensitive software (keepalived/etcd).Nothing too terrible we were just curious if this is normal behaviour. -
@arc1 so if you go to XOA and the console of the VM, what happends then?
Is the VM frozen for the amount of 10 pings? Open taskmanager to see if there is any CPU activity. -
@nikade Yes, the MV is frozen without cpu activity.
-
@arc1 said in Short VM freeze when migrating to another host:
@nikade Yes, the MV is frozen without cpu activity.
So the VM is actually frozen in the console?
Because if it wasn't I'd suggest adjusting the mac-aging in your switches, since the VM's mac adress will be bound to the physical hosts switch-port for a period of time after migrating. -
We're seeing this issue when trying to migrate a Debian VM with 16GB of RAM.
It is a worker node in a Kubernetes cluster so it is likely that the RAM changes a fair bit. It is not uncommon for the migration to fail due to the freeze hitting a 30 second time limit.
A Windows 10 Pro VM with 16GB of RAM migrates fine, because not much is changing in the RAM I expect.
Following along for recommendations! Our hosts sound very similar to @arc1 except our network speed is slower, which is one thing we are working on.
-
And you guys aren't using any kind of dynamic memory?
Can you post a screen dump of the Advanced tab where it shows the memory configuration?We have VM's with 128Gb ram that migrates just fine, when migrating it between hosts the network shows peaks at 7,6Gbit/s and it is migrated in about ~20 seconds.
Smaller VM's with 8, 16 or even 32Gb ram is migrated almost instantly.
