<![CDATA[Short VM freeze when migrating to another host]]>Hi,
We have XCP-ng 8.2.1 hosts with latest xen orchestra.
When we migrate VMs (mostly RockyLinux 9 hosts, some CentOS 7 too) there is a mini freeze of vm. VMs with databases/etcd or any other more sensitive programs they report an error for a shot moment. VM continue to work without any issue, but still, is there any solution to that freeze?
Kind regards!

Rockylinux9 error (On CentOS 7 we get similar error):

Aug 08 13:46:38 rocky9linux kernel: Freezing user space processes ... (elapsed 0.003 seconds) done.
Aug 08 13:46:38 rocky9linux kernel: OOM killer disabled.
Aug 08 13:46:38 rocky9linux kernel: Freezing remaining freezable tasks ... (elapsed 0.006 seconds) done.
Aug 08 13:46:38 rocky9linux kernel: ------------[ cut here ]------------
Aug 08 13:46:38 rocky9linux kernel: WARNING: CPU: 1 PID: 2176896 at kernel/workqueue.c:3162 __flush_work.isra.0+0x212/0x230
Aug 08 13:46:38 rocky9linux kernel: Modules linked in: tls nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink vfat fat ppdev joydev pcspkr bochs drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt parport_pc parport i2c_piix4 drm fuse xfs libcrc32c sr_mod cdrom sg ata_generic ata_piix libata xen_netfront xen_blkfront crc32c_intel serio_raw dm_mirror dm_region_hash dm_log dm_mod
Aug 08 13:46:38 rocky9linux kernel: CPU: 1 PID: 2176896 Comm: kworker/u128:4 Kdump: loaded Tainted: G        W         -------  ---  5.14.0-362.8.1.el9_3.x86_64 #1
Aug 08 13:46:38 rocky9linux kernel: Hardware name: Xen HVM domU, BIOS 4.13 01/31/2024
Aug 08 13:46:38 rocky9linux kernel: Workqueue: events_unbound async_run_entry_fn
Aug 08 13:46:38 rocky9linux kernel: RIP: 0010:__flush_work.isra.0+0x212/0x230
Aug 08 13:46:38 rocky9linux kernel: Code: 8b 4d 00 4c 8b 45 08 89 ca 48 c1 e9 04 83 e2 08 83 e1 0f 83 ca 02 89 c8 48 0f ba 6d 00 03 e9 25 ff ff ff 0f 0b e9 4e ff ff ff <0f> 0b 45 31 ed e9 44 ff ff ff e8 df 89 b2 00 66 66 2e 0f 1f 84 00
Aug 08 13:46:38 rocky9linux kernel: RSP: 0018:ffffa2f1850afcb8 EFLAGS: 00010246
Aug 08 13:46:38 rocky9linux kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffa9b929b7
Aug 08 13:46:38 rocky9linux kernel: RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8d6487f2cb30
Aug 08 13:46:38 rocky9linux kernel: RBP: ffff8d6487f2cb30 R08: 0000000000000000 R09: ffff8d638e1021f4
Aug 08 13:46:38 rocky9linux kernel: R10: 000000000000000f R11: 000000000000000f R12: ffff8d6487f2cb30
Aug 08 13:46:38 rocky9linux kernel: R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
Aug 08 13:46:38 rocky9linux kernel: FS:  0000000000000000(0000) GS:ffff8d648a640000(0000) knlGS:0000000000000000
Aug 08 13:46:38 rocky9linux kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 08 13:46:38 rocky9linux kernel: CR2: 00007fd03d4be2a2 CR3: 000000000302c006 CR4: 00000000000206e0
Aug 08 13:46:38 rocky9linux kernel: Call Trace:
Aug 08 13:46:38 rocky9linux kernel:  <TASK>
Aug 08 13:46:38 rocky9linux kernel:  ? show_trace_log_lvl+0x1c4/0x2df
Aug 08 13:46:38 rocky9linux kernel:  ? show_trace_log_lvl+0x1c4/0x2df
Aug 08 13:46:38 rocky9linux kernel:  ? __cancel_work_timer+0x103/0x190
Aug 08 13:46:38 rocky9linux kernel:  ? __flush_work.isra.0+0x212/0x230
Aug 08 13:46:38 rocky9linux kernel:  ? __warn+0x81/0x110
Aug 08 13:46:38 rocky9linux kernel:  ? __flush_work.isra.0+0x212/0x230
Aug 08 13:46:38 rocky9linux kernel:  ? report_bug+0x10a/0x140
Aug 08 13:46:38 rocky9linux kernel:  ? handle_bug+0x3c/0x70
Aug 08 13:46:38 rocky9linux kernel:  ? exc_invalid_op+0x14/0x70
Aug 08 13:46:38 rocky9linux kernel:  ? asm_exc_invalid_op+0x16/0x20
Aug 08 13:46:38 rocky9linux kernel:  ? __flush_work.isra.0+0x212/0x230
Aug 08 13:46:38 rocky9linux kernel:  __cancel_work_timer+0x103/0x190
Aug 08 13:46:38 rocky9linux kernel:  ? set_next_entity+0xda/0x150
Aug 08 13:46:38 rocky9linux kernel:  drm_kms_helper_poll_disable+0x1e/0x40 [drm_kms_helper]
Aug 08 13:46:38 rocky9linux kernel:  drm_mode_config_helper_suspend+0x1c/0x80 [drm_kms_helper]
Aug 08 13:46:38 rocky9linux kernel:  pci_pm_freeze+0x53/0xc0
Aug 08 13:46:38 rocky9linux kernel:  ? __pfx_pci_pm_freeze+0x10/0x10
Aug 08 13:46:38 rocky9linux kernel:  dpm_run_callback+0x4c/0x140
Aug 08 13:46:38 rocky9linux kernel:  __device_suspend+0x112/0x470
Aug 08 13:46:38 rocky9linux kernel:  async_suspend+0x1b/0x90
Aug 08 13:46:38 rocky9linux kernel:  async_run_entry_fn+0x30/0x130
Aug 08 13:46:38 rocky9linux kernel:  process_one_work+0x1e5/0x3b0
Aug 08 13:46:38 rocky9linux kernel:  worker_thread+0x50/0x3a0
Aug 08 13:46:38 rocky9linux kernel:  ? __pfx_worker_thread+0x10/0x10
Aug 08 13:46:38 rocky9linux kernel:  kthread+0xe0/0x100
Aug 08 13:46:38 rocky9linux kernel:  ? __pfx_kthread+0x10/0x10
Aug 08 13:46:38 rocky9linux kernel:  ret_from_fork+0x2c/0x50
Aug 08 13:46:38 rocky9linux kernel:  </TASK>
Aug 08 13:46:38 rocky9linux kernel: ---[ end trace 18c4db6d6eef5f95 ]---
Aug 08 13:46:38 rocky9linux kernel: suspending xenstore...
Aug 08 13:46:38 rocky9linux kernel: xen:grant_table: Grant tables using version 1 layout
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=9, pirq=16
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=8, pirq=17
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=12, pirq=18
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=1, pirq=19
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=6, pirq=20
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=4, pirq=21
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=7, pirq=22
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=23, pirq=23
Aug 08 13:46:38 rocky9linux kernel: xen: --> irq=28, pirq=24
Aug 08 13:46:38 rocky9linux kernel: usb usb1: root hub lost power or was reset
Aug 08 13:46:38 rocky9linux kernel: ata2: found unknown device (class 0)
Aug 08 13:46:38 rocky9linux kernel: usb 1-2: reset full-speed USB device number 2 using uhci_hcd
Aug 08 13:46:38 rocky9linux kernel: OOM killer enabled.
Aug 08 13:46:38 rocky9linux kernel: Restarting tasks ... done.
Aug 08 13:46:38 rocky9linux NetworkManager[687]: <info>  [1723117598.8391] device (enX0): carrier: link connected
Aug 08 13:46:38 rocky9linux kernel: Setting capacity to 41943040
Aug 08 13:46:39 rocky9linux xe-daemon[669]: Trigger refresh after system resume
]]>https://xcp-ng.org/forum/topic/9488/short-vm-freeze-when-migrating-to-another-hostRSS for NodeMon, 20 Apr 2026 09:22:41 GMTThu, 08 Aug 2024 12:11:21 GMT60<![CDATA[Reply to Short VM freeze when migrating to another host on Fri, 27 Dec 2024 13:16:47 GMT]]>IIRC, just remove it (small cross) so it should use something default

]]>https://xcp-ng.org/forum/post/87386https://xcp-ng.org/forum/post/87386Fri, 27 Dec 2024 13:16:47 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Fri, 27 Dec 2024 13:08:08 GMT]]>@olivierlambert ops.. why the best topology?

]]>https://xcp-ng.org/forum/post/87384https://xcp-ng.org/forum/post/87384Fri, 27 Dec 2024 13:08:08 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Fri, 27 Dec 2024 12:47:14 GMT]]>The topology looks insane 🤔 Also, a live migration shouldn't make the VM inaccessible for more than few seconds, except if there is a LOT of memory pages changes, at a pace that is close to the transfer speed.

]]>https://xcp-ng.org/forum/post/87379https://xcp-ng.org/forum/post/87379Fri, 27 Dec 2024 12:47:14 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Fri, 27 Dec 2024 12:20:07 GMT]]>@olivierlambert live migration, the vm is very important (today, in christmas holyday, i've received some phone calls for 7 minutes of freeze..)
17407a92-730e-4e68-885f-44a4141e863d-immagine.png

]]>https://xcp-ng.org/forum/post/87378https://xcp-ng.org/forum/post/87378Fri, 27 Dec 2024 12:20:07 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Fri, 27 Dec 2024 12:17:14 GMT]]>Can you provide more details? 96/96 in dynamic? Just doing live migration or suspending the VM?

]]>https://xcp-ng.org/forum/post/87377https://xcp-ng.org/forum/post/87377Fri, 27 Dec 2024 12:17:14 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Fri, 27 Dec 2024 12:05:47 GMT]]>@olivierlambert hi, today i've upgraded my host..
The big VM frozen for ~7 minutes, is a big vm (96 gbram and 32 cpu) but 7 minutes is a very long time (for customer!)
i've setting 96/06 in dynamic: is a normal time?

]]>https://xcp-ng.org/forum/post/87375https://xcp-ng.org/forum/post/87375Fri, 27 Dec 2024 12:05:47 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 01 Oct 2024 21:07:29 GMT]]>@andrewperry yeah try set it 16/16Gb instead, it will probably do some magic 🙂

]]>https://xcp-ng.org/forum/post/83316https://xcp-ng.org/forum/post/83316Tue, 01 Oct 2024 21:07:29 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 01 Oct 2024 08:53:29 GMT]]>Yes that's very likely.

]]>https://xcp-ng.org/forum/post/83270https://xcp-ng.org/forum/post/83270Tue, 01 Oct 2024 08:53:29 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 01 Oct 2024 08:08:50 GMT]]>@nikade thanks for the ideas of where to look!

In my case we're testing and just have a 1Gb link between these hosts, which is what I was putting it down to.

This particular VM is a freshly migrated PV from Debian Xen with:

Memory limits (min/max)
Static: 16 MiB/16 GiB
Dynamic: 8 GiB/16 GiB

Could that Dynamic setting be the problem because as I recall it reduces the VM to 8 on migrate, so when doing the migrate perhaps 8 isn't enough for the VM?

I will try changing it to 16/16 and see if that has any noticeable impact. Thanks!

]]>https://xcp-ng.org/forum/post/83265https://xcp-ng.org/forum/post/83265Tue, 01 Oct 2024 08:08:50 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 01 Oct 2024 08:02:32 GMT]]>@nikade 2568c5bf-5336-4461-8f1f-60cf093f93a2-immagine.png
in VM (linux) with a free i see 94 gb of total memory

]]>https://xcp-ng.org/forum/post/83264https://xcp-ng.org/forum/post/83264Tue, 01 Oct 2024 08:02:32 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 01 Oct 2024 07:19:01 GMT]]>And you guys aren't using any kind of dynamic memory?
Can you post a screen dump of the Advanced tab where it shows the memory configuration?

We have VM's with 128Gb ram that migrates just fine, when migrating it between hosts the network shows peaks at 7,6Gbit/s and it is migrated in about ~20 seconds.
Smaller VM's with 8, 16 or even 32Gb ram is migrated almost instantly.

]]>https://xcp-ng.org/forum/post/83258https://xcp-ng.org/forum/post/83258Tue, 01 Oct 2024 07:19:01 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Mon, 30 Sep 2024 04:18:46 GMT]]>We're seeing this issue when trying to migrate a Debian VM with 16GB of RAM.

It is a worker node in a Kubernetes cluster so it is likely that the RAM changes a fair bit. It is not uncommon for the migration to fail due to the freeze hitting a 30 second time limit.

A Windows 10 Pro VM with 16GB of RAM migrates fine, because not much is changing in the RAM I expect.

Following along for recommendations! Our hosts sound very similar to @arc1 except our network speed is slower, which is one thing we are working on.

]]>https://xcp-ng.org/forum/post/83176https://xcp-ng.org/forum/post/83176Mon, 30 Sep 2024 04:18:46 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Mon, 26 Aug 2024 10:01:14 GMT]]>@arc1 said in Short VM freeze when migrating to another host:

@nikade Yes, the MV is frozen without cpu activity.

So the VM is actually frozen in the console?
Because if it wasn't I'd suggest adjusting the mac-aging in your switches, since the VM's mac adress will be bound to the physical hosts switch-port for a period of time after migrating.

]]>https://xcp-ng.org/forum/post/81915https://xcp-ng.org/forum/post/81915Mon, 26 Aug 2024 10:01:14 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Mon, 26 Aug 2024 09:08:59 GMT]]>@nikade Yes, the MV is frozen without cpu activity.

]]>https://xcp-ng.org/forum/post/81914https://xcp-ng.org/forum/post/81914Mon, 26 Aug 2024 09:08:59 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Thu, 22 Aug 2024 12:35:42 GMT]]>@arc1 so if you go to XOA and the console of the VM, what happends then?
Is the VM frozen for the amount of 10 pings? Open taskmanager to see if there is any CPU activity.

]]>https://xcp-ng.org/forum/post/81824https://xcp-ng.org/forum/post/81824Thu, 22 Aug 2024 12:35:42 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Thu, 22 Aug 2024 11:40:10 GMT]]>@nikade 4cpu, 16ram and roughly 200gb disk.
10ping downtime was on test enviroment with slower speeds between hosts, so this explains longer freeze.
But on production 2x25gb lacp is still noticable freeze on VMs with more sensitive software (keepalived/etcd).Nothing too terrible we were just curious if this is normal behaviour.

]]>https://xcp-ng.org/forum/post/81813https://xcp-ng.org/forum/post/81813Thu, 22 Aug 2024 11:40:10 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 20 Aug 2024 07:12:42 GMT]]>@zmk yeah maybe, we're connected with 2x10G on each host to the network and while doing a migration (without storage migration) between 2 hosts in the pool I can see it spike at 6-7Gbit/s.

]]>https://xcp-ng.org/forum/post/81741https://xcp-ng.org/forum/post/81741Tue, 20 Aug 2024 07:12:42 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 20 Aug 2024 06:49:31 GMT]]>The problem may be in the transfer speed between hosts.

]]>https://xcp-ng.org/forum/post/81739https://xcp-ng.org/forum/post/81739Tue, 20 Aug 2024 06:49:31 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Tue, 20 Aug 2024 06:39:31 GMT]]>@arc1 how much ram/cpu/disk does your VM's have?
Seems like something is taking too long in the last phase of the migration, when the original source and destination VM are syncronized.

]]>https://xcp-ng.org/forum/post/81738https://xcp-ng.org/forum/post/81738Tue, 20 Aug 2024 06:39:31 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Mon, 19 Aug 2024 12:25:22 GMT]]>@zmk We only had the dmesg entris on Xen, not on VMWare and not on HyperV

]]>https://xcp-ng.org/forum/post/81717https://xcp-ng.org/forum/post/81717Mon, 19 Aug 2024 12:25:22 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Mon, 19 Aug 2024 12:33:08 GMT]]>What could be the algorithm for copying the RAM of a running virtual machine to another host?

  1. Copy the RAM of the running VM to another host.
  2. While the copying was in progress, the RAM of the running VM has already changed.
  3. Copy the changes.
  4. While the copying was in progress, the RAM of the running VM has already changed.
  5. Copy the changes.

Finally, we understand that this is an infinite loop.
Freeze the running virtual machine.
The RAM of the non-running virtual machine no longer changes.
Copy the changes RAM of the non-running virtual machine.
After copying the changes, the RAM of the non-running VM on the old host matches the RAM of the VM on the new host.
Unfreeze the VM on the new host.

The more uncopied changes at the time of freezing, the longer the freezing time.

Copying of uncopied changes after freezing cannot happen instantly.

]]>https://xcp-ng.org/forum/post/81716https://xcp-ng.org/forum/post/81716Mon, 19 Aug 2024 12:33:08 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Mon, 19 Aug 2024 09:21:21 GMT]]>@arc1 same situation here. we also had dmesg entries when doing live-migration. but the vm did not have any issues beside that.

]]>https://xcp-ng.org/forum/post/81706https://xcp-ng.org/forum/post/81706Mon, 19 Aug 2024 09:21:21 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Mon, 19 Aug 2024 09:09:08 GMT]]>@nikade @planedrop @zmk Thank you all for answering.
We did the test with RockyLinux, Centos 7, Ubuntu 22.04 and Windows Server 2022.
On the Windows Server we only loose a few pings (10 pings in testing enviroment) on Linux we see logs about VM freeze too.
Windows VM isn't busy at all, only test VM but we loose about 10 pings.

Vates support said that "depending on the load and the Ram size you can have some freeze of the VM during migration, unfortunately at the moment there is not a lot that can be done about that".

I'm just curious why @nikade and @planedrop don't get any freeze.

]]>https://xcp-ng.org/forum/post/81699https://xcp-ng.org/forum/post/81699Mon, 19 Aug 2024 09:09:08 GMT<![CDATA[Reply to Short VM freeze when migrating to another host on Thu, 15 Aug 2024 05:54:00 GMT]]>It depends on how much RAM has not yet been copied to the new VM-server at the time of the freeze.

If a test virtual machine does virtually nothing, then there are not many changes in its memory.

]]>https://xcp-ng.org/forum/post/81541https://xcp-ng.org/forum/post/81541Thu, 15 Aug 2024 05:54:00 GMT