Slow LiveMigration for VMs with large memory

[KPS]

Hi!

When I migrate VMs between hosts (with shared storage), the migration takes significantly longer compared to VMware.

Environment:

AMD EPYC 9374F

768 GB RAM

6 × 10 GbE

Observation:

Migrating a VM with 8 GB RAM takes 0:37 min.
Migrating a VM with 96 GB RAM takes around 5:20 min.

Additionally, during the final step of the migration, large VMs experience a noticeable freeze of about 8-10 seconds although they are nearly idle, which makes live pool upgrades during business hours impractical.

Is there anything, I can do to speed this up - especially the freeze?

Best wishes
KPS

[olivierlambert]

You mean 37 seconds for 8GiB, right?

The freeze at the end is a bit long but that depends on many factors. We could test an idea we discussed recently about emu-manager

Adding @TeddyAstie in the convo

[KPS]

@olivierlambert
Yes, it's 37s for the migration of a VM with 8GB RAM.
For large VMs, that is critical (5:20 min for 96GB RAM).

...but the core-issue is the freeze, as everything above 4s can leads to disconnects, etc.

[Pilow]

what do you observe as transfer speeds on your nics when live migrating the large one ?
I spin up a windows 2025 VM with 96Gb RAM, even if in-OS consumption is 2Gb/96Gb I get same time as you :

a very large transfer is occuring on the host

and we can see the 96Gb of RAM ingress on the host

any way to "only" migrate the USED ram ?
as you can see, only 2.22Gb of RAM is really used by the OS, but 96Gb is transfered...

CBT for the RAM ? ^^'

[KPS]

@Pilow
My migration speed is a bit lower than yours. I am at about 350MiB. What is your setup?

But: I think, the "migration time" is not as important, as the freeze-time... (at least for me)

[Pilow]

@KPS HPE DL360 G11 + 2x Mellanox ConnectX-6 dual ports 10/25Gb Nics, XCPNG 8.3, latest XOA.

[spelon]

I get ~510MiB on DL380 G10, 2.6ghz, 40gbe cx3, 8.3.