Live Migration on different CPU

abufrejoval

oVirt/KVM allow assigning a base CPU architecture or lowest common denominator to a cluster to ensure that VMs can be live migrated without issues between members. That does help quite a bit in scenarios where you can't just chose your oldest host to launch any VM that may have to be migrated automatically due to load and HA policies.

I've tried finding some info on heterogeneous CPU live migration in Xen, but only found really old information, which put most of the burden on the host, which should support "downgrading CPUs" in the BIOS: something I haven't seen for many years.

On oVirt this becomes quite complicated with Spectre/Meltdown vulnerabilities, distinct levels of hardware compensation and the presence of µ-code patches and the nature of the issue would be similar on Xen.

So a pointer to a) the pertinent documentation and b) some information on the most likely sources of trouble would be appreciated.

Currently you only get that one warning, which is the very same in all cases, even in scenarios, which should be safe (migrating a VM booted in an older CPU to a newer one).

My current hardware pools are:
a) Gemini Lake Pentiums/Atoms J5005, which on KVM need to be configured at Westmere or even Nehalem level to be compatible with all other Intels
b) Haswell Xeons (4 and 18 core)
c) Broadwell Xeon-D (8 core)
d) "Skylakes" NUC8 and NUC10
e) Tiger Lake NUC11

I'll most likely split between a) and b-e), but on oVirt I've actually had a) and c) in the same pool with the Cluster configured with Nehalem as lowest common denominator.

I definitely want to operate d) and e) in the same cluster, but actually if things like control flow integrity (CFI + shadow stacks) or encrypted VMs were to become supported with Xen, this would cause a hard split right there, unless it could be controlled.

I also have some Zen3 based hosts, but I understand that live migration for those aren't supported to Intel CPUs...

Yet even there I guess you could define a compatible base line, because you can even run a Hackintosh on a Ryzen CPU inside VMware, which can be instructed to fake control registers to enable that.

While Hackintosh VMs aren't the most urgent use case for Xcp-ng, a somewhat more modern shared baseline near Haswell would seem useful as AMD is regaining market share in EPYCly big and APU small servers.

olivierlambert

A pool will do that automatically for you: reducing the CPU capabilities to the oldest CPU in the pool. So you can live migrate easily.

That might work despite your very different CPU, please report

josuemotte

@olivierlambert is there a way to disable the warning from XOA ?

olivierlambert

@josuemotte why? Can you explain the use case?

josuemotte

@olivierlambert I had this message each time I needed to migrate some VMs however changing the pool master did the trick : https://support.citrix.com/article/CTX216127/how-to-change-the-xenserver-pool-master-role-from-xenserver-console , it disabled some instructions which were not available on another CPU

olivierlambert

CPU leveling is done dynamically at each host when it boots in a pool (or when joining a pool).

mochouinard

@olivierlambert Anyway to do it without them being part of the same pool ? Use case would be 2 different remote location, where I have 3 server at each site inside their own pool, but one of them have newer CPU. So I want to force one of the pool at one location to be the same as another pool.

olivierlambert

I think you can but I don't remember the commands. Let me ask around.

mochouinard

@olivierlambert By any chance, have you seen any info into how to do it ? I saw there were a way before the change to add dynamic cpu leveling, but I can't find any way to do it now.

olivierlambert

It's called CPU masking but I don't remember the exact commands.

You might take a look at xe host-set-cpu-features if it still works.