Memory in vm half as fast after migration of vm.

Andreas

Run latest xcp-ng 8.2
Memory half as fast after migration of vm.

Have 3 exactly the same servers.
I have a VM web server that uses redis
and if i run a memory test and then migrate the VM to another server, it's half as fast.
if I migrate back to the same server, it's still half as fast.
but if I restart the VM, the speed is as it should be again.

See speed test below

#redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 227401.94 requests per second
GET: 258431.33 requests per second
LPUSH: 254291.17 requests per second
LPOP: 263192.53 requests per second

after migrating to another server

#redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 100065.05 requests per second
GET: 111944.48 requests per second
LPUSH: 114129.20 requests per second
LPOP: 112549.23 requests per second

after migrating to the first server and restarting the vm

#redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 218245.31 requests per second
GET: 239721.91 requests per second
LPUSH: 261643.11 requests per second
LPOP: 261506.28 requests per second

Wonder if anyone else has had this problem and knows the cause or has a solution.

olivierlambert

Hi,

Are you using dynamic memory? If yes, please disable it and try again.

Andreas

@olivierlambert Okay I'll try it but where do I see if it has dynamic memory
i xen Orchestra?
I have previously used xcp-ng center.

olivierlambert

Advanced view of the VM, check if static max = dynamic min = dynamic max.

Andreas

@olivierlambert no it says

Static: 512 MiB / 32 GiB
Dynamic: 1 GiB / 32 GiB

should I change to this to make it static

Static: 32 GIB / 32 GiB
Dynamic: 32 GiB / 32 GiB

because I do not see where I can delete Dynamic

Andreas

@olivierlambert
I have set it to this

Static: 512 MiB/32 GiB
Dynamic: 32 GiB/32 GiB

could not change static value

okay did a migration again
but unfortunately the same result

Before migration

#redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 223363.84 requests per second
GET: 235793.44 requests per second
LPUSH: 247954.38 requests per second
LPOP: 251572.33 requests per second

After migration

#redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 111844.31 requests per second
GET: 118119.54 requests per second
LPUSH: 116083.34 requests per second
LPOP: 116795.14 requests per second

After reboot off VM on the second server

#redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -P 16 -q
SET: 244468.91 requests per second
GET: 254647.31 requests per second
LPUSH: 261848.66 requests per second
LPOP: 261574.69 requests per second

olivierlambert

@andreas That's the correct way to do it, indeed.

Andreas

@olivierlambert Yes, but the problem remains
how can you troubleshoot this or is it in xcp-ng?

olivierlambert

You should probably start to compare XAPI record for the VM, before and after the migration, in case anything changed.

Andreas

@olivierlambert XAPI records this is a bit beyond my knowledge.
I tried to make a brand new VM Ubuntu 20.04 server
just to see so it was not the virtual machine it was wrong in but
unfortunately the same result again.

olivierlambert

xe vm-param-list uuid=<VM UUID>. Compare the output before and after migration.

Andreas

@olivierlambert
Okay find a value that is different
the test is done on the new VM which has 6 GIB in Ram
This is the value before migration
memory-target ( RO): 0
after migration
memory-target ( RO): 6442450944
and after reboot of VM
memory-target ( RO): 0

olivierlambert

That's weird, despite the fact your VM is set to static, right? (same dynamic min, max and static max?)

edit: is it the only diff?

Andreas

@olivierlambert
no this change from this
memory-actual ( RO): 6442455040
to this
memory-actual ( RO): 6442450944

and its below but they probably have no significance
start-time
console-uuids
dom-id
VCPUs-utilisation
guest-metrics-last-updated

olivierlambert

That's weird. We'll see if we can reproduce this. @Darkbeldin will try when he can (probably in January)

Andreas

@olivierlambert
Okay thanks
and happy New Year

olivierlambert

You too!

Andreas

@olivierlambert
Sorry to disturb you.
Okay just to verify that there was nothing wrong with the physical servers.
So I took 2 identical PCs and installed clean new xcp-ng 8.2
then install a virtual machine with static 4GB of memory and with guest tools.
Install redis and ran the test
then migrated VM to other pc and ran the test and the speed was half.
Took out the result before and after attached the files.

1.Before migration.txt
2.After migration.txt

Darkbeldin

@andreas Hi Andreas,

After testing it on my side i can confirm i reproduce the issue.
I will discuss it at dev level and get back to you.

Andreas

@darkbeldin
Okay Thanks