OS free memory showing Static instead Dynamic

ivanobom

Hello.

I'm using XCP-ng version 8.2.1 and I have hundreds VMs running CentOS 7. I'm planing to migrate the Operating System of the VMs to Rocky Linux 8, due to EOL of CentOS 7 in a few months. However I'm facing a prohibitive issue on Rocky Linux 8 (and 9) about available memory on VM.

To explain the issue, let's assume I need a VM with 8GB of RAM, but I need to be able increase this memory without having to shutdown the VM, so I'm setting 8GB of Dynamic and 16GB of Static memory. The problem is that on CentOS 7, it shows the Dynamic amount as the total memory (8GB), but on CentOS 8 (or Rocky 8, o Rocky 9), the operating system shows the Static memory as total memory. Then the OS "thinks" that it have 16GB of memory available, but it only 8GB, causing the MV to reboot when OS tried to allocate memory over 8GB.

You guys can reproduce the problem issuing the following command at the Rocky 8 prompt:

</dev/zero head -c 15000m | tail

When doing this on CentOS 7 and the system reaches the total available RAM (8GB), then kernel is going to kill this process due to out of memory (OOM). In other hand, doing the same test on Rocky 8, the VM is going to reboot.

Free memory on CentOS 7:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7419        2059        3044         227        2315        4806
Swap:          2047           0        2047

Free memory on Rocky 8:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15738         178       15427           8         132       15327
Swap:          2047           0        2047

Note that on CentOS7 it shows about 8GB total RAM, while on Rocky 8 it shows about 16GB total RAM. Both VMs has exact the same memory configuration:
16GB Static
8GB Dynamic

Also, both VMs are using the latest guest utilities installed from EPEL repo:
xe-guest-utilities-latest-7.30.0-2.el7.x86_64 (for CentOS 7) and xe-guest-utilities-latest-7.30.0-3.el8.x86_64 (for Rocky 8).

Now, my questions are:

is it a XCP-ng bug? or
is it a xe-guest-utilities bug? or
is it a Rocky 8 (or 9) bug? or
is there anything I can do to solve it??

Thanks a lot!
Ivan Apolonio

olivierlambert

Hi,

Can you share all the numbers? static min, max and dynamic min/max from your VM (ideally in the advanced view of the VM in XO).

Note: we do not encourage people to use dynamic memory outside really specific cases.

ivanobom

This is the memory config for CentOS 7 VM

And this is the config for Rocky 8 VM

As you can see, they are exactly the same.
Thanks,

olivierlambert

Dynamic at 8/8 means you shouldn't be able to change memory while the VM is live (it's 8GiB when it runs, no more, no less).

Dynamic means when the VM runs, static can only be changed while it's halted.

ivanobom

As long Dynamic memory is Less or Equal Static, I AM ABLE to change it while VM is live to desired value (in this case, up to 16GB). The only case I have to halt the VM is to change the Static memory. So, in this case I have 16GB Static and 8GB Dynamic, I can live increase total memory up to 16GB (or reduce it) and CentOS 7 operating system recognize this memory change on the fly.

Please note that I'm just having the problem with Rocky 8, that shows 16GB of total RAM, even having limited Dynamic memory to 8GB. I'm able to decrease or increase Dynamic memory (up to 16GB set on Static), but the Rocky OS will always shows 16GB of total RAM.

olivierlambert

Ah yes sorry, I didn't read it correctly. Indeed, the range can change within the static boundaries while it's up (but not changing the static boundaries themselves).

I'm suspecting an issue with the ballooning driver of the kernel maybe

ivanobom

I just installed a new VM with original CentOS 8 (not Rocky) and it behaves exact like happens on Rocky 8 and Rocky 9. Is anything I can do to fix this supposed issue with the ballooning driver of the kernel? Or is it anything that must be resolved inside XCP-ng code?

Thanks

olivierlambert

Dunno, does it ring any bell @stormi ?

We tend avoid memory ballooning here (little benefits vs many quirks in some situation), so I never encountered those bugs myself

stormi

@olivierlambert said in OS free memory showing Static instead Dynamic:

Dunno, does it ring any bell @stormi ?

No, but it's worth trying to understand what goes on.

stormi

I raised the topic on IRC with Xen developers, and it turns out RHEL 8's kernel is built without CONFIG_XEN_BALLONING, and there's nothing we can do on hypervisor side to prevent the VM from crashing if dynamic max differs from static max in this situation.

I don't know why Red Hat disabled ballooning. You could try exposing your use case to them and also to their derivatives. Since Red Hat forced them to divert from RHEL when they stopped releasing package sources to the public, they might be more open to changing the configuration for better Xen support than when they were a bug to bug compatible rebuild.

ivanobom

Thanks for your reply.

I can see that both CentOS 7 and CentOS 8 (Rocky 8/9) have the balloon modules:

CentOS 7:

$ find /usr/lib/modules/`uname -r` -iname "*balloon*"
/usr/lib/modules/3.10.0-1160.83.1.el7.x86_64/kernel/drivers/hv/hv_balloon.ko.xz
/usr/lib/modules/3.10.0-1160.83.1.el7.x86_64/kernel/drivers/virtio/virtio_balloon.ko.xz
/usr/lib/modules/3.10.0-1160.83.1.el7.x86_64/kernel/drivers/misc/vmw_balloon.ko.xz

Rocky 8:

$ find /usr/lib/modules/`uname -r` -iname "*balloon*"
/usr/lib/modules/4.18.0-477.21.1.el8_8.x86_64/kernel/drivers/virtio/virtio_balloon.ko.xz
/usr/lib/modules/4.18.0-477.21.1.el8_8.x86_64/kernel/drivers/hv/hv_balloon.ko.xz
/usr/lib/modules/4.18.0-477.21.1.el8_8.x86_64/kernel/drivers/misc/vmw_balloon.ko.xz

Maybe it is possible to manually load this module on system boot, but I have no idea how to do it, nor which one is the correct module for Xen. Do you have any clue?

Thanks!

stormi

I don't think it's either of these modules. Ballooning support for Xen is not built at all in the kernel, according to the configuration. You need to rebuild the kernel with CONFIG_XEN_BALLONING set in its configuration, and possibly CONFIG_XEN_BALLOON_MEMORY_HOTPLUG.

olivierlambert

@stormi said in OS free memory showing Static instead Dynamic:

I don't know why Red Hat disabled ballooning.

I have one or two ideas about this...

john.c

@ivanobom I have made a request for packages to be added to the Synergy repository of Alma Linux.

Give them the time to add the packages to the repository then try the Alma Linux distribution.