RockyLinux VM's random reboots!

Sugarat

Hi all,

I am seeing random reboots on my RockyLinux VM's. In this example, I have a RockyLinux 9 VM running a number of docker containers. I was just SSH'd onto the VM and working, and BOOM.. it's gone again. Bringing up the console, I see it is rebooting. This keeps happening. The VM is not busy and it's not running out of memory.

What can I check..?
Why does it keep rebooting...?

I must get the VM stable or I'm going to have to go back to Proxmox pronto!

Many thanks.

olivierlambert

Hi,

Is your Rocky fully up to date?
Do you have any specific logs in the VM itself?
You could check xl dmesg or dmesg in the Dom0 to see if you can spot anything out of the ordinary.

Sugarat

@olivierlambert Thank you for the reply.

The VM is up to date. I wasn't aware of the xl dmesg command. I'm seeing:

(XEN) [ 5538.751053] p2m_pod_demand_populate: Dom6 out of PoD memory! (tot=1049114 ents=940768 dom6)                                                                                     
(XEN) [ 5538.751056] domain_crash called from p2m_pod_demand_populate+0x4d2/0x8b0         
(XEN) [ 5538.751057] Domain 6 (vcpu#0) crashed on cpu#3:                                                                                                                                 
(XEN) [ 5538.751059] ----[ Xen-4.17.5-4  x86_64  debug=n  Not tainted ]----               
(XEN) [ 5538.751059] CPU:    3                                                              
(XEN) [ 5538.751060] RIP:    0033:[<000000000079a336>]                                    
(XEN) [ 5538.751061] RFLAGS: 0000000000010202   CONTEXT: hvm guest (d6v0)                   
(XEN) [ 5538.751062] rax: 000000c00036e000   rbx: 0000000000400000   rcx: 0000000000294000                                                                                               
(XEN) [ 5538.751063] rdx: 00000000003d0000   rsi: 00000000003d0000   rdi: 00000000002d0000
(XEN) [ 5538.751064] rbp: 000000c00035f410   rsp: 000000c00035f318   r8:  0000000000001032                                                                                               
(XEN) [ 5538.751065] r9:  0000000000001032   r10: 00007ff8fefa9d28   r11: 0000000000000000                                                                                               
(XEN) [ 5538.751065] r12: 000000c00036e000   r13: 0000000000000000   r14: 000000c0000061c0                                                                                               
(XEN) [ 5538.751066] r15: 00ffffffffffffff   cr0: 0000000080050033   cr4: 00000000003706f0
(XEN) [ 5538.751067] cr3: 0000000139c51804   cr2: 00007ff8fdf5e0d8                        
(XEN) [ 5538.751068] fsb: 00007ff94747f740   gsb: 0000000000000000   gss: ffff9c4a6d600000
(XEN) [ 5538.751069] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 002b   cs: 0033
(XEN) [ 5538.751234] p2m_pod_demand_populate: Dom6 out of PoD memory! (tot=1049114 ents=940768 dom6)
(XEN) [ 5538.751236] domain_crash called from p2m_pod_demand_populate+0x4d2/0x8b0
(XEN) [ 5538.751420] p2m_pod_demand_populate: Dom6 out of PoD memory! (tot=1049114 ents=940768 dom6)
(XEN) [ 5538.751422] domain_crash called from p2m_pod_demand_populate+0x4d2/0x8b0

Do you know what PoD memory is? This indeed looks to be a candidate for the cause.

olivierlambert

It's populate on demand. See https://xenproject.org/blog/ballooning-rebooting-and-the-feature-youve-never-heard-of/

What's your VM configuration in terms of memory? (check the advanced tab in the "VM limits" section, then "Memory limits")

Sugarat

@olivierlambert Ah I see. Yes, I am familiar with memory ballooning from my VMWare days.

The VM is only a dual core with 4G RAM assigned, as it hosts only some reverse proxy containers (though they are essential). In the advanced tab, it shows the memory limits as:

Static: 2 GiB/7.59Gib
Dynamc: 4 GiB/4 Gib

olivierlambert

Try to set Static max to 4GiB, as the same for dynamic (4min/4max). And check if it's OK then. I think you might have not enough memory for 7.59GiB (that value is also weird, I don't know where it comes from?)

Sugarat

@olivierlambert When I go into advanced, those fields are not editable? How do I change the values?

Thanks

olivierlambert

Static field can only be edited while the VM is off.

Sugarat

@olivierlambert Ah I was clicking in slightly the wrong area. You have to be pixel perfect I think. Thanks

olivierlambert

Keep us posted on the outcome

Sugarat

@olivierlambert Will do... that VM is running a massive rsync job at the moment, but once it's finished I'll be able to shut it down.