RockyLinux VM's random reboots!
-
Hi all,
I am seeing random reboots on my RockyLinux VM's. In this example, I have a RockyLinux 9 VM running a number of docker containers. I was just SSH'd onto the VM and working, and BOOM.. it's gone again. Bringing up the console, I see it is rebooting. This keeps happening. The VM is not busy and it's not running out of memory.
What can I check..?
Why does it keep rebooting...?I must get the VM stable or I'm going to have to go back to Proxmox pronto!
Many thanks.
-
Hi,
- Is your Rocky fully up to date?
- Do you have any specific logs in the VM itself?
- You could check
xl dmesgordmesgin the Dom0 to see if you can spot anything out of the ordinary.
-
@olivierlambert Thank you for the reply.
The VM is up to date. I wasn't aware of the xl dmesg command. I'm seeing:
(XEN) [ 5538.751053] p2m_pod_demand_populate: Dom6 out of PoD memory! (tot=1049114 ents=940768 dom6) (XEN) [ 5538.751056] domain_crash called from p2m_pod_demand_populate+0x4d2/0x8b0 (XEN) [ 5538.751057] Domain 6 (vcpu#0) crashed on cpu#3: (XEN) [ 5538.751059] ----[ Xen-4.17.5-4 x86_64 debug=n Not tainted ]---- (XEN) [ 5538.751059] CPU: 3 (XEN) [ 5538.751060] RIP: 0033:[<000000000079a336>] (XEN) [ 5538.751061] RFLAGS: 0000000000010202 CONTEXT: hvm guest (d6v0) (XEN) [ 5538.751062] rax: 000000c00036e000 rbx: 0000000000400000 rcx: 0000000000294000 (XEN) [ 5538.751063] rdx: 00000000003d0000 rsi: 00000000003d0000 rdi: 00000000002d0000 (XEN) [ 5538.751064] rbp: 000000c00035f410 rsp: 000000c00035f318 r8: 0000000000001032 (XEN) [ 5538.751065] r9: 0000000000001032 r10: 00007ff8fefa9d28 r11: 0000000000000000 (XEN) [ 5538.751065] r12: 000000c00036e000 r13: 0000000000000000 r14: 000000c0000061c0 (XEN) [ 5538.751066] r15: 00ffffffffffffff cr0: 0000000080050033 cr4: 00000000003706f0 (XEN) [ 5538.751067] cr3: 0000000139c51804 cr2: 00007ff8fdf5e0d8 (XEN) [ 5538.751068] fsb: 00007ff94747f740 gsb: 0000000000000000 gss: ffff9c4a6d600000 (XEN) [ 5538.751069] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 002b cs: 0033 (XEN) [ 5538.751234] p2m_pod_demand_populate: Dom6 out of PoD memory! (tot=1049114 ents=940768 dom6) (XEN) [ 5538.751236] domain_crash called from p2m_pod_demand_populate+0x4d2/0x8b0 (XEN) [ 5538.751420] p2m_pod_demand_populate: Dom6 out of PoD memory! (tot=1049114 ents=940768 dom6) (XEN) [ 5538.751422] domain_crash called from p2m_pod_demand_populate+0x4d2/0x8b0Do you know what PoD memory is? This indeed looks to be a candidate for the cause.
-
It's populate on demand. See https://xenproject.org/blog/ballooning-rebooting-and-the-feature-youve-never-heard-of/
What's your VM configuration in terms of memory? (check the advanced tab in the "VM limits" section, then "Memory limits")
-
@olivierlambert Ah I see. Yes, I am familiar with memory ballooning from my VMWare days.
The VM is only a dual core with 4G RAM assigned, as it hosts only some reverse proxy containers (though they are essential). In the advanced tab, it shows the memory limits as:
Static: 2 GiB/7.59Gib
Dynamc: 4 GiB/4 Gib -
Try to set Static max to 4GiB, as the same for dynamic (4min/4max). And check if it's OK then. I think you might have not enough memory for 7.59GiB (that value is also weird, I don't know where it comes from?)
-
@olivierlambert When I go into advanced, those fields are not editable? How do I change the values?
Thanks
-
Static field can only be edited while the VM is off.
-
@olivierlambert Ah I was clicking in slightly the wrong area. You have to be pixel perfect I think. Thanks
-
Keep us posted on the outcome
-
@olivierlambert Will do... that VM is running a massive rsync job at the moment, but once it's finished I'll be able to shut it down.
