XCP-NG server crashes/reboots unexpectedly

nvs

Hi,

In the last 1.5 weeks my server seems to have rebooted itself at least two times. I noticed this because my VMs weren't running anymore. It seemed like a fresh reboot of the server. I want to figure out what the reason is and fix the issue. I started looking at logs i.e. at /var/log/kern.log and the /var/crash folder, but both file and folder are completely empty.

My conclusion on the above kern.log and crash folder being empty is that it probably wasn't XCP-NG crashing causing the reboot.

My conclusion would be that its likely power (PSU or motherboard) related. Any other logs I should check/or any other comments you may have (on my conclusions above) ?

Thanks!

nvs

I took a look at the performance graphs in XOA and the two reboots can clearly be seen. What looks interesting to me is that the server seems to have stayed offline for quite a while (when there are no data points in the graph) ? And only then came back up. Also after the 2nd reboot there seems to be a high load average, even though only one VM is on auto-start (xen orchestra) and no other VMs were started yet.

Anyone who can make something of this? It seems weird to me that whatever induces a reboot of the system would not bring it up directly again, but in fact have varying durations until XCP-NG+XOA is back up, according to the graphs. Based on this, it seems after 1st reboot it was down for ~3h. After 2nd reboot it was down for about ~23h.

Also note:

This server was running stable in this exact configuration for almost 2 years now.
I have two other pretty much identical servers that do not have this issue (same rack, same power source)

olivierlambert

Have you took a look at Xen logs to see if there's anything? Also, anything in the IPMI? (logs)

I would run a memtest on that machine.

nvs

@olivierlambert Hi, do you have some pointers which exact files to check for those?

I've looked at:

/var/log/xensource.log, but that log seems to have started earlier today. I don't see entries back to when the reboots happened.
Regarding IPMI logs: This machine is a Ryzen 9 5950X on a Asus Prime X570 Pro motherboard. It doesn't have IPMI unfortunately.

I will look into doing a memtest as you suggested.

olivierlambert

Check the /var/log/xen* stuff (on top of my head, @stormi probably knows that better than me)

nvs

@olivierlambert I looked through the xensource.log.1/2/3 etc files.

What sticks out to me is that there is a gap here:

xensource.log.2's last four lines:
Nov 19 16:43:27 xcp-ng xapi: [debug||3064510 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:SR.scan D:*** created by task D:***
Nov 19 16:43:27 xcp-ng xapi: [ info||3064512 /var/lib/xcp/xapi||taskhelper] task SR.scan R:*** (uuid:***) created (trackid=***) by task D:***
Nov 19 16:43:27 xcp-ng xapi: [debug||3064512 /var/lib/xcp/xapi|SR.scan R:***|message_forwarding] SR.scan: SR = '*** (20TB HDD)'
Nov 19 16:51:30 xcp-ng xapi: [debug||3069218 /var/lib/xcp/xapi|session.slave_local_login_with_password D:***|xapi_session] Add session to local storage

Then the next four lines are in a new xensource.log.1 file, but notice a 49 minute gap until then:

xensource.log.1's first four lines:
Nov 19 17:40:08 xcp-ng xenopsd-xc: [debug||5 ||xenops_server] Received an event on managed VM ***
Nov 19 17:40:08 xcp-ng xcp-rrdd: [ info||9 ||rrdd_main] memfree has changed to 4191660 in domain 9
Nov 19 17:40:08 xcp-ng xenopsd-xc: [debug||5 |queue|xenops_server] Queue.push ["VM_check_state","***"] onto ***:[  ]
Nov 19 17:40:08 xcp-ng xenopsd-xc: [debug||40 ||xenops_server] Queue.pop returned ["VM_check_state","***"]

I've redacted some UUID's with ***, probably wasn't needed but just in case.

From the earlier graphs I expected to not see any log here (assuming the machine was off or whatever), but it seems it was actually running most of the time. The above 49min gap seems to be the only longer gap I can spot at first sight in the last days log. Strange because in XOA the graph shows as if the host was down for like 23h or so. Any thoughts?

stormi

@olivierlambert said in XCP-NG server crashes/reboots unexpectedly:

Check the /var/log/xen* stuff (on top of my head, @stormi probably knows that better than me)

I think this would be the output of xl dmesg.

Also, if the kern.log file is empty, this means that there were no messages for this day, but older messages may be visible in kern.log.1, kern.log.2.gz, etc.

nvs

@stormi Thanks. I've gone through the kern.log.1/2/3 etc and I can see when the server seems to have rebooted and comes back up again, but there doesnt seem to be anything logged just before it quits.

nvs

Regarding memory test: Just running the normal mem test from Grub should do, I guess?