XCP-NG 8.3 Crash on Shutdown - HP DL360 g9

dlucas

Hi,
Wondering anybody can help with this one.

I am experiencing a hardware crash only when shutting down on this one HP DL360 G9

Crash Log:

code_text(XEN) [  744.922196] Preparing system for ACPI S5 state.
(XEN) [  744.922645] Disabling non-boot CPUs ...
(XEN) [  746.287805] 
(XEN) [  746.287808] 
(XEN) [  746.287809] NMI - I/O ERROR
(XEN) [  746.287815] ----[ Xen-4.17.5-20  x86_64  debug=n  Not tainted ]----
(XEN) [  746.287817] CPU:    0
(XEN) [  746.287820] RIP:    e008:[<ffff82d0402ff527>] __udelay+0x47/0x60
(XEN) [  746.287828] RFLAGS: 0000000000000293   CONTEXT: hypervisor
(XEN) [  746.287831] rax: 000000000971f080   rbx: 0000000000000053   rcx: 0000000000249348
(XEN) [  746.287834] rdx: 0000000000005bb6   rsi: 00005bb609653248   rdi: 00000000000003e8
(XEN) [  746.287837] rbp: 0000000000000000   rsp: ffff83083fff7d80   r8:  000000000000002f
(XEN) [  746.287839] r9:  0000000000000000   r10: ffff82d040987680   r11: 0000000000000008
(XEN) [  746.287842] r12: 000000000000000b   r13: ffff82d040455dd5   r14: ffff82d040834098
(XEN) [  746.287845] r15: ffff82d0409d5340   cr0: 000000008005003b   cr4: 00000000001526e0
(XEN) [  746.287847] cr3: 000000006aa2f000   cr2: 00007f4315aa0500
(XEN) [  746.287849] fsb: 00007f4316605740   gsb: ffff8881a4540000   gss: 0000000000000000
(XEN) [  746.287852] ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) [  746.287856] Xen code around <ffff82d0402ff527> (__udelay+0x47/0x60):
(XEN) [  746.287857]  00 f3 90 0f ae e8 0f 31 <48> c1 e2 20 89 c0 48 09 c2 48 29 f2 48 39 ca 72
(XEN) [  746.287866] Xen stack trace from rsp=ffff83083fff7d80:
(XEN) [  746.287867]    ffff82d04032a97a ffff82d040981720 000000000000000b 0000000000000000
(XEN) [  746.287871]    ffff83083fff7db0 ffff82d04020627b ffff82d0408362c0 000000000000000b
(XEN) [  746.287875]    ffff82d040820c00 0000000000000030 ffff82d04020659d 0000000000000005
(XEN) [  746.287878]    0000000000000000 ffff82d040980120 ffff82d04098006c 0000000000000005
(XEN) [  746.287882]    ffff82d040282c01 ffff82d0408362e8 ffff82d040980000 ffff831058b926e0
(XEN) [  746.287886]    ffff83083b733000 0000000000000000 0000000000000000 ffff82d0409d5340
(XEN) [  746.287889]    ffff82d040207f8c ffff82d0409803a0 ffff83083b7331c0 ffff82d040833718
(XEN) [  746.287893]    ffff82d040232e2d ffff82d0409803b0 ffff82d040833718 ffff82d040233000
(XEN) [  746.287897]    0000000000000000 0000000000000000 0000000000007fff ffff82d040820c00
(XEN) [  746.287900]    ffff82d040987680 ffff82d0403001f1 ffff82d040987680 ffff82d0409803b0
(XEN) [  746.287905]    ffff82d040300150 ffff82d040980120 ffff83083ffcb650 ffff82d0409d5340
(XEN) [  746.287909]    0000000000000000 000000ad70d252c1 ffff82d040302767 00000004a3bc6025
(XEN) [  746.287912]    0000000000000000 0000000000000000 00000004a3bc6025 ffff8881e969b568
(XEN) [  746.287915]    0000000000000568 0000000000000246 0000000000007ff0 00000000000000db
(XEN) [  746.287918]    000fffffffe00000 0000000000000000 ffffffff8100102a 0000000000000000
(XEN) [  746.287921]    0000000000000001 ffffc90043e1fd08 0000010000000000 ffffffff8100102a
(XEN) [  746.287925]    000000000000e033 0000000000000246 ffffc90043e1fcf0 000000000000e02b
(XEN) [  746.287928]    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) [  746.287930]    0000e01000000000 ffff83083ba7f000 0000000000000000 00000000001526e0
(XEN) [  746.287933]    0000000000000000 0000000000000000 0000000000000000 0000000e00000000
(XEN) [  746.287936] Xen call trace:
(XEN) [  746.287939]    [<ffff82d0402ff527>] R __udelay+0x47/0x60
(XEN) [  746.287944]    [<ffff82d04032a97a>] S __cpu_die+0x3a/0x90
(XEN) [  746.287950]    [<ffff82d04020627b>] S cpu_down+0x9b/0x1b0
(XEN) [  746.287954]    [<ffff82d04020659d>] S disable_nonboot_cpus+0xed/0x130
(XEN) [  746.287962]    [<ffff82d040282c01>] S arch/x86/acpi/power.c#enter_state_helper+0x101/0x590
(XEN) [  746.287967]    [<ffff82d040207f8c>] S common/domain.c#continue_hypercall_tasklet_handler+0x4c/0xf0
(XEN) [  746.287973]    [<ffff82d040232e2d>] S common/tasklet.c#do_tasklet_work+0x6d/0xd0
(XEN) [  746.287976]    [<ffff82d040233000>] S do_tasklet+0x50/0x90
(XEN) [  746.287979]    [<ffff82d0403001f1>] S arch/x86/domain.c#idle_loop+0xa1/0xf0
(XEN) [  746.287982]    [<ffff82d040300150>] S arch/x86/domain.c#idle_loop+0/0xf0
(XEN) [  746.287986]    [<ffff82d040302767>] S continue_running+0x57/0x60
(XEN) [  746.287987] 
(XEN) [  746.287990] 
(XEN) [  746.287991] ****************************************
(XEN) [  746.287992] Panic on CPU 0:
(XEN) [  746.287993] FATAL TRAP: vec 2, NMI[0000]
(XEN) [  746.287994] ****************************************
(XEN) [  746.287995] 
(XEN) [  746.287997] Reboot in five seconds...
(XEN) [  746.287999] Executing kexec image on cpu0
(XEN) [  746.289010] Shot down all CPUs

Latest firmware for System board and Raid controller are installed. This machine has been in use at another site for several years without problems. I recently returned it here and has begun these crashes. Previously I was running 8.2 in Legacy BIOS mode.

It only crashes when you ask for a system server shutdown which makes me wonder could this be APCI issues? Thing is my other G9 servers do not do this. The only difference I can think of between them is they are SFF and this one is LFF.

Any Suggestions?

I did wonder if this could be related as I have seen that text on the display and the red health LED comes on.

https://xcp-ng.org/forum/topic/6992/problem-install-8-2-xcp-ng-hp-dl360-gen9-p440

I will see if I can pull the ILO logs.

Thanks