host crash - guest_4.o#sh_page_fault__guest

[r1]

Woah! Something serious.. Eager.

[borzel]

@r1 hhm.....

[Steve_Sibilia]

Exact same situation here.
2 different host both crashed, I suspect is one of the vm because I've moved them from the first one that crashed to second one.
Let me know if you found something to address this issue.
Many thanks.

[r1]

@Steve_Sibilia @borzel I think you need to update hosts. https://xcp-ng.org/forum/post/9343

[Steve_Sibilia]

I already updated it.
[root@vivhv11 20190301-211627-CET]# rpm -q openvswitch
openvswitch-2.5.3-2.2.3.3.xcpng.x86_64
and it crashed afterward.

[r1]

Ok.. so its not related then.

Do you have other leads - such as crash logs or kern.log to guess?

[Steve_Sibilia]

Yes,
here: https://cloud.sinapto.net/index.php/s/13nUsnunRKmdUBC
you can find crash log. For what I can understand it is related to an host which was pv with an older version of the tools (7.2). After last crash I've update tools to the latest version hoping that this will fix the issue.
All the crashes (4 so far, on 2 different xcp hosts) happened outside peak hours for our infrastructure (friday evening after business hours and saturday during lunch) so I don't think is load average related but I'm not certain about it.
Let me know if you need further informations.
Thanks
Steve

[r1]

Guest tools should not be able to crash host - it will be treated as critical bug. Your stack is similar to that of reported earlier. And even though you have updated Xen version, the trigger could be something else...

[Danp]

FWIW, it appears that my fully patched 7.6 xcp host also rebooted the morning of 2/27

System Booted: 2019-02-27 04:46

I haven't dug into the logs yet.

[r1]

@Steve_Sibilia @Danp @borzel can you share # xenpm get-cpuidle-states and BIOS power saving settings?

[Danp]

Here the cpuidlestates.txt. Unsure how to get the BIOS settings without rebooting.

[Steve_Sibilia]

@r1 yes for sure.
Here: https://cloud.sinapto.net/index.php/s/0X8GdLxnTrw6bmd the output pf get-cpu-idle-states.
Power management is set up for maximun performance, power cap is disabled.
The system is a Dell poweredge r640.
Let me know if you need more details.
Steve

[r1]

@stormi

While digging on multiple things I'm curious on one patch

https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=9dc1e0cd81ee469d638d1962a92d9b4bd2972bfa

It seems to have been applied on

1. master
   https://xenbits.xen.org/gitweb/?p=xen.git;a=history;f=xen/arch/x86/mm/shadow/multi.c;h=7dc39d75651e0bd85fe9a4b5c6c586648d7a7ab2;hb=refs/heads/master

2. stable-4.9 https://xenbits.xen.org/gitweb/?p=xen.git;a=history;f=xen/arch/x86/mm/shadow/multi.c;h=7dc39d75651e0bd85fe9a4b5c6c586648d7a7ab2;hb=refs/heads/stable-4.9

But its not there on

3. stable-4.8 https://xenbits.xen.org/gitweb/?p=xen.git;a=history;f=xen/arch/x86/mm/shadow/multi.c;h=7dc39d75651e0bd85fe9a4b5c6c586648d7a7ab2;hb=refs/heads/stable-4.8

and

4. stable-4.7 https://xenbits.xen.org/gitweb/?p=xen.git;a=history;f=xen/arch/x86/mm/shadow/multi.c;h=7dc39d75651e0bd85fe9a4b5c6c586648d7a7ab2;hb=refs/heads/stable-4.7

And I think this could have relation of the host crash mentioned above. The function guest_walk_to_gfn is used in stable-4.7 but not at places mentioned by this specific patch.

Will try to find more on this, my tomorrow.

[Steve_Sibilia]

A quick update.
This morning we had another crash. Same host, same guest involved.

[stormi]

You can try this workaround: boot your host with the "pcid=0" parameter. This will likely have an impact on performance but should avoid the crash.

Reference: https://lists.xenproject.org/archives/html/xen-devel/2019-01/msg00006.html

[stormi]

Security advisories have just been published on http://xenbits.xen.org/xsa/

The host crash you experienced is probably related to http://xenbits.xen.org/xsa/advisory-294.html

We will provide updated Xen packages as soon as they are ready.

[Steve_Sibilia]

Thank You for the update.