XCP-ng 7.5 - MegaRAID SAS 9240-8i hang/reboot issue.

mpyusko

I am experiencing an issue I believe is directly related to Kernel/Driver compatibility. It is 100% repeatable by simply running:

"lspci -vv -s 05:00.0"

Note: -v does not trigger the issue.

As you'll see below, I updated the megaraid_sas driver to a newer version. I'm inclined to believe the issue lies more with the kernel. Since I do not experience this via a baremetal boot to Kali, the issue resides somewhere inside XCP-ng. Is there a "supported" method to upgrade the kernel to 4.14 or 4.15? (This issue existed in 7.4 as well.)

Dell R710

Dual Xeon L5620
48 GB DDR3 RAM @1333MHz
LSI MegaRAID SAS 9240-8i (HBA)
(Perc 6/i does not support over 2TB and was retired)
4x WD NAS Red 3TB (RAID 5 via HBA)
2x Toshiba P300 1TB (RAID1 via HBA)
1x Sandisk Ultra USB 3.0 64GB (XCP-ng 7.5 OS)

Running Kail 2018-2 Rolling, lspci -vv completes properly without issue.
Running XCP-ng 7.5, lspci -vv freezes and reboots
SEL output:

*Thu Aug 16 2018 19:19:44 A bus fatal error was detected on a component at bus 0 device 6 function 0.  
 Critical  0.000130
Thu Aug 16 2018 19:19:44 A bus fatal error was detected on a component at slot 1.*

KALI

*05:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2008 [Falcon] (rev 03)
        Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9240-8i
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 36
        Region 0: I/O ports at fc00 [size=256]
        Region 1: Memory at d0480000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at d0000000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at d0040000 [disabled] [size=256K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [d0] Vital Product Data
                Not readable
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
                Vector table: BAR=1 offset=00002000
                PBA: BAR=1 offset=00003800
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [138 v1] Power Budgeting <?>
        Capabilities: [150 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: 0073
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000d0484000 (64-bit, non-prefetchable)
                Region 2: Memory at 00000000d0080000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [190 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Kernel driver in use: megaraid_sas
        Kernel modules: megaraid_sas*

***** megaraid_sas Version Info *****
version:        07.703.05.00-rc1
srcversion:     5121FF8D56A8481586A5CB9
vermagic:       4.15.0-kali2-amd64 SMP mod_unload modversions

XCP-ng

***** megaraid_sas Version Info *****
version:        07.701.18.00-rc1
srcversion:     550B32DFFACE241631510C5
vermagic:       4.4.0+10 SMP mod_unload modversions

Following: https://support.citrix.com/article/CTX235759

        Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9240-8i
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 35
        Region 0: I/O ports at fc00 [size=256]
        Region 1: Memory at df1bc000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at df1c0000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at df100000 [disabled] [size=256K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [d0] Vital Product Data*

***** megaraid_sas Version Info *****
version:        07.706.03.00
srcversion:     41BA0F8DAEFE4CB4AC86A9C
vermagic:       4.4.0+10 SMP mod_unload modversions

olivierlambert

Please use the Markdown syntax for code block
It's not trivial nor support to add a more recent kernel
Did you installed the RPM? It seems yes. Without installing it, what's the behavior?
Same issue with XenServer 7.5?

mpyusko

Thank you for your response.

I do not see a "code /code" identifier option. I tried to format it as best I could.
I understand, however ZFS and other features are in a beta or testing mode, so I was wondering if there is documentation the Dev team is using to test an updated kernel. (Some linux dstros give options for CR and LTS kernels). Changing Kernels seems to be the most logical choice to compare.
I was having the issues prior to installing the megaraid_sas updated module and I experienced the same issue following. I used the directions found here as it is a supported fix.
I experienced the issue with XCP-ng 7.4, I was hoping 7.5 might fix it. It did not so I updated the megaraid_sas module. I have not tried installing Citrix Xenserver 7.5. I would be neutering my installations to do so. However as a troubleshooting step I could probably find a Flash drive laying around to test with if you feel there is value.

olivierlambert

Markdown is a well spread writing format. For code block, please read this: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet#code-and-syntax-highlighting Note: I edited your post accordingly.
Maybe @r1 can help
Yes, this could be interesting.

r1

@olivierlambert thanks.
@mpyusko Changing kernel on XCP/Xenserver is not as easy as any other Linux distro. Especially the dependency of blktap need a rebuild.

If you are interested in getting hands dirty, try patch from there to any upstream kernel. You can enable/disable drivers/FS from kernel and build one that can boot Dom0 on XCP/Xenserver. This will compile the blktap backend, which is necessary to run VHDs.

You may also need usermod blktap packages. There are dedicated threads on this "Dev" Category for that. Take a look.

All the best.

I seriously want to have XCP - latest easily customizable kernel of choice, its a dream far away though.

mpyusko

@r1 Any Kernel upgrade is a pain, regardless how seamless some distros try to make it. I've been using XS for years and it's always been rock-solid in my production environment. This is the first time I've ever experienced a critical issue. Unfortunately it translates to an unstable system, so there is no telling when the machine is going to suffer a "critical-error" and hang at reboot.

My low-level familiarity win XS/XCP-ng does not extend to the depth where I would attempt a self-guided kernel upgrade, however if there is a development fork where they are testing the 4.14 or 4.15 kernel, I would be inclined to evaluate it.

mpyusko

@olivierlambert said in XCP-ng 7.5 - MegaRAID SAS 9240-8i hang/reboot issue.:

Same issue with XenServer 7.5?

YES! (Just got around to testing it.)

r1

Let us see if a newer kernel would help. There is also an option of back porting the newer driver to older kernel with possible code changes. Both will be experimental though!

mpyusko

@r1

Same thing happens in XS 7.6 "upgrading" from XS 7.5. Interenstingly enough post upgrade, the upper-left corner says Xenserver 7.5 but the stats field and Xencenter report 7.6.
Same thing happens in a clean installation of XS 7.6 too.

r1

@mpyusko Let me see if we can build driver 07.703.05.00-rc1 for your XCP-NG 7.5/6, will let you know if it becomes available.

r1

@mpyusko Please get the driver from link and
[root@xcp-ng-rjv ~]# yum install megaraid_sas-07.703.05.00-1.x86_64.rpm
[root@xcp-ng-rjv ~]# rmmod megaraid_sas
[root@xcp-ng-rjv ~]# modprobe megaraid_sas

Then check for your lspci.

// Additional info

[root@xcp-ng-rjv ~]# modinfo /usr/lib/modules/4.4.0+10/weak-updates/megaraid_sas/megaraid_sas.ko
filename:       /usr/lib/modules/4.4.0+10/weak-updates/megaraid_sas/megaraid_sas.ko
description:    Avago MegaRAID SAS Driver
author:         megaraidlinux.pdl@avagotech.com
version:        07.703.05.00
license:        GPL
srcversion:     2A8AB66F9A16F0542FC2173

mpyusko

For the record...

@mpyusko said in XCP-ng 7.5 - MegaRAID SAS 9240-8i hang/reboot issue.:

@r1

Same thing happens in XS 7.6 "upgrading" from XS 7.5. Interenstingly enough post upgrade, the upper-left corner says Xenserver 7.5 but the stats field and Xencenter report 7.6.
Same thing happens in a clean installation of XS 7.6 too.

XS 7.5

***** megaraid_sas Version Info *****
version:        07.701.18.00-rc1
srcversion:     550B32DFFACE241631510C5
vermagic:       4.4.0+10 SMP mod_unload modversions

XS 7.6

***** megaraid_sas Version Info *****
version:        07.701.18.00-rc1
srcversion:     550B32DFFACE241631510C5
vermagic:       4.4.0+10 SMP mod_unload modversions

r1

@r1 said in XCP-ng 7.5 - MegaRAID SAS 9240-8i hang/reboot issue.:

get the driver from link and

Try with this driver.

mpyusko

@r1 said in XCP-ng 7.5 - MegaRAID SAS 9240-8i hang/reboot issue.:

@mpyusko Please get the driver from link and
[root@xcp-ng-rjv ~]# yum install megaraid_sas-07.703.05.00-1.x86_64.rpm
[root@xcp-ng-rjv ~]# rmmod megaraid_sas
[root@xcp-ng-rjv ~]# modprobe megaraid_sas

I did what you requested....

[root@vincent Downloads]# yum install megaraid_sas-07.703.05.00-1.x86_64.rpm
Loaded plugins: fastestmirror
Cannot open: megaraid_sas-07.703.05.00-1.x86_64.rpm. Skipping.
Error: Nothing to do
[root@vincent Downloads]# rpm -Uhv megaraid_sas-07.703.05.00-1.x86_64.rpm                                         
error: megaraid_sas-07.703.05.00-1.x86_64.rpm: not an rpm package (or package manifest):
[root@vincent Downloads]#

r1

Can you post #ls -lh and md5sum output of it?

mpyusko

@r1 said in XCP-ng 7.5 - MegaRAID SAS 9240-8i hang/reboot issue.:

Can you post #ls -lh and md5sum output of it?

-rw-r--r-- 1 root root  40K Oct  5 13:28 megaraid_sas-07.703.05.00-1.x86_64.rpm
e1e232eab5d90308144bf3c47665cedd  megaraid_sas-07.703.05.00-1.x86_64.rpm

r1

You seem to have downloaded something wrong.

my output is

[root@xcp-ng-rjv ~]# wget "https://github.com/rushikeshjadhav/MegaRAID-SAS-07.703.05.00/raw/master/megaraid_sas-07.703.05.00-1.x86_64.rpm"
[root@xcp-ng-rjv ~]# ls -lh megaraid_sas-07.703.05.00-1.x86_64.rpm 
-rw-r--r-- 1 root root 388K Oct  4 21:26 megaraid_sas-07.703.05.00-1.x86_64.rpm
[root@xcp-ng-rjv ~]# md5sum megaraid_sas-07.703.05.00-1.x86_64.rpm 
ef3064607545e0d390445f9e82ab8930  megaraid_sas-07.703.05.00-1.x86_64.rpm

r1

@mpyusko Did you happen to check this?

mpyusko

Just got to it again....

***** ahci Version Info *****
version:        3.0
srcversion:     35F0A9078B4BB938E54A1E7
vermagic:       4.4.0+10 SMP mod_unload modversions


***** megaraid_sas Version Info *****
version:        07.703.05.00
srcversion:     2A8AB66F9A16F0542FC2173
vermagic:       4.4.0+10 SMP mod_unload modversions

lspci -v output

[root@vincent nfs]# lspci -v -s 07:00.0
07:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2008 [Falcon] (rev 03)
        Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9240-8i
        Flags: bus master, fast devsel, latency 0, IRQ 40
        I/O ports at ec00 [size=256]
        Memory at df2bc000 (64-bit, non-prefetchable) [size=16K]
        Memory at df2c0000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at df200000 [disabled] [size=256K]
        Capabilities: [50] Power Management version 3
        Capabilities: [68] Express Endpoint, MSI 00
        Capabilities: [d0] Vital Product Data
        Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [138] Power Budgeting <?>
        Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: megaraid_sas

[root@vincent nfs]#

lspci -vv output

[root@vincent nfs]# lspci -vv -s 07:00.0
07:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2008 [Falcon] (rev 03)
        Subsystem: LSI Logic / Symbios Logic MegaRAID SAS 9240-8i
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 40
        Region 0: I/O ports at ec00 [size=256]
        Region 1: Memory at df2bc000 (64-bit, non-prefetchable) [size=16K]
        Region 3: Memory at df2c0000 (64-bit, non-prefetchable) [size=256K]
        Expansion ROM at df200000 [disabled] [size=256K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range BC, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-

And then same result. Ugh.

r1

@mpyusko If I understood correctly, lspci -vv -s 07:00.0 is crashing the host? Even on megaraid_sas version 07.703.05.00. But Kali linux host does not crash on same megaraid_sas version.

To resolve this, Do you have console access to the host? or remote KVM?

I would suggest you to boot your host in "XCP-ng in Safe Mode", this menu comes up when you start to boot the host. Instead of default "XCP-ng" choose "XCP-ng in Safe Mode".

This will allow us to see the messages generated in kern.log or onscreen about the crash and would point it right to the problem.

Meanwhile if you have some stack trace logs in kern.log, please share those.