Kernel trap (??) booting TrueNAS with 2 x Kingston NVMe SSDs
-
Not sure if my issue is hardware, XCP-ng, or TrueNAS, so asking this of all 3.
I'm trying to expand a TrueNAS mirrored pool currently on a pair of SiliconPower NVMes mounted in a 4-device sled on my ZimaCube. The plan was to replace one of the SP NVMes with a bigger Kingston NVMe. Rebuild the pool, then replace the 2nd SP NVMe with another matching Kingston.
Except that every time I try and boot TrueNAS with the 2 Kingston ssds in the sled I get what appears to be a "trap or panic" and I can't go much further with the boot process.
If I leave the boot running, it eventually drops into this:
Obviously 2 matched ssds in itself is not a problem, as that's what I have running now. I've tried every combination of the 2 ssds in 4 slots, and they all fail. It also works with 2 x SP drives + 1 Kingston in the sled.
All the devices are correctly set up for passthrough to the TrueNAS VM.
Does anyone have any thoughts here.
-
Just to be sure I get it: you are passing through the NVMe devices right? If yes, have you checked https://docs.xcp-ng.org/compute/#nvme-storage-devices-on-linux ?
-
Ha, I guess some basic information might be useful.
Here's a link to the ZimaCube hardware.
I'm running XCP: 8.2.1, buiid: release/yangtze/master/58 and the TrueNAS VM ir running SCALE, version 24.04.2.2.
-
@EddieA It looks like a hardware issue. If you are using the newer KC3000/FURY you should check for a firmware update to EIFK31.7
-
@Andrew said in Kernel trap (??) booting TrueNAS with 2 x Kingston NVMe SSDs:
If you are using the newer KC3000/FURY
Nope, they are the new NV3s.
-
Further updates:
The 2 x Kingston SSDs can be passed through to other Linux based VMs (like Proxmox) without any issues.
If TrueNAS is loaded bare-metal on this server, it boots cleanly.
Ergo (to me anyway) it's an XCP/TrueNAS interaction causing the problem. The TrueNAS thread can be found here.
Cheers.
-
Just to be sure I get it: you are passing through the NVMe devices right? If yes, have you checked https://docs.xcp-ng.org/compute/#nvme-storage-devices-on-linux ?
-
@olivierlambert Yes, the devices were passed through to TrueNAS.
Would the brand of the NVMe make a difference or would this be based purely on the number of devices. I was able to passthrough 2 x SiliconPower devices without issue. It also worked with 1 x SiliconPower plus 1 x Kingston.
The failures occur when there are 2 x Kingston either in addition to any SiliconPower NVMe's or just on their own.
Cheers.
-
The answer is "can be both", depending on the NVMe brand. Have you tried to give more IRQ to see the result?
-
I wanted the previous answer before trying that, as I re-installed TrueNAS as bare-metal in the interim and didn't want to re-image back to XCP if it was purely dependent on the number of NVMe's and not the brands, as I already tested with just the 2 x Kingston.
Let me find a time I can re-image XCP and try with that parameter. Stand by.
Cheers.
-
That appears to be the solution. After adding that additional parameter, my TrueNAS SCALE system now boots cleanly, with all 4 NVME's available to it.
Cheers, and Thanks.
-
This post is deleted! -
@EddieA Glad it worked
Not all NVMe are created equals 
-
O olivierlambert marked this topic as a question on
-
O olivierlambert has marked this topic as solved on
-
@olivierlambert said in Kernel trap (??) booting TrueNAS with 2 x Kingston NVMe SSDs:
Not all NVMe are created equals
Now all I need to do is determine if these errors are from the NVME's themselves or the 4 x NVMe sled that they're inserted in. LOL.
Again, thanks for the help.
