Kernel trap (??) booting TrueNAS with 2 x Kingston NVMe SSDs

EddieA

Not sure if my issue is hardware, XCP-ng, or TrueNAS, so asking this of all 3.

I'm trying to expand a TrueNAS mirrored pool currently on a pair of SiliconPower NVMes mounted in a 4-device sled on my ZimaCube. The plan was to replace one of the SP NVMes with a bigger Kingston NVMe. Rebuild the pool, then replace the 2nd SP NVMe with another matching Kingston.

Except that every time I try and boot TrueNAS with the 2 Kingston ssds in the sled I get what appears to be a "trap or panic" and I can't go much further with the boot process.

If I leave the boot running, it eventually drops into this:

Obviously 2 matched ssds in itself is not a problem, as that's what I have running now. I've tried every combination of the 2 ssds in 4 slots, and they all fail. It also works with 2 x SP drives + 1 Kingston in the sled.

All the devices are correctly set up for passthrough to the TrueNAS VM.

Does anyone have any thoughts here.

olivierlambert

Just to be sure I get it: you are passing through the NVMe devices right? If yes, have you checked https://docs.xcp-ng.org/compute/#nvme-storage-devices-on-linux ?

EddieA

Ha, I guess some basic information might be useful.

Here's a link to the ZimaCube hardware.

I'm running XCP: 8.2.1, buiid: release/yangtze/master/58 and the TrueNAS VM ir running SCALE, version 24.04.2.2.

Andrew

@EddieA It looks like a hardware issue. If you are using the newer KC3000/FURY you should check for a firmware update to EIFK31.7

EddieA

@Andrew said in Kernel trap (??) booting TrueNAS with 2 x Kingston NVMe SSDs:

If you are using the newer KC3000/FURY

Nope, they are the new NV3s.

EddieA

Further updates:

The 2 x Kingston SSDs can be passed through to other Linux based VMs (like Proxmox) without any issues.

If TrueNAS is loaded bare-metal on this server, it boots cleanly.

Ergo (to me anyway) it's an XCP/TrueNAS interaction causing the problem. The TrueNAS thread can be found here.

Cheers.

olivierlambert

Just to be sure I get it: you are passing through the NVMe devices right? If yes, have you checked https://docs.xcp-ng.org/compute/#nvme-storage-devices-on-linux ?

EddieA

@olivierlambert Yes, the devices were passed through to TrueNAS.

Would the brand of the NVMe make a difference or would this be based purely on the number of devices. I was able to passthrough 2 x SiliconPower devices without issue. It also worked with 1 x SiliconPower plus 1 x Kingston.

The failures occur when there are 2 x Kingston either in addition to any SiliconPower NVMe's or just on their own.

Cheers.

olivierlambert

The answer is "can be both", depending on the NVMe brand. Have you tried to give more IRQ to see the result?

EddieA

I wanted the previous answer before trying that, as I re-installed TrueNAS as bare-metal in the interim and didn't want to re-image back to XCP if it was purely dependent on the number of NVMe's and not the brands, as I already tested with just the 2 x Kingston.

Let me find a time I can re-image XCP and try with that parameter. Stand by.

Cheers.

EddieA

That appears to be the solution. After adding that additional parameter, my TrueNAS SCALE system now boots cleanly, with all 4 NVME's available to it.

Cheers, and Thanks.

TS79

This post is deleted!

olivierlambert

@EddieA Glad it worked Not all NVMe are created equals

EddieA

@olivierlambert said in Kernel trap (??) booting TrueNAS with 2 x Kingston NVMe SSDs:

Not all NVMe are created equals

Now all I need to do is determine if these errors are from the NVME's themselves or the 4 x NVMe sled that they're inserted in. LOL.

Again, thanks for the help.