Older intel (E5-2620) Supermicro server (X9DRW-iF) Running TrueNAS13 VM

lightingman117

I wanted to detail my experience installing XCP-ng on a few different systems.

Here's an older system. I'm very happy with this for what it is (I have Threadripper system that is awesome).

Interesting details:
TrueNAS13 running inside XCP-ng (Just follow Tom's Guide)

Equipment:
Chassis: CSE-825
Mobo: X9DRW-iF
Proc: 2x E5-2620
Mem: 64GB M393B2G70BH0-CK0 | DDR3 1600MHz PC3-12800R | forced to 1600
SSD: MX500, Evo870: 500GB Software Raid (XCP-NG) Boot -> UEFI:XCP-NG
HBA: Avago (Broadcom) 9311-8i
Expander: SAS825TQ (SAS2, 8xSATA)

Notes:
Live swapping SATA or SFF-8643 cables works
SilverStone Technology SFF-8643 to SATA+Sideband (CPS05-RE) does not work
0.5M is barely enough, 0.6M is just about right, 1.0M would be way better
Flashing IT over TrueNAS did not work (could not erase)
64G of RDIMM was cheaper than UDIMM due to larger stick capcaity (8x 8G vs 4x 16G)
IPMI - IP changed from GUI not from BIOS (weird)
Leaving the system powered on will make swapping/testing faster due to IPMI not needing to boot everytime you remove power (obv don't do this for system resources [cpu] [Memory, PCIe are debatable; perhaps with grounding]
XCP-NG 8.2.1 works very well https://xcp-ng.org/#easy-to-install

Updating the bios/IPMI (BMC) on the mobo is a PITA, but worth it so IPMI works better

I did IPMI through the web interface
http://blog.smalleycreative.com/linux/flashing-the-ipmi-firmware-on-a-supermicro-x9drw-if/

You have to unlock the bios for it to update
https://peterkleissner.com/2018/05/27/reverse-engineering-supermicro-ipmi/
https://www.supermicro.com/support/resources/bios_ipmi.php
http://blog.smalleycreative.com/administration/updating-the-bios-firmware-on-the-supermicro-x9drw-if/

HBA Cards:

SAS2 - 2xxx controller (Sata2, PCIE 2.0, 6Gbps)
SAS3 - 3xxx controller (SATA3, PCIE 3.0, 12Gbps)

IR = raid
IT = HBA ("initiator target")
https://forums.servethehome.com/index.php?threads/what-is-it-mode.328/#:~:text=IT mode stands for "initiator,raid formats the HBA supports.

fix boottime issues (I haven't tried this):
https://www.reddit.com/r/DataHoarder/comments/65w19b/do_hbas_increase_boot_time/

Flashing HBA (IR to IT):

-You could flash inside TrueNAS, but it didn't work for me.
-Read the sas3Flash_quickRefGuide_rev1-0 guide downloaded with the package for windows

You're looking for these downloads (PXX - replace for your version [currently 16.0, 6-26-2022]:
SAS3_UEFI_BSD_PXX
9300_8i_Package_PXX_IR_IT_FW_BIOS_for_MSDOS_Windows

you want these files:
sas3flsh.efi or sas3flash.efi
SAS9300_8i_IT.bin (you do NOT want to use the IR bin)
mptsas3.rom
mpt3x64.rom

Format flash drive fat32
place files on it
uefi boot
switch to removable drive (see UEFI shell commands below) [fs0/1/2], ls -al
sas3flash.efi -list
sas3flash.efi -o -e 6 [6 preserves the MFG area]
sas3flash.efi -f SAS9300_8i_IT.bin -b mptsas3.rom -b mpt3x64.rom
flash to 16.00.1200 next (for TrueNAS SAS3 cards)

SAS 3008 (9311-8i)
https://www.broadcom.com/support/download-search?pg=&pf=&pn=&pa=&po=&dk=9300&pl=&l=false
https://www.servethehome.com/flash-lsi-sas-3008-hba-e-g-ibm-m1215-mode/
https://www.truenas.com/community/threads/how-do-i-flash-lsi-avago-sas-9311-i8.72964/

Flashing HBA (SAS3 16.00.12.00 < version is required for TrueNAS)
File here: https://www.truenas.com/community/resources/lsi-9300-xx-firmware-update.145/

Inside TrueNAS (I don't recommend this; card gets stuck 'resetting' requiring a host reboot):
-make a pool, smb it, allow guests, open ACL
-copy the downloaded file to the pool
-ssh or shell into TN, launch mc (midnight commander)
-make folder /etc/driver
-copy /mnt/tank/pool/SAS9300_8i_IT.bin to /etc/driver
-cd /etc/driver
-sas3flash -o -f SAS9300_8i_IT.bin [resetting adapter always gets stuck]

Outside:
-copy to flashdrive you used for IR to IT [overwrite]
-UEFI boot
-sas3flash -o -f SAS9300_8i_IT.bin [I renamed my v12 to SAS9300_8i_IT_1200.bin so I could do both at the same time]
-reboot

UEFI shell:
#basically a built-in unix (?) OS that can do things for you (instead of building a bootable flashdrive or loading DOS)
-use page up/down # scroll up/down
-map # show all devices
-fs0/1/2 # will be one of the drives (usb too)
-tab # auto-completes a word

https://docstore.mik.ua/manuals/hp-ux/en/5991-1247B/ch04s13.html
https://www.intel.com/content/dam/support/us/en/documents/motherboards/server/sb/efi_instructions.pdf
https://superuser.com/questions/1412403/how-to-scroll-up-and-down-in-efi-shell

What does this post have to do with XCP-NG?
Well...TrueNAS 13.0 is running inside of it with Pass-thru HBA SAS9300-8i to 6x HDD'same
XCP-NG is running on two SSD's 500GB
NIC-teaming is setup
Hypervisor host is performing very well with a VMS and SNMP server (both are resource hogs) and a few W10/SVR19 VMs

olivierlambert

Nice! Thanks for the feedback Happy to know you are happy with your setup!