Sata controller passthrough. VM no bootable device

impovich

Hi all, trying to install Truenas Core 12 on xcp-ng 8.2.
Passed through SATA controller to a VM, installer detected hard drives, installed Truenas and after reboot, I'm getting "no bootable device".
What I did:

[10:56 chicago ~]# lspci
00:00.0 Host bridge: Intel Corporation Device 3e33 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.4 Communication controller: Intel Corporation Device a364 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #22 (rev f0)
00:1b.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #23 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
00:1f.0 ISA bridge: Intel Corporation Device a30a (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(00:17.0)"

[11:08 chicago ~]# xe vm-list
uuid ( RO)           : 7d16b32f-fe80-4271-8543-68c5a642220e
     name-label ( RW): Control domain on host: chicago
    power-state ( RO): running


uuid ( RO)           : adfa344c-d136-885f-8396-4f332ad40770
     name-label ( RW): Truenas
    power-state ( RO): halted

[11:25 chicago ~]# xl pci-assignable-list
0000:00:17.0

 xe vm-param-set other-config:pci=0/0000:00:17.0 uuid=adfa344c-d136-885f-8396-4f332ad40770

[11:27 chicago ~]# xe host-dmesg | grep VT-d
(XEN) [    2.053436] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    2.066660] Intel VT-d Snoop Control enabled.
(XEN) [    2.079753] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    2.092849] Intel VT-d Queued Invalidation enabled.
(XEN) [    2.105873] Intel VT-d Interrupt Remapping enabled.
(XEN) [    2.118799] Intel VT-d Posted Interrupt not enabled.
(XEN) [    2.131762] Intel VT-d Shared EPT tables enabled.

What am I doing wrong?

EDIT:
I didn't add virtual HDD, want to install on the real HDD

olivierlambert

Can you reboot the entire host and see if it works after? Trying to see if it's a PCI passthrough device reset issue.

impovich

@olivierlambert the same issue

impovich

I tried to install Truenas on virtual HDD which is M.2 SSD and passthrough the SATA controller, ended up having two ada0 devices.

olivierlambert

Sorry I'm not a TrueNAS specialist so I'm not sure what's wrong here. Hopefully, someone in the community will assist

impovich

@olivierlambert just tried to install Debian using the same approach, got exactly the same issue.

jmccoy555

Hi, it sounds to me like you are trying to install TrueNAS on one of the passed through drives, which I think still requires a virtual drive for the bootloader and a bit of jiggery-pokery, and is probably not what you actually want to do as you will lose (I think) all the good stuff that come with using VMs.... backups, snapshots etc.

Also you'll be using one of your disks or slots that you could otherwise be using for your storage pool.

I would not pass through the controller to begin with, get TrueNAS up and running on a virtual disk, shut down, pass though the controller, boot, create your pool from your passed thought disks and off you go.

impovich

@jmccoy555 Actually that is simple to do, it is already up and running I just wanted to pass through the whole controller in order to have SMART checks and so on, as for snapshots - it is not possible to make them until everything is under xcp-ng itself.

EDIT:

get TrueNAS up and running on a virtual disk, shut down, pass through the controller, boot, create your pool from your passed thought disks, and off you go.

hm, that might work actually, have not tried it this way. Currently, I passed through separate HDD drives.

jmccoy555

@impovich what did you have to do to get it working then?

impovich

@jmccoy555 sorry, I was too quick and didn't read your message properly. (sorry)
Truenas installed on virtual HDD, physical HDD drives are passed though individually using symlinks.

jmccoy555

@impovich Ah ok.... at least you're up and running.

Edit: read your edit now..... I think you are better off passing through the controller (been running this way for many years) rather than symlinks, but could be wrong.

impovich

@jmccoy555 i just tried what you've suggested, unfortunately two ada0 disks again

root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

jmccoy555

@impovich just checked, my virtual disc is ada0 and my passedthough discs come up as da0, da1, etc. etc

impovich

@jmccoy555 i guess you have an HBA controller?

jmccoy555

@impovich Ah yeah. Only passed though a controller to Linux VMs and that works fine. Could try TrueNAS Scale??

https://forums.freebsd.org/threads/dev-ad4-becomes-dev-ada0.29899/ Any help?

I'm out of ideas I'm afraid.

impovich

@jmccoy555 said in Sata controller passthrough. VM no bootable device:

@impovich Ah yeah. Only passed though a controller to Linux VMs and that works fine. Could try TrueNAS Scale??

https://forums.freebsd.org/threads/dev-ad4-becomes-dev-ada0.29899/ Any help?

I'm out of ideas I'm afraid.

I tried to do the same with Debian, no success. Reading freebsd forum regarding labeling

jmccoy555

@impovich Passing though my second onbord SATA controller to a Debian 10 VM gives me xvda for the virtual disk and sda, sdb, etc for discs on the passed though controller.

impovich

@jmccoy555 so managed to do it with a dirty hack

Spwaned a new VM with 3 virtual drives
ada0 - 5gb
ada1 - 5gb
ada2 - 50gb
installed Truenas on ada2.
After installation removed ada0 and ada1
Rebooted VM and confirmed that ada2 is still ada2
passed through SATA controller
Bingo!!


root@truenas[~]# geom disk list
Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)


Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5

   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

impovich

Finally, I found a solution to the issue:
https://www.truenas.com/community/threads/multiple-ada0-partitions-on-xen-pv-w-passthrough.16574/

set via GUI tunables interface:

hint.ada.0.at="scbus100".

Works perfectly!

root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

jmccoy555

@impovich How odd that that can even happen. I still prefer the hack