Sata controller passthrough. VM no bootable device

olivierlambert

Can you reboot the entire host and see if it works after? Trying to see if it's a PCI passthrough device reset issue.

impovich

impovich

I tried to install Truenas on virtual HDD which is M.2 SSD and passthrough the SATA controller, ended up having two ada0 devices.

olivierlambert

Sorry I'm not a TrueNAS specialist so I'm not sure what's wrong here. Hopefully, someone in the community will assist

impovich

@olivierlambert just tried to install Debian using the same approach, got exactly the same issue.

jmccoy555

Hi, it sounds to me like you are trying to install TrueNAS on one of the passed through drives, which I think still requires a virtual drive for the bootloader and a bit of jiggery-pokery, and is probably not what you actually want to do as you will lose (I think) all the good stuff that come with using VMs.... backups, snapshots etc.

Also you'll be using one of your disks or slots that you could otherwise be using for your storage pool.

I would not pass through the controller to begin with, get TrueNAS up and running on a virtual disk, shut down, pass though the controller, boot, create your pool from your passed thought disks and off you go.

impovich

@jmccoy555 Actually that is simple to do, it is already up and running I just wanted to pass through the whole controller in order to have SMART checks and so on, as for snapshots - it is not possible to make them until everything is under xcp-ng itself.

EDIT:

get TrueNAS up and running on a virtual disk, shut down, pass through the controller, boot, create your pool from your passed thought disks, and off you go.

hm, that might work actually, have not tried it this way. Currently, I passed through separate HDD drives.

jmccoy555

@impovich what did you have to do to get it working then?

impovich

@jmccoy555 sorry, I was too quick and didn't read your message properly. (sorry)
Truenas installed on virtual HDD, physical HDD drives are passed though individually using symlinks.

jmccoy555

@impovich Ah ok.... at least you're up and running.

Edit: read your edit now..... I think you are better off passing through the controller (been running this way for many years) rather than symlinks, but could be wrong.

impovich

@jmccoy555 i just tried what you've suggested, unfortunately two ada0 disks again

root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

jmccoy555

@impovich just checked, my virtual disc is ada0 and my passedthough discs come up as da0, da1, etc. etc

impovich

@jmccoy555 i guess you have an HBA controller?

jmccoy555

@impovich Ah yeah. Only passed though a controller to Linux VMs and that works fine. Could try TrueNAS Scale??

https://forums.freebsd.org/threads/dev-ad4-becomes-dev-ada0.29899/ Any help?

I'm out of ideas I'm afraid.

impovich

@jmccoy555 said in Sata controller passthrough. VM no bootable device:

@impovich Ah yeah. Only passed though a controller to Linux VMs and that works fine. Could try TrueNAS Scale??

https://forums.freebsd.org/threads/dev-ad4-becomes-dev-ada0.29899/ Any help?

I'm out of ideas I'm afraid.

I tried to do the same with Debian, no success. Reading freebsd forum regarding labeling

jmccoy555

@impovich Passing though my second onbord SATA controller to a Debian 10 VM gives me xvda for the virtual disk and sda, sdb, etc for discs on the passed though controller.

impovich

@jmccoy555 so managed to do it with a dirty hack

Spwaned a new VM with 3 virtual drives
ada0 - 5gb
ada1 - 5gb
ada2 - 50gb
installed Truenas on ada2.
After installation removed ada0 and ada1
Rebooted VM and confirmed that ada2 is still ada2
passed through SATA controller
Bingo!!


root@truenas[~]# geom disk list
Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)


Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5

   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

impovich

Finally, I found a solution to the issue:
https://www.truenas.com/community/threads/multiple-ada0-partitions-on-xen-pv-w-passthrough.16574/

set via GUI tunables interface:

hint.ada.0.at="scbus100".

Works perfectly!

root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

jmccoy555

@impovich How odd that that can even happen. I still prefer the hack

jmccoy555

@impovich I'm running with a 10gb boot drive if you want to save a bit of space, speed up backups etc.