XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Sata controller passthrough. VM no bootable device

    Scheduled Pinned Locked Moved
    Compute
    3
    24
    4.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • I
      impovich
      last edited by impovich

      Hi all, trying to install Truenas Core 12 on xcp-ng 8.2.
      Passed through SATA controller to a VM, installer detected hard drives, installed Truenas and after reboot, I'm getting "no bootable device".
      What I did:

      [10:56 chicago ~]# lspci
00:00.0 Host bridge: Intel Corporation Device 3e33 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.4 Communication controller: Intel Corporation Device a364 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #22 (rev f0)
00:1b.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #23 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
00:1f.0 ISA bridge: Intel Corporation Device a30a (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

      /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(00:17.0)"

      [11:08 chicago ~]# xe vm-list
uuid ( RO)           : 7d16b32f-fe80-4271-8543-68c5a642220e
     name-label ( RW): Control domain on host: chicago
    power-state ( RO): running


uuid ( RO)           : adfa344c-d136-885f-8396-4f332ad40770
     name-label ( RW): Truenas
    power-state ( RO): halted

      [11:25 chicago ~]# xl pci-assignable-list
0000:00:17.0

       xe vm-param-set other-config:pci=0/0000:00:17.0 uuid=adfa344c-d136-885f-8396-4f332ad40770

      [11:27 chicago ~]# xe host-dmesg | grep VT-d
(XEN) [    2.053436] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    2.066660] Intel VT-d Snoop Control enabled.
(XEN) [    2.079753] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    2.092849] Intel VT-d Queued Invalidation enabled.
(XEN) [    2.105873] Intel VT-d Interrupt Remapping enabled.
(XEN) [    2.118799] Intel VT-d Posted Interrupt not enabled.
(XEN) [    2.131762] Intel VT-d Shared EPT tables enabled.

      What am I doing wrong?

      EDIT:
      I didn't add virtual HDD, want to install on the real HDD

      1 Reply Last reply Reply Quote 0
      • olivierlambertO
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Can you reboot the entire host and see if it works after? Trying to see if it's a PCI passthrough device reset issue.

        I 1 Reply Last reply Reply Quote 0
        • I
          impovich @olivierlambert
          last edited by

          @olivierlambert the same issue

          I 1 Reply Last reply Reply Quote 0
          • I
            impovich @impovich
            last edited by

            I tried to install Truenas on virtual HDD which is M.2 SSD and passthrough the SATA controller, ended up having two ada0 devices.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Sorry I'm not a TrueNAS specialist so I'm not sure what's wrong here. Hopefully, someone in the community will assist 🙂

              I 1 Reply Last reply Reply Quote 0
              • I
                impovich @olivierlambert
                last edited by impovich

                @olivierlambert just tried to install Debian using the same approach, got exactly the same issue.

                1 Reply Last reply Reply Quote 0
                • J
                  jmccoy555
                  last edited by jmccoy555

                  Hi, it sounds to me like you are trying to install TrueNAS on one of the passed through drives, which I think still requires a virtual drive for the bootloader and a bit of jiggery-pokery, and is probably not what you actually want to do as you will lose (I think) all the good stuff that come with using VMs.... backups, snapshots etc.

                  Also you'll be using one of your disks or slots that you could otherwise be using for your storage pool.

                  I would not pass through the controller to begin with, get TrueNAS up and running on a virtual disk, shut down, pass though the controller, boot, create your pool from your passed thought disks and off you go.

                  I 1 Reply Last reply Reply Quote 0
                  • I
                    impovich @jmccoy555
                    last edited by impovich

                    @jmccoy555 Actually that is simple to do, it is already up and running 🙂 I just wanted to pass through the whole controller in order to have SMART checks and so on, as for snapshots - it is not possible to make them until everything is under xcp-ng itself.

                    EDIT:

                    get TrueNAS up and running on a virtual disk, shut down, pass through the controller, boot, create your pool from your passed thought disks, and off you go.

                    hm, that might work actually, have not tried it this way. Currently, I passed through separate HDD drives.

                    J 1 Reply Last reply Reply Quote 0
                    • J
                      jmccoy555 @impovich
                      last edited by

                      @impovich what did you have to do to get it working then?

                      I 1 Reply Last reply Reply Quote 0
                      • I
                        impovich @jmccoy555
                        last edited by

                        @jmccoy555 sorry, I was too quick and didn't read your message properly. (sorry)
                        Truenas installed on virtual HDD, physical HDD drives are passed though individually using symlinks.

                        J 1 Reply Last reply Reply Quote 0
                        • J
                          jmccoy555 @impovich
                          last edited by jmccoy555

                          @impovich Ah ok.... at least you're up and running.

                          Edit: read your edit now..... I think you are better off passing through the controller (been running this way for many years) rather than symlinks, but could be wrong.

                          I 1 Reply Last reply Reply Quote 0
                          • I
                            impovich @jmccoy555
                            last edited by impovich

                            @jmccoy555 i just tried what you've suggested, unfortunately two ada0 disks again 😞

                            root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16
                            J 1 Reply Last reply Reply Quote 0
                            • J
                              jmccoy555 @impovich
                              last edited by

                              @impovich just checked, my virtual disc is ada0 and my passedthough discs come up as da0, da1, etc. etc

                              I 1 Reply Last reply Reply Quote 0
                              • I
                                impovich @jmccoy555
                                last edited by

                                @jmccoy555 i guess you have an HBA controller?

                                J 1 Reply Last reply Reply Quote 0
                                • J
                                  jmccoy555 @impovich
                                  last edited by jmccoy555

                                  @impovich Ah yeah. Only passed though a controller to Linux VMs and that works fine. Could try TrueNAS Scale??

                                  https://forums.freebsd.org/threads/dev-ad4-becomes-dev-ada0.29899/ Any help?

                                  I'm out of ideas I'm afraid.

                                  I 1 Reply Last reply Reply Quote 0
                                  • I
                                    impovich @jmccoy555
                                    last edited by

                                    @jmccoy555 said in Sata controller passthrough. VM no bootable device:

                                    @impovich Ah yeah. Only passed though a controller to Linux VMs and that works fine. Could try TrueNAS Scale??

                                    https://forums.freebsd.org/threads/dev-ad4-becomes-dev-ada0.29899/ Any help?

                                    I'm out of ideas I'm afraid.

                                    I tried to do the same with Debian, no success. Reading freebsd forum regarding labeling

                                    J 1 Reply Last reply Reply Quote 0
                                    • J
                                      jmccoy555 @impovich
                                      last edited by

                                      @impovich Passing though my second onbord SATA controller to a Debian 10 VM gives me xvda for the virtual disk and sda, sdb, etc for discs on the passed though controller.

                                      I 1 Reply Last reply Reply Quote 0
                                      • I
                                        impovich @jmccoy555
                                        last edited by impovich

                                        @jmccoy555 so managed to do it with a dirty hack 🙂

                                        1. Spwaned a new VM with 3 virtual drives
                                          ada0 - 5gb
                                          ada1 - 5gb
                                          ada2 - 50gb
                                        2. installed Truenas on ada2.
                                        3. After installation removed ada0 and ada1
                                        4. Rebooted VM and confirmed that ada2 is still ada2
                                        5. passed through SATA controller
                                        6. Bingo!!
                                        
root@truenas[~]# geom disk list
Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)


Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5

   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
                                        1 Reply Last reply Reply Quote 0
                                        • I
                                          impovich
                                          last edited by impovich

                                          Finally, I found a solution to the issue:
                                          https://www.truenas.com/community/threads/multiple-ada0-partitions-on-xen-pv-w-passthrough.16574/

                                          set via GUI tunables interface:

                                          hint.ada.0.at="scbus100".

                                          Works perfectly!

                                          root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16
                                          J 2 Replies Last reply Reply Quote 0
                                          • J
                                            jmccoy555 @impovich
                                            last edited by

                                            @impovich How odd that that can even happen. I still prefer the hack 😆

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post