XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Guest receiving passthrough SATA controllers does not see attached drives

    Scheduled Pinned Locked Moved Compute
    pcipassthroughmissingdisksx570x570d4i-2t
    10 Posts 3 Posters 967 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      SeanS
      last edited by

      I've got an ASrock Rack x570d4i-2t.
      PCIe slot has an i340-T4 card passed through to a guest. <-- This guy is giving me a different set of woes, but that's for another post.
      Another (NAS appliance) VM guest has both of the onboard SATA controllers exposed by two OCulink ports. As expected these controllers are attached to the x570 chipset.

      From the NAS VM I can see the controllers via lspci but lsblk doesn't show the attached disks. I'm inclined to think there is another device that must be passed through as well because the OCulink ports can be used for PCIe to NVME, PCIe to SAS, or SATA via a breakout cable. I've opted for the latter. An lscpi of the host is below.

      00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7
10:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
10:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
10:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
10:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
21:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
21:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
23:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
23:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
27:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
28:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
2a:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
2a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
30:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Renoir (rev d8)
30:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1637
30:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
30:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
30:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1
30:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller
31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)
31:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)

      10:00.x Is the I340-T4
      By studying the manual and the /sys filesystem, I've determined that 2b:00.0 and 2c:00.0 are the SATA controllers mapped to the OCulink ports.

      In my use case, the NAS will handle file-based storage for many external and internal services and offer iSCSI targets for a couple VM's on the same host. I have a full rack of gear now, but the wife and I moving into a much smaller space so something like this has to happen, but not having storage is a show-stopper.

      Thoughts?

      S 1 Reply Last reply Reply Quote 0
      • S
        SeanS @SeanS
        last edited by

        Is my question missing something, or am I so far out in the backfield that no one has anything to contribute?

        Advice on this specific board would be useful, but all I'm really hoping for is some troubleshooting guidance.

        1 Reply Last reply Reply Quote 0
        • olivierlambertO
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Hmm I think getting the internal controller might be not trivial, I think I've read recent feedback on similar problems 🤔

          S 1 Reply Last reply Reply Quote 0
          • S
            SeanS @olivierlambert
            last edited by

            @olivierlambert This is...sub-optimal, but thank you for responding.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Let's wait for more community feedback 🙂 Maybe @fohdeesha remember something about this?

              1 Reply Last reply Reply Quote 0
              • splastunovS
                splastunov
                last edited by splastunov

                Can you see drives on the host if disable pass through?

                Recently I have some troubles with OCulink to NVMe cables.
                That was Supermicro server and they have different cables for SATA/SAS and NVMe.
                The difference is in wire resistance - for SATA/SAS you should use cables with 100Ohm resistance and for NVM with 85Ohm.
                In such way motherboard detects what type of drives are connected to hybrid port.

                It is just my ideas what happens.

                First you should check if host can see drives without pass through.

                S 1 Reply Last reply Reply Quote 0
                • S
                  SeanS @splastunov
                  last edited by

                  @splastunov If I disable passthrough all drives are visible on my XCP-ng host, with passthrough enabled, they disappear but are also not visible on my guest--though the PCI devices are.

                  The only reasons I care about this are to extract every last bit of performance out of the drives since they're magnetic and not solid-state and to ensure that no other OS has access to these drives.

                  1 Reply Last reply Reply Quote 0
                  • splastunovS
                    splastunov
                    last edited by

                    That's good sign.

                    On host you can check to which pci-e controller drives are connected by command
                    ls -al /sys/block/sd*

                    You will see something like this
                    6c356f83-9d8a-40dc-a3c4-9c70bf363307-image.png
                    Before ata... you will find pci-e address

                    Check if you passthrough correct SATA controller.
                    I thing you have 2 SATA controllers because your MB have M.2 connector.

                    S 1 Reply Last reply Reply Quote 0
                    • S
                      SeanS @splastunov
                      last edited by

                      @splastunov

                      lrwxrwxrwx 1 root root 0 May  4 08:58 /sys/block/sda -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata2/host1/target1:0:0/1:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 May  4 08:58 /sys/block/sdb -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb
lrwxrwxrwx 1 root root 0 May  4 08:58 /sys/block/sdc -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata6/host5/target5:0:0/5:0:0:0/block/sdc
lrwxrwxrwx 1 root root 0 May  4 08:58 /sys/block/sdd -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdd
lrwxrwxrwx 1 root root 0 May  4 08:58 /sys/block/sde -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata8/host7/target7:0:0/7:0:0:0/block/sde
lrwxrwxrwx 1 root root 0 May  4 08:58 /sys/block/sdf -> ../devices/pci0000:00/0000:00:08.2/0000:31:00.0/ata11/host10/target10:0:0/10:0:0:0/block/sdf

                      The first 5 are the disks I'm interested in. What's interesting is that in this run, those disks appear on 0000:2b:00.0. Previously the disks were spread across 0000:2b:00.0 and 0000:2c:00.0
                      What changed? I swapped out my XCP-ng boot drive from a 512G Samsung m.2 NVMe drive to a Crucial 1000G m.2 SATA drive. I needed the extra space and it gave me a chance to try the Xen-Orchestra Terraform provider (It's lacking features, but what it has works well).

                      I suspect there is some funny business going on with another PCIe device and probably the SATA controller at 0000:31:00.0 and 0000:31.00.1, the former of which is the address of the m.2 drive now.

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Outside of the initial topic scope, but if you have missing Terraform features, please report them on the repo so we can know and decide what to do 🙂

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post