Guest receiving passthrough SATA controllers does not see attached drives
-
I've got an ASrock Rack x570d4i-2t.
PCIe slot has an i340-T4 card passed through to a guest. <-- This guy is giving me a different set of woes, but that's for another post.
Another (NAS appliance) VM guest has both of the onboard SATA controllers exposed by two OCulink ports. As expected these controllers are attached to the x570 chipset.From the NAS VM I can see the controllers via
lspci
butlsblk
doesn't show the attached disks. I'm inclined to think there is another device that must be passed through as well because the OCulink ports can be used for PCIe to NVME, PCIe to SAS, or SATA via a breakout cable. I've opted for the latter. An lscpi of the host is below.00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex 00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 51) 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 10:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 10:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 10:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 10:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01) 20:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream 21:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 21:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 21:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 21:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 21:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge 23:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01) 23:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01) 27:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04) 28:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41) 2a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP 2a:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller 2a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller 2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) 2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) 30:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Renoir (rev d8) 30:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 1637 30:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor 30:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 30:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 30:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller 31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81) 31:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81)
10:00.x Is the I340-T4
By studying the manual and the /sys filesystem, I've determined that 2b:00.0 and 2c:00.0 are the SATA controllers mapped to the OCulink ports.In my use case, the NAS will handle file-based storage for many external and internal services and offer iSCSI targets for a couple VM's on the same host. I have a full rack of gear now, but the wife and I moving into a much smaller space so something like this has to happen, but not having storage is a show-stopper.
Thoughts?
-
Is my question missing something, or am I so far out in the backfield that no one has anything to contribute?
Advice on this specific board would be useful, but all I'm really hoping for is some troubleshooting guidance.
-
Hmm I think getting the internal controller might be not trivial, I think I've read recent feedback on similar problems
-
@olivierlambert This is...sub-optimal, but thank you for responding.
-
Let's wait for more community feedback Maybe @fohdeesha remember something about this?
-
Can you see drives on the host if disable pass through?
Recently I have some troubles with OCulink to NVMe cables.
That was Supermicro server and they have different cables for SATA/SAS and NVMe.
The difference is in wire resistance - for SATA/SAS you should use cables with 100Ohm resistance and for NVM with 85Ohm.
In such way motherboard detects what type of drives are connected to hybrid port.It is just my ideas what happens.
First you should check if host can see drives without pass through.
-
@splastunov If I disable passthrough all drives are visible on my XCP-ng host, with passthrough enabled, they disappear but are also not visible on my guest--though the PCI devices are.
The only reasons I care about this are to extract every last bit of performance out of the drives since they're magnetic and not solid-state and to ensure that no other OS has access to these drives.
-
That's good sign.
On host you can check to which pci-e controller drives are connected by command
ls -al /sys/block/sd*
You will see something like this
Beforeata...
you will find pci-e addressCheck if you passthrough correct SATA controller.
I thing you have 2 SATA controllers because your MB have M.2 connector. -
lrwxrwxrwx 1 root root 0 May 4 08:58 /sys/block/sda -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata2/host1/target1:0:0/1:0:0:0/block/sda lrwxrwxrwx 1 root root 0 May 4 08:58 /sys/block/sdb -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata4/host3/target3:0:0/3:0:0:0/block/sdb lrwxrwxrwx 1 root root 0 May 4 08:58 /sys/block/sdc -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata6/host5/target5:0:0/5:0:0:0/block/sdc lrwxrwxrwx 1 root root 0 May 4 08:58 /sys/block/sdd -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata7/host6/target6:0:0/6:0:0:0/block/sdd lrwxrwxrwx 1 root root 0 May 4 08:58 /sys/block/sde -> ../devices/pci0000:00/0000:00:01.2/0000:20:00.0/0000:21:09.0/0000:2b:00.0/ata8/host7/target7:0:0/7:0:0:0/block/sde lrwxrwxrwx 1 root root 0 May 4 08:58 /sys/block/sdf -> ../devices/pci0000:00/0000:00:08.2/0000:31:00.0/ata11/host10/target10:0:0/10:0:0:0/block/sdf
The first 5 are the disks I'm interested in. What's interesting is that in this run, those disks appear on 0000:2b:00.0. Previously the disks were spread across 0000:2b:00.0 and 0000:2c:00.0
What changed? I swapped out my XCP-ng boot drive from a 512G Samsung m.2 NVMe drive to a Crucial 1000G m.2 SATA drive. I needed the extra space and it gave me a chance to try the Xen-Orchestra Terraform provider (It's lacking features, but what it has works well).I suspect there is some funny business going on with another PCIe device and probably the SATA controller at 0000:31:00.0 and 0000:31.00.1, the former of which is the address of the m.2 drive now.
-
Outside of the initial topic scope, but if you have missing Terraform features, please report them on the repo so we can know and decide what to do