Multiple PCI-E passthrough problem

[kasn_code]

Hi,

I have followed this guide here -- https://xcp-ng.org/docs/compute.html#pci-passthrough

I have two pci devices that I want to passthrough to the VM. What I've noticed is that when I use this cmd

xe vm-param-set other-config:pci=0/0000:05:00.0,0/0000:06:00.0 uuid=5cb573b0-cfae-4625-8feb-e14b030ed09f

It only seems to see the last one on the list and not the other. I have verified that if I do one at a time that it'll work.

Not sure how to troubleshoot this, any pointers here would be great!

Thanks

[kasn_code]

I should also add that that lspci shows both connected

lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:05.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
00:06.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

It's just that fdisk doesn't.

Disk /dev/loop0: 111.95 MiB, 117387264 bytes, 229272 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 63.46 MiB, 66547712 bytes, 129976 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 40.86 MiB, 42840064 bytes, 83672 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/xvda: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C62F22EE-6B8C-4F79-846D-A06BA64A39C6

Device       Start      End  Sectors  Size Type
/dev/xvda1  227328 20971486 20744159  9.9G Linux filesystem
/dev/xvda14   2048    10239     8192    4M BIOS boot
/dev/xvda15  10240   227327   217088  106M EFI System

Partition table entries are not in disk order.


Disk /dev/xvdb: 12 MiB, 12582912 bytes, 24576 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start   End Sectors Size Id Type
/dev/xvdb1 *        1 20481   20481  10M  e W95 FAT16 (LBA)


Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000DM004-2U91
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F5A010FC-D1A3-11ED-9FD7-97E7C1552BE2

lsblk
NAME     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0      7:0    0 111.9M  1 loop /snap/lxd/24322
loop1      7:1    0  63.5M  1 loop /snap/core20/2015
loop2      7:2    0  40.9M  1 loop /snap/snapd/20290
sda        8:0    0   7.3T  0 disk
xvda     202:0    0    10G  0 disk
├─xvda1  202:1    0   9.9G  0 part /
├─xvda14 202:14   0     4M  0 part
└─xvda15 202:15   0   106M  0 part /boot/efi
xvdb     202:16   0    12M  0 disk
└─xvdb1  202:17   0    10M  0 part

I can confirm that xen-orchestra I can create ext(local) on both /dev/sda and /dev/sdb.

[kasn_code]

So I managed to get the VM to see both drives.

In dmesg I can see something like this

[    1.160149] ahci 0000:00:08.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x20 impl SATA mode
[    1.161043] ahci 0000:00:08.0: flags: 64bit ncq sntf ilck led clo only pmp fbs pio slum part
[    1.187888] ahci 0000:00:09.0: SSS flag set, parallel bus scan disabled
[    2.186211] ahci 0000:00:09.0: controller reset failed (0xffffffff)
[    2.187307] ahci: probe of 0000:00:09.0 failed with error -5

So I removed the pci and did a rescan with

root@mars-test-uefi:~# echo "1" > /sys/bus/pci/devices/0000\:00\:09.0/remove
root@mars-test-uefi:~# echo "1" > /sys/bus/pci/rescan

Now when I checked dmesg it shows up properly

[  795.538214] ahci 0000:00:09.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x4 impl SATA mode
[  795.538219] ahci 0000:00:09.0: flags: 64bit ncq sntf ilck led clo only pmp fbs pio slum part

fdisk -l also shows up both drives now too

root@mars-test-uefi:~# fdisk -l

Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000DM004-2U91
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/sdb: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: WDC WD80EFZZ-68B
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

It seems like on boot it tries to find the device but it fails to, but on a remove/rescan it seems like its fine?

The question now becomes, how do I make this work ON boot.