Ok so I'm going to answer my own question.
The answer is that the vdi (local SR) you are trying to boot from and the pci slots are on two different machines and you need to migrate the boot vdi to a shared pool sr or to the local sr of the machine with the pci slots!
Trying to pass two storage controllers into a new vm after deleting the vm that used them before
I have verified the hardware is still there:
18:39 xen2 ~]# /opt/xensource/libexec/xen-cmdline --delete-dom0 xen-pciback.hide
[18:40 xen2 ~]# reboot
[18:44 xen2 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdf 8:80 0 931.5G 0 disk
nvme0n1 259:0 0 477G 0 disk
├─nvme0n1p5 259:5 0 4G 0 part /var/log
├─nvme0n1p3 259:3 0 435.4G 0 part
│ └─XSLocalEXT--6570b794--3efe--80fe--d273--8ccf1e7d1ec3-6570b794--3efe--80fe--d273--8ccf1e7d1ec3 253:0 0 435.4G 0 lvm /run/sr-mount/6570b794-3efe-80fe-d273-8ccf1e7d1ec3
├─nvme0n1p1 259:1 0 18G 0 part /
├─nvme0n1p6 259:6 0 1G 0 part [SWAP]
├─nvme0n1p4 259:4 0 512M 0 part /boot/efi
└─nvme0n1p2 259:2 0 18G 0 part
sdd 8:48 0 931.5G 0 disk
sdb 8:16 0 931.5G 0 disk
sdk 8:160 0 223.6G 0 disk
sdi 8:128 0 238.5G 0 disk
sdg 8:96 0 931.5G 0 disk
sde 8:64 0 931.5G 0 disk
sdc 8:32 0 931.5G 0 disk
sda 8:0 0 931.5G 0 disk
sdj 8:144 0 223.6G 0 disk
sdh 8:112 0 931.5G 0 disk
[18:44 xen2 ~]# lspci | egrep -i "sata|sas"
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
04:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
[18:44 xen2 ~]# /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:00:1f.2)(0000:04:00.0)"
[18:46 xen2 ~]# reboot
[18:49 xen2 ~]# xl pci-assignable-list
0000:00:1f.2
0000:04:00.0
[18:49 xen2 ~]# xe vm-param-set other-config:pci=0/0000:00:1f.2,0/0000:04:00.0 uuid=0bbfdac9-8ec0-938b-ccdb-0cfe7927e6c7
getting this error when trying to start the vm
vm.start
{
"id": "0bbfdac9-8ec0-938b-ccdb-0cfe7927e6c7",
"bypassMacAddressesCheck": false,
"force": false
}
{
"code": "INTERNAL_ERROR",
"params": [
"xenopsd internal error: Cannot_add(0000:04:00.0, Device_common.QMP_Error(138, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"Failed to initialize 11/15, type = 0x1, rc: -1\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-001026-138\\\"}\"))"
],
"call": {
"method": "VM.start",
"params": [
"OpaqueRef:90abe1a0-8fa2-4913-a01b-6933fb75d442",
false,
false
]
},
"message": "INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Device_common.QMP_Error(138, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"Failed to initialize 11/15, type = 0x1, rc: -1\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-001026-138\\\"}\")))",
"name": "XapiError",
"stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:04:00.0, Device_common.QMP_Error(138, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"Failed to initialize 11/15, type = 0x1, rc: -1\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-001026-138\\\"}\")))
at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202409101827/packages/xen-api/_XapiError.mjs:16:12)
at file:///opt/xo/xo-builds/xen-orchestra-202409101827/packages/xen-api/transports/json-rpc.mjs:38:21
at runNextTicks (node:internal/process/task_queues:60:5)
at processImmediate (node:internal/timers:454:9)
at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}
any thoughts ?
This machine is a pool master, so contemplating moving the master to another machine and reinstalling this one.