@sapcode Sorry to hear you're experiencing the same issue. I couldn't find a solution, and QAT was sold long ago—no QAT, no issues.

Hello all, having an issue passing through QAT adapter to PFsense that I do not understand, so need some assistance.
On PFsense forum suggested that
"it does look like it's not passing that device through correctly. Or not completely."

c5:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
c6:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
c7:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)

[23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat
11    1 0xffffffff8475d000     4348 qat.ko (/boot/kernel/qat.ko)
		695 nexus/qat
12    5 0xffffffff84762000    10e10 qat_hw.ko (/boot/kernel/qat_hw.ko)
		694 pci/qat_c4xxx
		690 pci/qat_c62x
		693 pci/qat_dh895xcc
		691 pci/qat_200xx
		692 pci/qat_c3xxx
13    7 0xffffffff84773000    29840 qat_common.ko (/boot/kernel/qat_common.ko)
		688 qat_common
14    6 0xffffffff8479d000    66b78 qat_api.ko (/boot/kernel/qat_api.ko)
		689 qat_api
15    1 0xffffffff84804000    11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko)
		696 pci/qat_c2xxx

[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19

[21:44 dallas ~]# dmesg | grep pciback
[    0.000000] Command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0)
[    1.811384] Kernel command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0)
[    4.419889] pciback 0000:c5:00.0: seizing device
[    4.419895] pciback 0000:c6:00.0: seizing device
[    4.419900] pciback 0000:c7:00.0: seizing device
[    4.771730] pciback 0000:c7:00.0: enabling device (0140 -> 0142)
[    4.771804] pciback 0000:c7:00.0: %(#e41111)[Disabling System Error reporting on root port c2:04.0]
[    4.876894] pciback 0000:c6:00.0: enabling device (0140 -> 0142)
[    4.984902] pciback 0000:c5:00.0: enabling device (0140 -> 0142)
[    5.092811] xen_pciback: backend is vpci

c5:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
	Subsystem: Intel Corporation QuickAssist Adapter 8960
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 16
	Region 2: Memory at fb540000 (64-bit, non-prefetchable) [size=256K]
	Region 4: Memory at fb500000 (64-bit, non-prefetchable) [size=256K]
	Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [60] MSI-X: Enable- Count=17 Masked-
		Vector table: BAR=2 offset=0003b000
		PBA: BAR=2 offset=0003b800
	Capabilities: [6c] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [74] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [140 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 8, stride: 1, Device ID: 37c9
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000fb590000 (64-bit, non-prefetchable)
		Region 2: Memory at 00000000fb580000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1b0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: pciback

c6:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
	Subsystem: Intel Corporation QuickAssist Adapter 8960
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 17
	Region 2: Memory at fb440000 (64-bit, non-prefetchable) [size=256K]
	Region 4: Memory at fb400000 (64-bit, non-prefetchable) [size=256K]
	Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [60] MSI-X: Enable- Count=17 Masked-
		Vector table: BAR=2 offset=0003b000
		PBA: BAR=2 offset=0003b800
	Capabilities: [6c] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [74] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [140 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 8, stride: 1, Device ID: 37c9
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000fb490000 (64-bit, non-prefetchable)
		Region 2: Memory at 00000000fb480000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1b0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: pciback

c7:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
	Subsystem: Intel Corporation QuickAssist Adapter 8960
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 32 bytes
	Interrupt: pin A routed to IRQ 18
	Region 2: Memory at fb340000 (64-bit, non-prefetchable) [size=256K]
	Region 4: Memory at fb300000 (64-bit, non-prefetchable) [size=256K]
	Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [60] MSI-X: Enable- Count=17 Masked-
		Vector table: BAR=2 offset=0003b000
		PBA: BAR=2 offset=0003b800
	Capabilities: [6c] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [74] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
		UESvrt:	DLP+ SDES- TLP+ FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [140 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+
		IOVSta:	Migration-
		Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 8, stride: 1, Device ID: 37c9
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 00000000fb390000 (64-bit, non-prefetchable)
		Region 2: Memory at 00000000fb380000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [1b0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Kernel driver in use: pciback

Also the card is visible via xl pci-assignable-list after i passed it to pfsense. not sure if it is expected or not

@olivierlambert having exactly the same issue as wcom939 reported.
What did you mean by "Can you try with xe command to see if you have the same error?" ? Migrate using CLI? there are too many options:)

@jmccoy555 got it. Thank you for being with me during this amazing journey

@jmccoy555 sorry but didn't get it. I'm a newbie

Finally, I found a solution to the issue:
https://www.truenas.com/community/threads/multiple-ada0-partitions-on-xen-pv-w-passthrough.16574/

set via GUI tunables interface:

hint.ada.0.at="scbus100".

Works perfectly!

root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

@jmccoy555 so managed to do it with a dirty hack

1. Spwaned a new VM with 3 virtual drives
   ada0 - 5gb
   ada1 - 5gb
   ada2 - 50gb
2. installed Truenas on ada2.
3. After installation removed ada0 and ada1
4. Rebooted VM and confirmed that ada2 is still ada2
5. passed through SATA controller
6. Bingo!!

root@truenas[~]# geom disk list
Geom name: ada2
Providers:
1. Name: ada2
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)


Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5

   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

@jmccoy555 said in Sata controller passthrough. VM no bootable device:

@impovich Ah yeah. Only passed though a controller to Linux VMs and that works fine. Could try TrueNAS Scale??

https://forums.freebsd.org/threads/dev-ad4-becomes-dev-ada0.29899/ Any help?

I'm out of ideas I'm afraid.

I tried to do the same with Debian, no success. Reading freebsd forum regarding labeling

@jmccoy555 i guess you have an HBA controller?

@jmccoy555 i just tried what you've suggested, unfortunately two ada0 disks again

root@truenas[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 53687091200 (50G)
   Sectorsize: 512
   Mode: r1w1e2
   descr: (null)
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: QEMU QEMU DVD-ROM
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b4af0c
   ident: WCK7ABLM
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

Geom name: ada1
Providers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: ST4000LM024-2U817V
   lunid: 5000c500d5b3d1cb
   ident: WCK7ADS1
   rotationrate: 5400
   fwsectors: 63
   fwheads: 16

@jmccoy555 sorry, I was too quick and didn't read your message properly. (sorry)
Truenas installed on virtual HDD, physical HDD drives are passed though individually using symlinks.

@jmccoy555 Actually that is simple to do, it is already up and running I just wanted to pass through the whole controller in order to have SMART checks and so on, as for snapshots - it is not possible to make them until everything is under xcp-ng itself.

EDIT:

get TrueNAS up and running on a virtual disk, shut down, pass through the controller, boot, create your pool from your passed thought disks, and off you go.

hm, that might work actually, have not tried it this way. Currently, I passed through separate HDD drives.

@olivierlambert just tried to install Debian using the same approach, got exactly the same issue.

I tried to install Truenas on virtual HDD which is M.2 SSD and passthrough the SATA controller, ended up having two ada0 devices.

@olivierlambert the same issue

Hi all, trying to install Truenas Core 12 on xcp-ng 8.2.
Passed through SATA controller to a VM, installer detected hard drives, installed Truenas and after reboot, I'm getting "no bootable device".
What I did:

[10:56 chicago ~]# lspci
00:00.0 Host bridge: Intel Corporation Device 3e33 (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.4 Communication controller: Intel Corporation Device a364 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #22 (rev f0)
00:1b.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #23 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
00:1f.0 ISA bridge: Intel Corporation Device a30a (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
03:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(00:17.0)"

[11:08 chicago ~]# xe vm-list
uuid ( RO)           : 7d16b32f-fe80-4271-8543-68c5a642220e
     name-label ( RW): Control domain on host: chicago
    power-state ( RO): running


uuid ( RO)           : adfa344c-d136-885f-8396-4f332ad40770
     name-label ( RW): Truenas
    power-state ( RO): halted

[11:25 chicago ~]# xl pci-assignable-list
0000:00:17.0

 xe vm-param-set other-config:pci=0/0000:00:17.0 uuid=adfa344c-d136-885f-8396-4f332ad40770

[11:27 chicago ~]# xe host-dmesg | grep VT-d
(XEN) [    2.053436] Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB
(XEN) [    2.066660] Intel VT-d Snoop Control enabled.
(XEN) [    2.079753] Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) [    2.092849] Intel VT-d Queued Invalidation enabled.
(XEN) [    2.105873] Intel VT-d Interrupt Remapping enabled.
(XEN) [    2.118799] Intel VT-d Posted Interrupt not enabled.
(XEN) [    2.131762] Intel VT-d Shared EPT tables enabled.

What am I doing wrong?

EDIT:
I didn't add virtual HDD, want to install on the real HDD