Hello all, having an issue passing through QAT adapter to PFsense that I do not understand, so need some assistance.
On PFsense forum suggested that
"it does look like it's not passing that device through correctly. Or not completely."
c5:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
c6:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
c7:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
[23.01-RELEASE][root@pfSense.home.lan]/root: kldstat -v | grep qat
11 1 0xffffffff8475d000 4348 qat.ko (/boot/kernel/qat.ko)
695 nexus/qat
12 5 0xffffffff84762000 10e10 qat_hw.ko (/boot/kernel/qat_hw.ko)
694 pci/qat_c4xxx
690 pci/qat_c62x
693 pci/qat_dh895xcc
691 pci/qat_200xx
692 pci/qat_c3xxx
13 7 0xffffffff84773000 29840 qat_common.ko (/boot/kernel/qat_common.ko)
688 qat_common
14 6 0xffffffff8479d000 66b78 qat_api.ko (/boot/kernel/qat_api.ko)
689 qat_api
15 1 0xffffffff84804000 11240 qat_c2xxx.ko (/boot/kernel/qat_c2xxx.ko)
696 pci/qat_c2xxx
[23.01-RELEASE][root@pfSense.home.lan]/root: dmesg | grep qat
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat_ocf0: <QAT engine>
qat_ocf0: no QAT IRQ instances available
device_attach: qat_ocf0 attach returned 6
qat0: <Intel c6xx QuickAssist> mem 0xf1800000-0xf183ffff,0xf1840000-0xf187ffff irq 21 at device 9.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1880000-0xf18bffff,0xf18c0000-0xf18fffff irq 25 at device 10.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
qat0: <Intel c6xx QuickAssist> mem 0xf1900000-0xf193ffff,0xf1940000-0xf197ffff irq 29 at device 11.0 on pci0
qat0: Unable to find AER capability of the device
device_attach: qat0 attach returned -19
[21:44 dallas ~]# dmesg | grep pciback
[ 0.000000] Command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0)
[ 1.811384] Kernel command line: root=LABEL=root-bhmxgw ro nolvm hpet=disable console=hvc0 console=tty0 quiet vga=785 splash plymouth.ignore-serial-consoles xen-pciback.hide=(0000:c5:00.0)(0000:c6:00.0)(0000:c7:00.0)
[ 4.419889] pciback 0000:c5:00.0: seizing device
[ 4.419895] pciback 0000:c6:00.0: seizing device
[ 4.419900] pciback 0000:c7:00.0: seizing device
[ 4.771730] pciback 0000:c7:00.0: enabling device (0140 -> 0142)
[ 4.771804] pciback 0000:c7:00.0: %(#e41111)[Disabling System Error reporting on root port c2:04.0]
[ 4.876894] pciback 0000:c6:00.0: enabling device (0140 -> 0142)
[ 4.984902] pciback 0000:c5:00.0: enabling device (0140 -> 0142)
[ 5.092811] xen_pciback: backend is vpci
c5:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
Subsystem: Intel Corporation QuickAssist Adapter 8960
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 2: Memory at fb540000 (64-bit, non-prefetchable) [size=256K]
Region 4: Memory at fb500000 (64-bit, non-prefetchable) [size=256K]
Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [60] MSI-X: Enable- Count=17 Masked-
Vector table: BAR=2 offset=0003b000
PBA: BAR=2 offset=0003b800
Capabilities: [6c] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [74] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [140 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
VF offset: 8, stride: 1, Device ID: 37c9
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000fb590000 (64-bit, non-prefetchable)
Region 2: Memory at 00000000fb580000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1b0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pciback
c6:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
Subsystem: Intel Corporation QuickAssist Adapter 8960
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 17
Region 2: Memory at fb440000 (64-bit, non-prefetchable) [size=256K]
Region 4: Memory at fb400000 (64-bit, non-prefetchable) [size=256K]
Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [60] MSI-X: Enable- Count=17 Masked-
Vector table: BAR=2 offset=0003b000
PBA: BAR=2 offset=0003b800
Capabilities: [6c] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [74] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [140 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
VF offset: 8, stride: 1, Device ID: 37c9
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000fb490000 (64-bit, non-prefetchable)
Region 2: Memory at 00000000fb480000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1b0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pciback
c7:00.0 Co-processor: Intel Corporation C62x Chipset QuickAssist Technology (rev 04)
Subsystem: Intel Corporation QuickAssist Adapter 8960
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 18
Region 2: Memory at fb340000 (64-bit, non-prefetchable) [size=256K]
Region 4: Memory at fb300000 (64-bit, non-prefetchable) [size=256K]
Capabilities: [b0] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [60] MSI-X: Enable- Count=17 Masked-
Vector table: BAR=2 offset=0003b000
PBA: BAR=2 offset=0003b800
Capabilities: [6c] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [74] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <128ns, L1 <1us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [138 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [140 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
IOVSta: Migration-
Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
VF offset: 8, stride: 1, Device ID: 37c9
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000fb390000 (64-bit, non-prefetchable)
Region 2: Memory at 00000000fb380000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Capabilities: [1b0 v1] Access Control Services
ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
Kernel driver in use: pciback
Also the card is visible via xl pci-assignable-list
after i passed it to pfsense. not sure if it is expected or not