PCIe Pass-through lanes and lane performance
-
So I ordered some mini DP dummy plugs and took a look at this further today.
First thought...Just load Windows on the bare metal and see what the card does.
No problem. Loaded Win10, loaded the latest drivers, PCIe Gen4x8 lanes. Sweet. At least that works.
Put the XCP-ng drive back in, booted up, passed-through the GPU to a Win11 guest, updated the drivers, PCIe Gen1x1.
As I explained briefly in https://xcp-ng.org/forum/post/106479, this is mostly display as the real hardware may be running at a different speed.
Ultimately I need to install a Linux distro with a really current kernel on the bare metal and see how it sees the GPU. That might be a good project for tomorrow.
I think this just comes down to a combination of too new, not well supported hardware with some quirks and somewhat dated code and support in XCP-ng (ReBAR support in guests for example).
I did quick checks and I believe that ReBAR works; at least with UEFI guests (our OVMF build does support resizable bar), and it actually works as suggest https://xcp-ng.org/forum/post/106489 (VRAM BAR is 16 GB); but I didn't do extensive testing in that area.
-
Alright....I might be seeing something here...From the CLI of the XCP-ng host itself...
Here's the lspci tree view of interest:
+-[0000:80]-+-00.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex | +-00.2 Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU | +-01.0 Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge | +-01.1-[81-84]----00.0-[82-84]--+-01.0-[83]----00.0 Intel Corporation Device e212 | | \-02.0-[84]----00.0 Intel Corporation Device e2f7So PCI slots 81-84 are what I'm interested in...Digging in on 81:
lspci -vv -s 0000:81:00.0 81:00.0 PCI bridge: Intel Corporation Device e2ff (rev 01) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 56 Region 0: Memory at 20003000000 (64-bit, prefetchable) [size=8M] Bus: primary=81, secondary=82, subordinate=84, sec-latency=0 Memory behind bridge: f7c00000-f7efffff Prefetchable memory behind bridge: 000001f800000000-0000020002ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Upstream Port, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed unknown, Width x8, ASPM L1, Exit Latency L0s <4us, L1 <32us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 16GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [158 v1] #19 Capabilities: [178 v1] #26 Capabilities: [1a0 v1] #27 Capabilities: [1c8 v1] #2a Capabilities: [1fc v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [204 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=10us PortTPowerOnTime=14us Capabilities: [220 v1] Vendor Specific Information: ID=0002 Rev=4 Len=100 <?> Capabilities: [320 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?> Capabilities: [358 v1] #25 Kernel driver in use: pcieportSpecifically there I see the correct width and speed of the PCIe interface for the card.
Nothing shows up for 82 (I presume that's a bridge to get to the functions on the card)
83 is the actual graphics processor itself:
lspci -vv -s 0000:83:00.0 83:00.0 VGA compatible controller: Intel Corporation Device e212 (prog-if 00 [VGA controller]) Subsystem: Intel Corporation Device 1114 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Region 0: Memory at 20002000000 (64-bit, prefetchable) [disabled] [size=16M] Region 2: Memory at 1f800000000 (64-bit, prefetchable) [disabled] [size=16G] Expansion ROM at f7c00000 [disabled] [size=2M] Capabilities: [40] Vendor Specific Information: Len=0c <?> Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [d0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 0 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [110 v1] #00 Capabilities: [200 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [420 v1] #15 Capabilities: [220 v1] #24 Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- Initial VFs: 2, Total VFs: 2, Number of VFs: 0, Function Dependency Link: 00 VF offset: 1, stride: 1, Device ID: e212 Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 0000020000000000 (64-bit, prefetchable) Region 2: Memory at 000001fc00000000 (64-bit, prefetchable) VF Migration: offset: 00000000, BIR: 0 Capabilities: [400 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Kernel driver in use: pcibackAnd 84 is the audio portion of the GPU...
lspci -vv -s 0000:84:00.0 84:00.0 Audio device: Intel Corporation Device e2f7 Subsystem: Intel Corporation Device 1114 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Region 0: Memory at f7e00000 (64-bit, non-prefetchable) [disabled] [size=16K] Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=55mA PME(D0-,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [c0] Vendor Specific Information: Len=14 <?> Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [80] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [140 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0nsAs far as XO (from sources) is concerned, I can turn on pass-through for 83, but it doesn't show me 81, 82, or 84.
Should I just manually block off 81-84 and pass all of those devices to the guest VM?
-
Argh...Correction... 82 shows up as 82:01 and 82:02...
lspci -vv -s 0000:82:01.0 82:01.0 PCI bridge: Intel Corporation Device e2f0 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=83, subordinate=83, sec-latency=0 Memory behind bridge: f7c00000-f7dfffff Prefetchable memory behind bridge: 000001f800000000-0000020002ffffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+ LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieportAnd 82:02...
lspci -vv -s 0000:82:02.0 82:02.0 PCI bridge: Intel Corporation Device e2f1 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=82, secondary=84, subordinate=84, sec-latency=0 Memory behind bridge: f7e00000-f7efffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [94] Subsystem: Intel Corporation Device 0000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Capabilities: [220 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Kernel driver in use: pcieport -
Before you go further down the bridge tree, redakula called this one early in the thread and it's worth rereading.
Intel Arc cards put an internal PCIe bridge between the slot and the GPU, and
lspcion Linux only reports the top of that chain. His own A310 shows PCIe x8 4.0 at x4 4.0 in GPU-Z under Windows while Linux insists it is x1. That is the same bridge topology you're now staring at on 81, 82 and 83.TeddyAstie said much the same when he called it mostly a display issue, and he later checked that resizable BAR does work on UEFI guests since the OVMF build supports it. So the Gen1x1 reading may just be
lspcidescribing the bridge rather than the card. I could easily be wrong, and the people already in this thread know this hardware far better than I do.
-
I'm in agreement with all of this. I can clearly see the Intel card starting at slot 81 and ending at 84. Unfortunately, in XCP-ng 8.3, only 83 (the "gpu" itself) is visible as an assignable PCI pass-through device. When I pass through just slot 83, I get a dysfunctional result.
Again...I'm a little out of my expertise here, but my assumption here is that in order to actually get the full GPU to work, I need to get slots 81-84 passed over to the VM.
Problem...In XCP-ng 8.2, you could mask off PCI devices with the pciback.hide function:
/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:04:01.0)(0000:00:19.0)" Or in my case: /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:81:00.0)(0000:82.01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)"Then you would assign those same PCI devices to the guest VM.
In 8.3 you do this with an xe command:
xe pci-list and xe pci-disable-dom0-access uuid=<pci uuid>The bridge devices for the Intel card aren't being included in the pci-list of XCP-ng and are therefore not assignable to guests. For example, running "xe pci-list" on a host shows me this single entry relevant to the Intel card:
uuid ( RO) : f3e6842b-ad1b-9e93-dbea-214dde28618a vendor-name ( RO): Intel Corporation device-name ( RO): Device e212 pci-id ( RO): 0000:83:00.0With the new 8.3 method of assigning PCI resources to guest VM's, I don't see how I can fully pass-through the Intel GPU to a guest. I won't necessarily call this a "bug" but it is an unexpected and perhaps edge case behavior that Vates didn't expect.
-
Also a question for @Team-XAPI-Network
-
@JamesG You can still manually use
xen-cmdlineto pass through 81-84 to the VM on 8.3 (this is in fact whatxe pci-disable-dom0-accessdoes behind the scenes).I'll try to see why other devices are omitted and how this can be fixed.
-
@JamesG Could you pass through 81-84 manually and confirm that it actually fixes the issues for you?
-
@andriy.sultanov Absolutely. I wasn't sure if the old way was incompatible with the new way or not.
So the old 8.2 method will still work without conflict in 8.3??
If so, I'll have an answer for you shortly.
-
@JamesG it should be compatible, yes.
you can check the current command-line value with
/opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hideand add all 81-84 devices with--set-dom0 -
@andriy.sultanov First problem...
/opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide xen-pciback.hide=(0000:81:00.0)(0000:82.01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)That looks good...
xl pci-assignable-list libxl: error: libxl_pci.c:444:libxl_device_pci_assignable_list: Looks like pciback driver not loadedNot so much...
I did go ahead and assign those PCI devices to the guest, but of course it won't start, complaining that the pciback driver isn't loaded.
-
Nevermind...Typo...Rebooting now to see...
-
Argh...
Okay...First up:
lspci:
81:00.0 PCI bridge: Intel Corporation Device e2ff (rev 01)
82:01.0 PCI bridge: Intel Corporation Device e2f0
82:02.0 PCI bridge: Intel Corporation Device e2f1
83:00.0 VGA compatible controller: Intel Corporation Device e212
84:00.0 Audio device: Intel Corporation Device e2f7Execute the hide:
/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:81:00.0)(0000:82:01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)"
Reboot and verify the hide:
xl pci-assignable-list
0000:84:00.0
0000:82:01.0
0000:83:00.0
0000:81:00.0
0000:82:02.0Looks good....make the assignment:
xe vm-param-set other-config:pci=0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0 uuid=9d76f849-16bf-0414-9072-6904176bbb01
Verify:
xe vm-param-list uuid=9d76f849-16bf-0414-9072-6904176bbb01
other-config (MRW): pci: 0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0; import_task: OpaqueRef:9e15d1aa-5e79-79d9-af81-4dc299725217; xo:9d76f849: {"creation":{"date":"2025-12-16T19:19:14.724Z","template":"1f8728f9-6354-4a68-adc3-222c34f2bb94","user":"ae9cf9c9-bbcc-4dc9-b1bb-d9fb1b657c60"}}; base_template_name: Windows 11; mac_seed: 5891089b-e9ec-3050-30e5-564db3e6e170; install-methods: cdrom
Starting up the VM results in this:
"params": [ "xenopsd internal error: Cannot_add(0000:82:02.0, Device_common.QMP_Error(1, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"Failed to initialize 11/15, type = 0x1, rc: -1\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000006-1\\\"}\"))" ],I'm gonna strip it all back, reboot and start over.
-
Yep...
Stripped it all out....Rebooted...
Set the pcihideback for dom0...Rebooted.
Verified all devices were assignable.
Assigned to guest VM, booted...No go.
Initially the log said it couldn't assign 82:02.0
So I removed it.
Then it couldn't assign 82:01.0
So I removed it.
Then it couldn't assign 81:00.0
So I removed it.Now the VM works...But we're effectively back to just passing through the GPU itself, and I guess the audio device.
No functional change.
-
You can't PCI Passthrough a PCI bridge (and even if you could, it wouldn't be very useful), as it's a part of PCI internals and not really something the guest requires to use the device.
As I asked previously, is there any issue (aside display like PCIe 1.0 1x) with the device, is it behavior correctly or having the expected performance ?
-
@TeddyAstie From what I've tried...It won't work at all in Linux.
It kind of works in Windows, and while I do see some load on the GPU when running something, there's as much if not more load on the CPU. It's not what I expected at least.
I've never really gamed on PC, and definitely never Linux. My only uses for GPU's have been in media composition which was always on dedicated systems and Nvidia GPUs. More recently I've been doing some AI tasks, again with passed-through NVidia GPUs.
Side tangent...
I ran into another workload last night where the app wanted OpenGL...But OpenGL doesn't work with the "normal" VGA adapter in XCP-ng, and I don't have a GPU on that particular pool to try a pass-through. That lead me to VirGL, but it doesn't look like that's gone anywhere.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login