PCIe Pass-through lanes and lane performance

JamesG

Argh...Correction... 82 shows up as 82:01 and 82:02...

lspci -vv -s 0000:82:01.0
82:01.0 PCI bridge: Intel Corporation Device e2f0 (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=82, secondary=83, subordinate=83, sec-latency=0
        Memory behind bridge: f7c00000-f7dfffff
        Prefetchable memory behind bridge: 000001f800000000-0000020002ffffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd+
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [94] Subsystem: Intel Corporation Device 0000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [220 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
        Kernel driver in use: pcieport

And 82:02...

lspci -vv -s 0000:82:02.0
82:02.0 PCI bridge: Intel Corporation Device e2f1 (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=82, secondary=84, subordinate=84, sec-latency=0
        Memory behind bridge: f7e00000-f7efffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Downstream Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag+ RBE+
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #8, Speed 2.5GT/s, Width x1, ASPM L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported ARIFwd+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [94] Subsystem: Intel Corporation Device 0000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [220 v1] Access Control Services
                ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans+
                ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
        Kernel driver in use: pcieport

poddingue

Before you go further down the bridge tree, redakula called this one early in the thread and it's worth rereading.

Intel Arc cards put an internal PCIe bridge between the slot and the GPU, and lspci on Linux only reports the top of that chain. His own A310 shows PCIe x8 4.0 at x4 4.0 in GPU-Z under Windows while Linux insists it is x1. That is the same bridge topology you're now staring at on 81, 82 and 83.

TeddyAstie said much the same when he called it mostly a display issue, and he later checked that resizable BAR does work on UEFI guests since the OVMF build supports it. So the Gen1x1 reading may just be lspci describing the bridge rather than the card. I could easily be wrong, and the people already in this thread know this hardware far better than I do.

JamesG

@poddingue

I'm in agreement with all of this. I can clearly see the Intel card starting at slot 81 and ending at 84. Unfortunately, in XCP-ng 8.3, only 83 (the "gpu" itself) is visible as an assignable PCI pass-through device. When I pass through just slot 83, I get a dysfunctional result.

Again...I'm a little out of my expertise here, but my assumption here is that in order to actually get the full GPU to work, I need to get slots 81-84 passed over to the VM.

Problem...In XCP-ng 8.2, you could mask off PCI devices with the pciback.hide function:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:04:01.0)(0000:00:19.0)"

Or in my case:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:81:00.0)(0000:82.01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)"

Then you would assign those same PCI devices to the guest VM.

In 8.3 you do this with an xe command:

xe pci-list

and

xe pci-disable-dom0-access uuid=<pci uuid>

The bridge devices for the Intel card aren't being included in the pci-list of XCP-ng and are therefore not assignable to guests. For example, running "xe pci-list" on a host shows me this single entry relevant to the Intel card:

uuid ( RO)           : f3e6842b-ad1b-9e93-dbea-214dde28618a
    vendor-name ( RO): Intel Corporation
    device-name ( RO): Device e212
         pci-id ( RO): 0000:83:00.0

With the new 8.3 method of assigning PCI resources to guest VM's, I don't see how I can fully pass-through the Intel GPU to a guest. I won't necessarily call this a "bug" but it is an unexpected and perhaps edge case behavior that Vates didn't expect.

olivierlambert

Also a question for @Team-XAPI-Network

andriy.sultanov

@JamesG You can still manually use xen-cmdline to pass through 81-84 to the VM on 8.3 (this is in fact what xe pci-disable-dom0-access does behind the scenes).

I'll try to see why other devices are omitted and how this can be fixed.

andriy.sultanov

@JamesG Could you pass through 81-84 manually and confirm that it actually fixes the issues for you?

JamesG

@andriy.sultanov Absolutely. I wasn't sure if the old way was incompatible with the new way or not.

So the old 8.2 method will still work without conflict in 8.3??

If so, I'll have an answer for you shortly.

andriy.sultanov

@JamesG it should be compatible, yes.

you can check the current command-line value with /opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide and add all 81-84 devices with --set-dom0

JamesG

@andriy.sultanov First problem...

/opt/xensource/libexec/xen-cmdline --get-dom0 xen-pciback.hide
xen-pciback.hide=(0000:81:00.0)(0000:82.01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)

That looks good...

xl pci-assignable-list
libxl: error: libxl_pci.c:444:libxl_device_pci_assignable_list: Looks like pciback driver not loaded

Not so much...

I did go ahead and assign those PCI devices to the guest, but of course it won't start, complaining that the pciback driver isn't loaded.

JamesG

Nevermind...Typo...Rebooting now to see...

JamesG

Argh...

Okay...First up:

lspci:

81:00.0 PCI bridge: Intel Corporation Device e2ff (rev 01)
82:01.0 PCI bridge: Intel Corporation Device e2f0
82:02.0 PCI bridge: Intel Corporation Device e2f1
83:00.0 VGA compatible controller: Intel Corporation Device e212
84:00.0 Audio device: Intel Corporation Device e2f7

Execute the hide:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:81:00.0)(0000:82:01.0)(0000:82:02.0)(0000:83:00.0)(0000:84:00.0)"

Reboot and verify the hide:

xl pci-assignable-list
0000:84:00.0
0000:82:01.0
0000:83:00.0
0000:81:00.0
0000:82:02.0

Looks good....make the assignment:

xe vm-param-set other-config:pci=0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0 uuid=9d76f849-16bf-0414-9072-6904176bbb01

Verify:

xe vm-param-list uuid=9d76f849-16bf-0414-9072-6904176bbb01

other-config (MRW): pci: 0/0000:81:00.0,0/0000:82:01.0,0/0000:82:02.0,0/0000:83:00.0,0/0000:84:00.0; import_task: OpaqueRef:9e15d1aa-5e79-79d9-af81-4dc299725217; xo:9d76f849: {"creation":{"date":"2025-12-16T19:19:14.724Z","template":"1f8728f9-6354-4a68-adc3-222c34f2bb94","user":"ae9cf9c9-bbcc-4dc9-b1bb-d9fb1b657c60"}}; base_template_name: Windows 11; mac_seed: 5891089b-e9ec-3050-30e5-564db3e6e170; install-methods: cdrom

Starting up the VM results in this:

  "params": [
    "xenopsd internal error: Cannot_add(0000:82:02.0, Device_common.QMP_Error(1, \"{\\\"error\\\":{\\\"class\\\":\\\"GenericError\\\",\\\"desc\\\":\\\"Failed to initialize 11/15, type = 0x1, rc: -1\\\",\\\"data\\\":{}},\\\"id\\\":\\\"qmp-000006-1\\\"}\"))"
  ],

I'm gonna strip it all back, reboot and start over.

JamesG

Yep...

Stripped it all out....Rebooted...

Set the pcihideback for dom0...Rebooted.

Verified all devices were assignable.

Assigned to guest VM, booted...No go.

Initially the log said it couldn't assign 82:02.0
So I removed it.
Then it couldn't assign 82:01.0
So I removed it.
Then it couldn't assign 81:00.0
So I removed it.

Now the VM works...But we're effectively back to just passing through the GPU itself, and I guess the audio device.

No functional change.

TeddyAstie

@JamesG

You can't PCI Passthrough a PCI bridge (and even if you could, it wouldn't be very useful), as it's a part of PCI internals and not really something the guest requires to use the device.

As I asked previously, is there any issue (aside display like PCIe 1.0 1x) with the device, is it behavior correctly or having the expected performance ?

JamesG

@TeddyAstie From what I've tried...It won't work at all in Linux.

It kind of works in Windows, and while I do see some load on the GPU when running something, there's as much if not more load on the CPU. It's not what I expected at least.

I've never really gamed on PC, and definitely never Linux. My only uses for GPU's have been in media composition which was always on dedicated systems and Nvidia GPUs. More recently I've been doing some AI tasks, again with passed-through NVidia GPUs.

Side tangent...
I ran into another workload last night where the app wanted OpenGL...But OpenGL doesn't work with the "normal" VGA adapter in XCP-ng, and I don't have a GPU on that particular pool to try a pass-through. That lead me to VirGL, but it doesn't look like that's gone anywhere.