PCIe Pass-through lanes and lane performance

[JamesG]

A curiosity here...

I've got two (identical) hosts in a pool (2nd Gen Epyc).

Host 1 has an NVidia Tesla P4 GPU passed-through to a VM.
Host 2 has an Intel B50 Pro GPU passed-through to a VM.

I've been attempting to get the Intel GPU running properly in a VM.

The NVidia GPU is PCIe Gen3 x16.
The Intel should be PCIe Gen5 x8, but should not have any issues running at PCIe Gen4 x8.

The NVidia GPU shows up as PCIe Gen3 x16 lanes.
The Intel GPU shows up as PCIe Gen1 x1 lane (and as such does not work).

What determines how many of what generation PCIe lanes gets passed through? Is XCP-ng just not sorting out the Intel GPU properly?

I'm pretty sure the BIOS settings are the same between the two hosts, but I'll see about doing a full comparison between the two.

Thanks!

James

[acebmxer]

@JamesG

Are these GPUs installed in actual servers or consumer motherboards? If the later is the intel gpu in the top most pcie slot?

[JamesG]

@acebmxer Supermicro server systems. The Eypc CPU's have 128 lanes of PCIe Gen4. The cards are inserted into appropriate slots.

I just updated the BIOS and BMC code on the server with the Intel GPU. I went through and re-enabled all the SR-IOV stuff and made sure ReBAR was enabled.

Still, GPU-Z reports that a Windows guest with the Intel GPU passed-through is linked up with a single lane of PCIe gen 1.

I suppose that could be cosmetic, however passing through the same Intel GPU to a Debian VM has issues when trying to initialize the card on boot and I can't get it to work there at all. It will at least do something in Windows.

[redakula]

@JamesG

The intel Arc GPUs have som weird internal pcie bridge setup that lspci shows only the top of so the actual speed is probably not x1...
https://www.intel.com/content/www/us/en/support/articles/000094587/graphics.html