Issues with PCIE Passthrough

olivierlambert

Don't mix between vGPU and PCI passthrough. It might be the issue. Please only use the CLI. vGPU and PCI passthrough are different beasts. Focus on https://docs.xcp-ng.org/compute/#pci-passthrough only, be sure to have nothing vGPU related enabled anywhere.

planedrop

You need to attach the GPU to the VM via CLI, rather than using the vGPU option, that is an entirely different thing than passthrough (vGPU, in short, lets you slice certain GPUs so more than 1 VM can use a single GPU, it's cool but very limited right now).

Follow the docs like Olivier mentioned and things should go smoothly. I've passed through GPUs a number of times now and they work quite well. Only thing to note is that some GPUs will show their audio devices as an additional PCIe sub device (IDK the right terminology) and you have to make sure to pass the entire device through, not just the GPU portion.

Looks like Intel doesn't do this, but with NVidia for example, you'd have 84:00.1 (possibly more, like .2 etc...) and you have to be sure all are passed through. At least I had to, to get mine working.

ImThatFluffy

@planedrop I somehow just misread a bunch of stuff. PCIE pass through seems to be working okay.

But the device is not showing up in the /dev/dri directory and I cant figure out why?
Any help would be appreciated.

planedrop

@ImThatFluffy are you sure it's working as it should then? I believe it should show up, I'll try some testing in my lab but if you passed through correctly then I think it should show up in /dev/dri just fine.

ImThatFluffy

@planedrop It shows up when I run LSCPI


00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Cirrus Logic GD 5446
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:05.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05)

planedrop

@ImThatFluffy Maybe try running a GPU load on it and installing Nvidia SMI to see if the GPU is being used?

https://developer.nvidia.com/nvidia-system-management-interface

ImThatFluffy

@planedrop I cant get the GPU to pull anything without it being in /dev/dri/ & its a intel ARC GPU SMI is going to be pretty pointless AFIAIK.

planedrop

@ImThatFluffy Oh yes, sorry, forgot this was an ARC GPU.

Have you tried getting the ARC drivers installed? I imagine they should work if it shows up in lspci.

I've never passed an ARC GPU through though so it's possible there's something additional needed to get it going.

ImThatFluffy

@planedrop All the correct drivers are loaded in the VM

:/home/debian# sudo apt list --installed | grep intel

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

intel-igc-core/now 1.0.15985.7 amd64 [installed,local]
intel-igc-opencl/now 1.0.15985.7 amd64 [installed,local]
intel-media-va-driver-non-free/stable,now 23.1.1+ds1-1 amd64 [installed]
intel-microcode/stable,stable-security,now 3.20231114.1~deb12u1 amd64 [installed]
intel-opencl-icd/now 24.05.28454.6 amd64 [installed,local]
libdrm-intel1/stable,stable,now 2.4.114-1+b1 amd64 [installed,automatic]

olivierlambert

Are you sure your device doesn't have another PCI address? (like some graphic cards got the GPU at a PCI address and the HDMI/sound to another, if you don't pass both the driver will fail to load)

ImThatFluffy

@olivierlambert

LSPCI on the host shows this

84:00.0 VGA compatible controller: Intel Corporation Device 56a5 (rev 05)
85:00.0 Audio device: Intel Corporation Device 4f92

Im currently only passing 84:00.0 through I will attempt to pass 85:00.0 through as well and re update.

olivierlambert

Yes, please hide both from the dom0 and pass both to the guest, start again and I think that might work

ImThatFluffy

@olivierlambert

Passed 84 and 85 into the VM and still no luck.

 lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
00:02.0 VGA compatible controller: Device 1234:1111
00:03.0 SCSI storage controller: XenSource, Inc. Xen Platform Device (rev 01)
00:05.0 Audio device: Intel Corporation DG2 Audio Controller
00:06.0 VGA compatible controller: Intel Corporation DG2 [Arc A380] (rev 05)

ls -l /dev/dri
total 0
drwxr-xr-x 2 root root      60 Feb 16 11:44 by-path
crw-rw---- 1 root video 226, 0 Feb 16 11:44 card0

Re installed

intel-level-zero-gpu
intel-media-va-driver-non-free
intel-opencl-icd
libdrm-intel1

olivierlambert

Hmm that's weird Anyone else around using an ARC card in passthrough?

john.c

@ImThatFluffy Have you installed the firmware for the card that your proposing to pass through? That way you can be sure that the card is fully functional.

I read in an article about getting the card working written during 2023 that it requires the firmware to be installed (present) on the Linux system along with the compatible drivers.

So you may be missing the firmware also is all parts of the Intel Arc drivers fully installed on the system for the device to fully function?

ImThatFluffy

@john-c

Yea im not sure, it was either an issue with the way I had Debian setup or compatibility things booted up Ubuntu 22.04LTS with the HWE kernel and it worked perfectly.

john.c

@ImThatFluffy said in Issues with PCIE Passthrough:

@john-c

Yea im not sure, it was either an issue with the way I had Debian setup or compatibility things booted up Ubuntu 22.04LTS with the HWE kernel and it worked perfectly.

Well if you are using Ubuntu Linux 22.04.1 LTS or one of the later point releases then it would be using a Linux Kernel version 6.1 or later, when its a HWE kernel. So any bugs from earlier versions of the kernel would have been fixed, also the Intel ARC graphics hardware would have been released during one of the point releases. On the Debian Linux front a distribution version earlier than version 12.0 would have been unlikely to have complete properly functioning support, due to that release being the first one with the Linux kernel version of 6.1 or later.