GPU Passthrough

gb.123

In the new Xen Orchestra there are two places where passthrough can be used in 'Advanced Section'

Here :

When you click on the (+) Sign

And this section :

Which one should be used for passthrough ? Or can either be used ? Or both should be used ?

tjkreidl

@gb.123 You need one video card for your administrative console and another can be used for GPU passthrough. There must be two separate physical devices.
So make sure you have two video boards, one that has the GPU capabilities you want to use in your passthrough configuration.

gb.123

@tjkreidl
I do have one iGPU & another RTX 4060. I was just confirming which option to use

tjkreidl

@gb.123 Ah, OK. then the more powerful GP is the RTX4060, right? If so, use it for the passthrough. Also, on some CPUs you have to do a BIOS setting to allow this to work, because of memory limitations, but only on probably much older systems, if I recall correctly.

gb.123

@tjkreidl

Still struggling to make it work. Simple passthrough is 'hanging' the VM and the VM gets stuck at splash screen.

Trying to figure out what to do

Do you know if I need to put the card in attach pci or just select it in the gpu section or both ?

tjkreidl

@gb.123 YOu are trying to do passthrough to a specific VM? I don't think that used to be supported, but maybe is now.
Are NVIDIA drivers installed on the VM, as needed?
Sorry, it's been a while since doing this so I'm digging back into my memory.
Also, is IOMMU supported and enabled in the BIOS?
Also, check this out and see if it may be of some help:
https://www.youtube.com/watch?v=_JPmxmxqhds

gb.123

@olivierlambert

Can you please confirm if I need to attach PCI AND select the GPU in the upper section or either should work ?

tjkreidl

@gb.123 You need to do both. After adding the PCI device, you might also need to specifically enable "passthrough" for that device within the VM's settings. It may do it automatically when you add it.
Make sure the appropriate NVIDIA driver is also installed on the the VM.

gb.123

@tjkreidl

You are trying to do passthrough to a specific VM? I don't think that used to be supported, but maybe is now.

Yes. 1 Card -> 1 VM. Nvidia opened the passthrough lock in 2021. (Before that it was locked)

Are NVIDIA drivers installed on the VM, as needed?

I am installing them(tried the open one), but they are causing issues.

Also, is IOMMU supported and enabled in the BIOS?

Yes

gb.123

@tjkreidl said in GPU Passthrough:

After adding the PCI device, you might also need to specifically enable "passthrough" for that device within the VM's settings. It may do it automatically when you add it.

This is one of my confusions. It is not automatically adding it. Whether I should select it or leave it blank is the confusion.

gb.123

@tjkreidl

Once I install graphics drivers (nvidia) and allow pci passthrough of the graphics card, i get:

NVRM: cpuidInfoAMD Unrecognised AMD processor in cpuInfoAMD

and the VM refuses to boot

If I remove the drivers or detach the card, then the VM boots

tjkreidl

@gb.123 Are you sure you have the correct drivers installed? Also, check if the GPU is compatible with AMD CPUs -- I only had Intel CPUs, so am not sure if that's an issue or not.

gb.123

@tjkreidl

Thanks so much for your response !
I agree with you that the drivers may be a problem. I am starting again from scratch and re-installing everything.
Though a question does remain that whether we need to use both options (Attach PCI + GPU Selection) or either option for GPU pass through. Maybe someone in @vates-tech-support can clarify exactly what the option in Xen Orchestra means.

From what I know, the GPU option in the upper menu wasn't there a few version before. Seems to be a new entry.

tjkreidl

@gb.123 I'm sure you can also find some NVIDIA "hoe to" guides that might be helpful. As mentioned before, I've only done server passthrough so that all VMs would get access,
so sorry I can't provide more specifics. You can always try with one or the other option and add the other if it still doesn't work. I'm pretty sure, though, that you do need both enabled.
Keep us posted!

gb.123

@tjkreidl
Thanks once again for your help and guidance!
I have seen/read many 'how to' videos/docs. The problem was not the method I was using. I managed to get this working albeit there is a bug in XCG-ng I suppose.

If USB Keyboard & Mouse is passed-through along-with GPU:
The GPU gets stuck in D3 state (Classic GPU reset problem)
If no vUSB is passed but GPU is passed through:
The GPU works correctly and resets correctly

tjkreidl

@gb.123 Interesting -- alert the XCP-ng team to take a closer look, if they haven't seen this already.

forbiddenera

@gb.123 what xen and gurst version etc were you using?

I'm about to try GPU PT and having USB st the same time is important

Vagrantin

Hello

Hope this help here is how I manage to configure GPU Passthrough.

In my case I have follow the documentation on XCP-ng website : https://docs.xcp-ng.org/compute/

To sum up with XOA 8.3 here is what I did:

ssh to XCP host, to identify the GPU PCI and run lspci to find my GPU.

[root@xen ~]# lspci
...
0f:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1)

Then disable the GPU for being used by XCP-ng itself( Dom0) using the XCP-ng uuid

[root@xen ~]# xe pci-list
uuid ( RO)           : 7970825f-dd1c-c270-93a3-6761f00e1362
    vendor-name ( RO): NVIDIA Corporation
    device-name ( RO): GA102 [GeForce RTX 3090]
         pci-id ( RO): 0000:0f:00.0

xe pci-disable-dom0-access uuid=<pci uuid>

3 reboot the XCP-ng host

4 Then from XOA attached the PCI related to my GPU to my guest VM.

5 install official Nvidia drivers ( closed source ) and have then the card available in the guest.

What I didn't do is adding a vGPU in the guest configuration:

I did it some times ago, also I don't recall having to do any specific configuration in the Bios to make this work.

PS: I have an issue, I don't have output to the Display port or HDMI of this video card, which is not a blocker for me as that's not my immediate needs.

Best.

gskger

I am running NVIDIA P40s on a DELL R720/R730 and a NVIDIA A2000 12GB on a DELL Optiplex 9010 in my playlab. Getting the GPUs to be accessible for VMs was quite easy: shutdown the XCP-ng host, install the GPUs, start the host again, activate the GPU in the advanced settings of the host in the PCI Devices section (“Passthrough enabled”). The host will restart at this point after displaying a warning. That's it – the GPU can now be assigned to a VM in the VMs advanced settings. I do use the GPUs for AI workloads under Debian though, so your use case might vary.