Nvidia Quadro P400 not working on Ubuntu server via GPU/PCIe passthrough

TheFrisianClause

@olivierlambert I think the conclusion we can make is that I need to hide the Hypervisor from the Nvidia driver which is also mentioned here: https://www.reddit.com/r/XenServer/comments/r12p0q/pci_passthrough_quadro_p400_to_ubuntucentos_vm/

So it is an Error 43, as I think this is plausible as for Proxmox I do this as well by adding this into the vm .conf file

cpu: host,hidden=1,flags=+pcid

Which hides the hypervisor from the VM in KVM perspective.
Is there an equivalent for XCP-NG?

olivierlambert

No equivalent in Xen yet
Nvidia changed its policty recently to avoid blocking virt in their drivers. So the problem should not be here.

TheFrisianClause

@olivierlambert Could it be something with the 'VFIO' modules maybe in KVM? I honestly have no clue anymore... So I think my best guess is to wait your research on this out...

olivierlambert

I just got the card, but my agenda is very very busy ATM. I'll try to do the PCI passthrough on my spare time (which is not very often either)

TheFrisianClause

@olivierlambert No problem take your time, I will check in regularly to see if there has been an update or some sorts...

olivierlambert

Okay so doing some tests now, I can reproduce the issue. So the questions are:

Did it work before? (older versions of XCP-ng?) -> removing regressions from the equation
Is P400 limited for PCI passthough by NV driver? It's still not clear. If it's the problem, this require a code change in upstream Xen to be able to hide it.

TheFrisianClause

@olivierlambert

I have no idea if it worked on the earlier versions of XCP-NG, I don't think so as I have I believe tested this on XCP-NG 8.0 and 8.1 if I remember correctly. (Also created forum posts about this in 2019/2020).

I dont think it is limited for PCI passthrough as I am using an NVidia driver on the VM within proxmox without any issues.

I am pasting a screenshot of the Nvidia driver I am currently using on the VM inside proxmox:

olivierlambert

That might be because Proxmox is hiding the hypervisor underneath. Hard to tell because of this fracking drivers

TheFrisianClause

@olivierlambert
Hmm yeah its quite a hassle with these drivers for some reason.... If you need some extra information which could help let me know... I can send some other details which are now on the proxmox host and the Ubuntu VM?

How about the VFIO modules which I also mentioned earlier? Is this something that has to be added to XCP-NG maybe? As I also have a topic on Reddit and this person has the same problem but then with an T400.
https://www.reddit.com/r/XenServer/comments/r12p0q/pci_passthrough_quadro_p400_to_ubuntucentos_vm/hrlaqxl/?context=3

olivierlambert

If it's hypervisor detection, the "only" thing needed is a Xen modification, but this is not trivial (if it's really that). I can assume it's the case.

In the meantime, can you double check if XCP-ng 7.6 is affected too? (last hope to check if it's not a regression).

TheFrisianClause

@olivierlambert Can try that on my spare server, will try and see if I can do it today. I will update this once I am finished.

TheFrisianClause

Currently I have no time to test this as the machine itself is also heavily used by other users.... But I believe the 7.6 version has this issue as well, as I remember testing this on version 7.x.

TheFrisianClause

Alright tested it with 7.6
Seems to not work as well...

[  165.594038] [drm] [nvidia-drm] [GPU ID 0x00000006] Loading driver
[  165.594040] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:06.0 on minor 1
[  171.958377] NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x22:0x56:667)
[  171.958424] NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0
[  171.963805] NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x22:0x56:667)
[  171.963848] NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0

Same error....

olivierlambert

Okay so at least "it's good news": it's not a regression So I assume it's required to hide the hypervisor underneath.

TheFrisianClause

@olivierlambert Yes I believe so, do you need more information on this?

olivierlambert

Not yet, I need to talk with Xen devs when I can on how to achieve that.

TheFrisianClause

@olivierlambert Alright, thanks for the effort

warriorcookie

@olivierlambert just wanted to clarify a couple things:
-nvidia drivers support passthrough recently on windows guests only. Linux guests are still unsupported.
-while most quadro cards are supported for passthrough including P4000, the P400 is excluded from support. It's considered consumer grade (GP107 basically a striped down gtx1050)
-might be a given but for those that stumble on this thread, both the video device and audio device need to be passed through separately as they show as separate devices under lspci.

I'm confident this is an issue with the nvidia driver disabling when it see's the CPUID hypervisor present bit.

On my hardware (SM x9 with e5-2667v2) proxmox and esxi both worked flawlessly with near bare metal performance but you're able to change the CPUID hypervisor present bit.

On xcp-ng 8.2 with windows guest it works with no issues.
Once I install Linux as the guest then smi-nvidia produces "no devices were found" even though it shows with lspci.

I'm trying to migrate from esxi. I'm jumping in on this thread as this is a major issue for me that would prevent me switching. Unfortunate as this project ticks all the other boxes that vmware failed to....

olivierlambert

Okay so it's "just" a matter of hiding the CPUID then Thanks a lot for the detailed feedback @warriorcookie !

Let me push again on that feature request on my side, I'll see what we can do (and how much time/money to get a solution)

warriorcookie

Parralel work into what I think could be the solution, but stalled as of now: https://xcp-ng.org/forum/topic/4643/nested-virtualization-of-windows-hyper-v-on-xcp-ng/39