Nvidia Quadro P400 not working on Ubuntu server via GPU/PCIe passthrough
-
Hi,
FYI, I just ordered a card for our lab, so we'll be able to try to reproduce the issue on our side
-
@olivierlambert Great let me know how it goes! As I have bought an WX2100 but unfortunately this one cannot be used for transcoding in plex... So I have to get back to Proxmox again with the Quadro P400.
-
Now back on Proxmox, although I also had some RMINIT errors on here, but these were related to the Hypervisor which were resolved pretty quick.
Also on Proxmox I have to change some Grub parameters and such, isn't this something that has to be done on Xen as well? And then on hypervisor level?
-
Like what parameter exactly? I think Xen doesn't support yet hiding the hypervisor information.
-
Parameters such as these in /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
GRUB_CMDLINE_LINUX="textonly video=astdrmfb video=efifb:off"Also someone replied to the topic I created on Nvidia forums:
https://forums.developer.nvidia.com/t/xcp-ng-ubuntu-vm-error-quadro-p400/199084 -
That answer is incorrect. Passing an entire PCIe device shouldn't make a diff.
Maybe it's a problem on the IOMMU side, I don't know. It will be easier to work on it with an actual card.
-
@olivierlambert In that case, lets hope you can resolve it once the P400 is delivered. Curious what you find and how there is a way to resolve the issue...
-
I have no idea and not great hope due to our other priorities but we'll see.
-
@olivierlambert I think the conclusion we can make is that I need to hide the Hypervisor from the Nvidia driver which is also mentioned here: https://www.reddit.com/r/XenServer/comments/r12p0q/pci_passthrough_quadro_p400_to_ubuntucentos_vm/
So it is an Error 43, as I think this is plausible as for Proxmox I do this as well by adding this into the vm .conf file
cpu: host,hidden=1,flags=+pcid
Which hides the hypervisor from the VM in KVM perspective.
Is there an equivalent for XCP-NG? -
- No equivalent in Xen yet
- Nvidia changed its policty recently to avoid blocking virt in their drivers. So the problem should not be here.
-
@olivierlambert Could it be something with the 'VFIO' modules maybe in KVM? I honestly have no clue anymore... So I think my best guess is to wait your research on this out...
-
I just got the card, but my agenda is very very busy ATM. I'll try to do the PCI passthrough on my spare time (which is not very often either)
-
@olivierlambert No problem take your time, I will check in regularly to see if there has been an update or some sorts...
-
Okay so doing some tests now, I can reproduce the issue. So the questions are:
- Did it work before? (older versions of XCP-ng?) -> removing regressions from the equation
- Is P400 limited for PCI passthough by NV driver? It's still not clear. If it's the problem, this require a code change in upstream Xen to be able to hide it.
-
I have no idea if it worked on the earlier versions of XCP-NG, I don't think so as I have I believe tested this on XCP-NG 8.0 and 8.1 if I remember correctly. (Also created forum posts about this in 2019/2020).
I dont think it is limited for PCI passthrough as I am using an NVidia driver on the VM within proxmox without any issues.
I am pasting a screenshot of the Nvidia driver I am currently using on the VM inside proxmox:
-
That might be because Proxmox is hiding the hypervisor underneath. Hard to tell because of this fracking drivers
-
@olivierlambert
Hmm yeah its quite a hassle with these drivers for some reason.... If you need some extra information which could help let me know... I can send some other details which are now on the proxmox host and the Ubuntu VM?How about the VFIO modules which I also mentioned earlier? Is this something that has to be added to XCP-NG maybe? As I also have a topic on Reddit and this person has the same problem but then with an T400.
https://www.reddit.com/r/XenServer/comments/r12p0q/pci_passthrough_quadro_p400_to_ubuntucentos_vm/hrlaqxl/?context=3 -
If it's hypervisor detection, the "only" thing needed is a Xen modification, but this is not trivial (if it's really that). I can assume it's the case.
In the meantime, can you double check if XCP-ng 7.6 is affected too? (last hope to check if it's not a regression).
-
@olivierlambert Can try that on my spare server, will try and see if I can do it today. I will update this once I am finished.
-
Currently I have no time to test this as the machine itself is also heavily used by other users.... But I believe the 7.6 version has this issue as well, as I remember testing this on version 7.x.
