XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Nvidia Quadro P400 not working on Ubuntu server via GPU/PCIe passthrough

    Scheduled Pinned Locked Moved Compute
    106 Posts 8 Posters 36.0k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      If it's hypervisor detection, the "only" thing needed is a Xen modification, but this is not trivial (if it's really that). I can assume it's the case.

      In the meantime, can you double check if XCP-ng 7.6 is affected too? (last hope to check if it's not a regression).

      T 1 Reply Last reply Reply Quote 0
      • T Offline
        TheFrisianClause @olivierlambert
        last edited by

        @olivierlambert Can try that on my spare server, will try and see if I can do it today. I will update this once I am finished.

        T 1 Reply Last reply Reply Quote 0
        • T Offline
          TheFrisianClause @TheFrisianClause
          last edited by

          Currently I have no time to test this as the machine itself is also heavily used by other users.... But I believe the 7.6 version has this issue as well, as I remember testing this on version 7.x.

          1 Reply Last reply Reply Quote 0
          • T Offline
            TheFrisianClause
            last edited by

            Alright tested it with 7.6
            Seems to not work as well...

            [  165.594038] [drm] [nvidia-drm] [GPU ID 0x00000006] Loading driver
            [  165.594040] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:06.0 on minor 1
            [  171.958377] NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x22:0x56:667)
            [  171.958424] NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0
            [  171.963805] NVRM: GPU 0000:00:06.0: RmInitAdapter failed! (0x22:0x56:667)
            [  171.963848] NVRM: GPU 0000:00:06.0: rm_init_adapter failed, device minor number 0
            

            Same error....

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Okay so at least "it's good news": it's not a regression 🙂 So I assume it's required to hide the hypervisor underneath.

              T W 2 Replies Last reply Reply Quote 0
              • T Offline
                TheFrisianClause @olivierlambert
                last edited by

                @olivierlambert Yes I believe so, do you need more information on this?

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  Not yet, I need to talk with Xen devs when I can on how to achieve that.

                  T 1 Reply Last reply Reply Quote 0
                  • T Offline
                    TheFrisianClause @olivierlambert
                    last edited by

                    @olivierlambert Alright, thanks for the effort 🙂

                    1 Reply Last reply Reply Quote 0
                    • W Offline
                      warriorcookie @olivierlambert
                      last edited by warriorcookie

                      @olivierlambert just wanted to clarify a couple things:
                      -nvidia drivers support passthrough recently on windows guests only. Linux guests are still unsupported.
                      -while most quadro cards are supported for passthrough including P4000, the P400 is excluded from support. It's considered consumer grade (GP107 basically a striped down gtx1050)
                      -might be a given but for those that stumble on this thread, both the video device and audio device need to be passed through separately as they show as separate devices under lspci.

                      I'm confident this is an issue with the nvidia driver disabling when it see's the CPUID hypervisor present bit.

                      On my hardware (SM x9 with e5-2667v2) proxmox and esxi both worked flawlessly with near bare metal performance but you're able to change the CPUID hypervisor present bit.

                      On xcp-ng 8.2 with windows guest it works with no issues.
                      Once I install Linux as the guest then smi-nvidia produces "no devices were found" even though it shows with lspci.

                      I'm trying to migrate from esxi. I'm jumping in on this thread as this is a major issue for me that would prevent me switching. Unfortunate as this project ticks all the other boxes that vmware failed to....

                      T 1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Okay so it's "just" a matter of hiding the CPUID then 🙂 Thanks a lot for the detailed feedback @warriorcookie !

                        Let me push again on that feature request on my side, I'll see what we can do (and how much time/money to get a solution)

                        1 Reply Last reply Reply Quote 0
                        • W Offline
                          warriorcookie
                          last edited by

                          Parralel work into what I think could be the solution, but stalled as of now: https://xcp-ng.org/forum/topic/4643/nested-virtualization-of-windows-hyper-v-on-xcp-ng/39

                          1 Reply Last reply Reply Quote 0
                          • T Offline
                            TheFrisianClause @warriorcookie
                            last edited by

                            @warriorcookie Well this is a nice and detailed post, never knew about the CPUID though so I think that could be the problem here.

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              It is, but it doesn't answer how much effort is needed to "solve it". ATM, there's no way to change it in Xen. So the great question is the modification scope required 🙂

                              W 1 Reply Last reply Reply Quote 0
                              • W Offline
                                warriorcookie @olivierlambert
                                last edited by

                                @olivierlambert said in Nvidia Quadro P400 not working on Ubuntu server via GPU/PCIe passthrough:

                                It is, but it doesn't answer how much effort is needed to "solve it". ATM, there's no way to change it in Xen. So the great question is the modification scope required 🙂

                                My Pa always said "don't bring me a dead cat without a shovel..."

                                Sorry, I'm all cats and no shovel today!

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by olivierlambert

                                  Maybe a better approach would be to modify Nvidia drivers to change or remove the check. After all, it should be only a kind of grep on the word Xen 🤔

                                  And obviously, this would be also against the EULA.

                                  W 1 Reply Last reply Reply Quote 0
                                  • T Offline
                                    TheFrisianClause
                                    last edited by

                                    Apparently this seem to be an bigger issue than thought or am I wrong thinking that?

                                    1 Reply Last reply Reply Quote 0
                                    • W Offline
                                      warriorcookie @olivierlambert
                                      last edited by

                                      @olivierlambert said in Nvidia Quadro P400 not working on Ubuntu server via GPU/PCIe passthrough:

                                      Maybe a better approach would be to modify Nvidia drivers to change or remove the check. After all, it should be only a kind of grep on the word Xen 🤔

                                      And obviously, this would be also against the EULA.

                                      I've found this but have not had time to play: https://github.com/DualCoder/vgpu_unlock

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        So the most advanced discussion within Xen about it, is here: https://lists.archive.carbon60.com/xen/devel/439501

                                        Nothing happened since sadly, but it might be a start. Also, hiding Xen will probably have a lot of consequences regarding performances (no PV drivers and so on)

                                        T 1 Reply Last reply Reply Quote 0
                                        • T Offline
                                          TheFrisianClause @olivierlambert
                                          last edited by

                                          @olivierlambert hmmm seems something, but why can this so easily be accomplished with Proxmox and even ESXi (Which I thought would be a pain considering the age of the P400). So if I for example would buy an nvidia K4000 I would be able to succesfully passthrough the GPU to Plex? (I don't know if you have done that)

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by olivierlambert

                                            Because… they are totally different projects? ESXi got thousand engineers working on it. KVM model is different and less isolated than Xen and ESXi, so you can make important modifications in it more easily.

                                            T 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post