XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    PCIe passthrough AMD/NVIDIA

    Scheduled Pinned Locked Moved
    Development
    2
    3
    805
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • U
      utorpute
      last edited by

      so I got this idea in my head that I can get 2x AMD rx 6400s that are single slot and low profile capable and stand up 2 VM's capable of running a game like BDO at min settings that could be left online, or possibly doing some PLEX transcodering.

      first, I tried with an x9spu (UIO) with RSC-R1UU-2E8 (with the pcie ends cut out) in a 1U chassis, but it would only see one of the 2 video cards. i somewhat assumed this mus be some limitation of the UIO, just that the board is old-ish and not handling it well, or that my dremeling the riser caused damage (on 2 risers, the exact same way, which seems somewhat unlikely).

      then i got my hands on a r730, and really started playing around. got the low profile riser and now have
      1x gt 8400s
      1x gtx 1050
      2x amd rx 6400

      and it looks like, while xcp-ng does see all 4 video cards, if i add any of the AMDs to a VM, ALL of the AMDs are counted as being in that VM, and the same with the nvidias, though at least the 1050 and 8400 show up in the VM as separate GPUs. the AMD's do not, making it far more confusing to figure out wtf was going on.

      does anyone know if this is this expected behaviour? is there a way around it beyond getting the fairly expensive vGPUs or using no more than 1 card from AMD/nvidia per xcp-ng host?

      this is my first real, serious, foray into trying to virtualize GPU.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Can you show us the result of lspci?

        U 1 Reply Last reply Reply Quote 0
        • U
          utorpute @olivierlambert
          last edited by

          @olivierlambert sure. it's very long though, I grepped out the intel devices for a shorter version on the forum, and attached txt files with the full output.

          I forgot there is also a gt 8800 present but i don't have the internal power cables for it (yet) and it looks like it needs them to even show up (some will work just die when they need more power). ill take it out next time I open it up.

          the riser layout physically is this (from the back):
          10gbe_8800_1050
          6400__8400
          6400

           lspci | grep -vi intel

01:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
01:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
03:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)
04:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
06:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8400 GS] (rev a1)
08:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
09:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe Switch [PS]
0a:00.0 PCI bridge: Renesas Technology Corp. SH7758 PCIe-PCI Bridge [PPB]
0b:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2 (rev 01)
81:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
83:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c7)
84:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
85:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 743f (rev c7)
85:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
86:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c7)
87:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
88:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 743f (rev c7)
88:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]

          lspci -tv | grep -vi intel

 +-[0000:80]-+-01.0-[81]----00.0  Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s]
 |           +-02.0-[82]--
 |           +-03.0-[83-85]----00.0-[84-85]----00.0-[85]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Device 743f
 |           |                                            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
 |           +-03.2-[86-88]----00.0-[87-88]----00.0-[88]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Device 743f
 |           |                                            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
             +-01.0-[03]----00.0  Broadcom / LSI MegaRAID SAS-3 3108 [Invader]
             +-02.0-[04]--+-00.0  NVIDIA Corporation GP107 [GeForce GTX 1050]
             |            \-00.1  NVIDIA Corporation GP107GL High Definition Audio Controller
             +-02.2-[05]--
             +-03.0-[02]--+-00.0  Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
             |            \-00.1  Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
             +-03.1-[01]--+-00.0  Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
             |            \-00.1  Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
             +-03.2-[06]----00.0  NVIDIA Corporation G86 [GeForce 8400 GS]
             +-1c.0-[07]--
             +-1c.7-[08-0c]----00.0-[09-0c]----00.0-[0a-0b]----00.0-[0b]----00.0  Matrox Electronics Systems Ltd. G200eR2

          r730 lspci.txt
          r730 lspci -vt.txt

          1 Reply Last reply Reply Quote 0
          • First post
            Last post