XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    USB + GPU pass-though issue

    Scheduled Pinned Locked Moved News
    11 Posts 4 Posters 587 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic was forked from XCP-ng 8.3 updates announcements and testing stormi
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      Question for @TeddyAstie maybe

      1 Reply Last reply Reply Quote 1
      • TeddyAstieT Offline
        TeddyAstie Vates 🪐 XCP-ng Team Xen Guru @gb.123
        last edited by

        @gb.123 said in XCP-ng 8.3 updates announcements and testing:

        Here is the summary:

        If USB Keyboard & Mouse is passed-through along-with GPU:
        The GPU gets stuck in D3 state (on Shutdown/Restart of VM) (Classic GPU reset problem)

        If no vUSB is passed but GPU is passed through:
        The GPU works correctly and resets correctly (on Shutdown/Restart of VM)

        I have no clue what vUSB may change regarding GPU passthrough.

        When I run :

        $> lspci
        Extract of Output (Partial):

        07:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8

        However, this controller does not show up when I run :
        xe pci-list

        Is it a bug that lspci & xe pci-list have different number of devices ?

        How can I pass this controller since xe pci-list does not show it so I can't get the UUID ?
        Will kernel parameters (like XCP-ng 8.2) work in this case ?

        Question for @Team-XAPI-Network regarding the filtering on PCI IDs.
        I don't think XAPI allows using arbitrary BDF, but I may be wrong.

        Is it safe to run on XCP-ng host ?

         echo 1 > /sys/bus/pci/rescan

        (I'm trying to find a way where the PCI card is reset by the host without complete reboot, though I am aware that the above command will not reset it.)

        Probably. But it's not going to change anything as the device doesn't completely leave the Dom0 when passed-through.
        FYI a function-level-reset is systematically performed by Xen when doing PCI passthrough, thus your device should be reset before entering another guest (aside reset bugs like you may have).

        Also is it advisable to use :

        xl pci-assignable-add 07:00.0

        in XCP-ng 8.3 ? or is this method deprecated ?

        I don't think XAPI supports this PCI passthrough approach.
        This is a command which allows dynamically to remove a device from Dom0 and put it into "quarantine domain", so that it will be ready to passthrough it.

        Current XAPI uses the approach of having a set of "passthrough-able" devices at boot time by modifying the xen-pciback.hide kernel parameter, which does the same but at boot time.

        G 1 Reply Last reply Reply Quote 2
        • G Offline
          gb.123 @TeddyAstie
          last edited by gb.123

          @TeddyAstie @olivierlambert

          Thank you sooo much for your prompt response ! 🙂

          FYI a function-level-reset is systematically performed by Xen when doing PCI passthrough, thus your device should be reset before entering another guest (aside reset bugs like you may have).

          This is exactly what the problem is. The Dom0 is unable to perform FLR when I also pass vUSB to the Guest. However, if USB is not passed (i mean if it is not attached; even though USB passthrough is enabled in host), the FLR seems to be performed correctly and I am able to restart the guest without problems.

          If the FLR is not performed, the Guest (even if it is the same one being restarted) is unable to detect the pass-through cards and also waits for about 136 seconds (about 65.5 seconds for each card) after which it continues without adding the card. This wait time is the kernel default wait time which cannot be changed without rebuilding the kernel I think.

          I am trying to find out where the conflict is. (Basically I am unable to understand if this is a driver problem or a Xen problem; since the FLR is being performed correctly when there is no usb being passed through)
          Ideally passing USB should not have in impact on PCI-pass through.

          Current XAPI uses the approach of having a set of "passthrough-able" devices at boot time by modifying the xen-pciback.hide kernel parameter, which does the same but at boot time.

          Will this still work with XCP-ng 8.3 ?

          /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:07:00.0)"

          I am unable to use xe pci-disable-dom0-access uuid=<pci uuid> since UUID for the above pci is not generated (and not visible in xe pci-list)

          1 Reply Last reply Reply Quote 0
          • G Offline
            gb.123
            last edited by gb.123

            @olivierlambert @TeddyAstie
            Whelp !
            I tried using /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:07:00.0)" but now XCP-ng refuses to boot !
            Any way I can reverse the command by booting in safe mode ?

            UPDATE :
            Manually edited /boot/efi/EFI/xenserver/grub.cfg and removed the entry, now the server boots.

            Hoping that /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:07:00.0)" only alters /boot/efi/EFI/xenserver/grub.cfg and not some internal Xen settings.

            A clarification on the above would be highly appreciated!

            1 Reply Last reply Reply Quote 0
            • stormiS Offline
              stormi Vates 🪐 XCP-ng Team
              last edited by

              I moved this discussion to its own topic, as we need the other one for update candidate testing.

              1 Reply Last reply Reply Quote 0
              • G Offline
                gb.123
                last edited by

                @stormi
                Thanks!

                @olivierlambert @TeddyAstie

                I am unable to test further since I don't have a 'passable'/'assignable usb controller which can be passed through. Have ordered one and would keep you guys posted once I get it and test it.

                Though I can confirm that the bug (dunno if it is the driver / xcp-ng) persists.

                1 Reply Last reply Reply Quote 0
                • G Offline
                  gb.123
                  last edited by

                  Update:

                  I purchased Renesas Technology Corp. uPD720202 USB 3.0 Host Controller which is a controller that goes into PCI Express Slot (A/E Key, usually slot for WiFi cards) specifically for testing this.
                  The host seems to recognize the controller and also attaching the keyboard works.

                  However, XCP-NG does not show it in xe pci-list despite being available in lspci
                  Since its not available in xe pci-list, I am unable to assign it as a passthrough.

                  @TeddyAstie @olivierlambert
                  Guys, any help on how I can make the item available in xe pci-list so that I can continue my testing ?
                  Is there any reason or decision factor for XCP-NG to put/not put the controller in xe pci-list (maybe uses some kind of logic based on configuration files) ?

                  I have noticed that lspci and xe pci-list give different lists. However, only the xe pci-list items are pci-passthrough enabled.

                  Would depend on your urgent help to continue my test and diagnostics on this.

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by olivierlambert

                    You can still pass the device even if it's not in xe pci-list. Use the lspci address to pass it directly to the VM via the xe CLI.

                    G 1 Reply Last reply Reply Quote 1
                    • G Offline
                      gb.123 @olivierlambert
                      last edited by gb.123

                      @olivierlambert

                      You can still pass the device even if it's not in xe pci-list. Use the lspci address to pass it directly to the VM via the xe CLI.

                      Thanks for the prompt response.

                      The docs say I need to pass xe pci-disable-dom0-access uuid=<pci uuid>
                      but the UUID is not present in the lspci.

                      Are you meaning the old method:

                      1. I edit grub or use /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=<lspci address>"
                      2. Asssign to VM using xe vm-param-set other-config:pci=0/<lspci-address> uuid=<vm-uuid>
                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Yes, the old method should work 🙂

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post