XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XCP-ng 8.1 PCIe onboard usb controller passthrough does not work

    Scheduled Pinned Locked Moved Development
    13 Posts 6 Posters 5.0k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M Offline
      Maggehagge
      last edited by

      Hi

      Sorry for starting a new thread but i dont know if the old one was in the correct area for people to see.

      I have XCP-ng 8.1 beta installed and im trying to passthrough my onboard usb controller on my gigabyte X79-UD5 and vm cannot be started.

      This is the error message i receive: "Failed","Starting VM 'test'
      Internal error: xenopsd internal error: Cannot_add(0000:00:1d.0, Xenctrlext.Unix_error(30, "1: Operation not permitted"))
      Time: 00:00:08","xcp-ng-vkdehkmr","Feb 3, 2020 1:09 PM"

      I know this worked fine in Xenserver 6.5 SP1. i have tested 8.0 but have problems even running default VMs on it and i believe this passthrough blocking issue is a thing that came in Xenserver 7 or something has i temporarily switched to proxmox around that time for this reason IIRC.

      I believe there is some rule blocking this since it is usually not stable? maybe like USB passthrough usb-policy blocking some usb class devices.

      Thanks in advance

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Do you have all hardware virt options enabled?

        1 Reply Last reply Reply Quote 0
        • M Offline
          Maggehagge
          last edited by

          All that exist in my UEFI/BIOS yes ofc. VT-d and so on. im running an xeon E5-1660v1 btw

          1 Reply Last reply Reply Quote 0
          • stormiS Offline
            stormi Vates 🪐 XCP-ng Team
            last edited by

            Does this page help in any way? https://github.com/xcp-ng/xcp/wiki/USB-Pass-Through

            1 Reply Last reply Reply Quote 0
            • M Offline
              Maggehagge
              last edited by

              No, i tried that and its doesnt work for keyboard and mouse even with the usb policy change, my keyboard and mouse shows up in the gui and VMs start fine with no errors but they still control dom0 and not the guests.

              it does not matter because the topic is actually about pcie passthrough that refuse to passthrough my onboard moterboard usb. i think there is some configuration that citrix has implemented.

              1 Reply Last reply Reply Quote 0
              • H Offline
                hkparker
                last edited by

                I'm experiencing the exact same issue on 8.1 stable. I'm able to passthrough and use any PCIe devices except USB controllers. There must be something preventing USB controllers from being able to be assigned (even though the PCI address shows up in xl pci-assignable-list after it's added to the pciback kernel parameter), but I'm not sure where to go looking for it.

                1 Reply Last reply Reply Quote 0
                • H Offline
                  hkparker
                  last edited by

                  I realize this is old but I figured out what's going on here. Has to do with RMRR: https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf

                  Check out your /var/log/xen/hypervisor.log and you should see something like this:

                  [2020-08-22 10:09:03] (XEN) [  297.542134] [VT-D] It's disallowed to assign 0000:08:00.0 with shared RMRR at 7ba77000 for Dom32753.
                  [2020-08-22 10:09:03] (XEN) [  297.542136] d[IO]: assign (0000:08:00.0) failed (-1)
                  

                  The redhat paper goes into good detail about what's going on here, but the tl;dr is that your BIOS has access to memory segments of USB devices for PS/2 emulation and therefore DMA isn't allowed in those regions.

                  Additional reading: https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt

                  Lots of people have been trying to figure this one out, especially in the Qubes community. Here's some discussions from all over the place that I found:

                  https://discussions.citrix.com/topic/378774-xenserver-7-usb-passthrough-on-xen-46/
                  http://xen.1045712.n5.nabble.com/shared-RMRR-issue-td5736717.html
                  https://groups.google.com/g/qubes-users/c/gS1XTal8XYs?pli=1
                  https://bugs.xenserver.org/browse/XSO-720
                  https://discussions.citrix.com/topic/381091-how-to-pass-rdm_policyrelaxed-through-xe-vm-param-set-in-xenserver-70/
                  https://groups.google.com/g/qubes-users/c/5x6iU28GadQ
                  https://lists.xen.org/archives/html/xen-users/2017-05/msg00053.html
                  http://xen.1045712.n5.nabble.com/quot-It-s-disallowed-to-assign-lt-device-gt-with-shared-RMRR-at-lt-address-gt-for-Dom-lt-N-gt-quot-a6-td5733137.html
                  https://github.com/QubesOS/qubes-issues/issues/1544

                  In my case appending ,rdm_policy=relaxed to my other-config:pci value didn't change anything, though some tickets indicated that had to be passed another way. I wasn't sure where I was supposed to set pci_strictreset, as all the examples were qubes-specific tooling.

                  Some of these discussions, as well as this part of xen's wiki give me the impression that maybe there's sub devices in my BDF address that need to be passed too, but if there are I couldn't find them with lspci. I haven't found any way to see what devices share an RMRR space other than one of the guys in one of those google groups discussing manually reading the hex of a dump from some tool.

                  Ahhh, oh well. This isn't mission critical for me, so I'm likely going to give up here. Hopefully this information provides helpful context for anyone running into this in the future, and if someone does manage to get PCI passthrough of a USB controller working, please do share your hardware and configuration.

                  Rudd-O created this issue in QubesOS/qubes-issues

                  closed fail to assign pci device at start of usbvm #1544

                  1 Reply Last reply Reply Quote 1
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    That's a really interesting feedback, thanks a lot @hkparker for sharing this with us 🙂 (it might be even interesting to write a recap on that into our official doc, what do you think?)

                    1 Reply Last reply Reply Quote 0
                    • H Offline
                      hkparker
                      last edited by

                      Yeah, I'd be happy to write up a condensed version of this for the docs. Is that something I can PR somewhere?

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Using the link on the bottom of the doc will create a PR 🙂

                        1 Reply Last reply Reply Quote 0
                        • D Offline
                          darabontors
                          last edited by

                          Hi,

                          I'm experiencing the same thing on 8.2 stable. The thing is, I could do the exact same passthrough via ESXi 6.7. I understand the DMA part of it, but how can it work without any problems on ESXi and not on XCP-ng? In my case, I tried USB device passthrough of a UPS to my OPNsense VM, but it didn't pick it up. I'll try to pass it to a TrueNAS VM which worked on my old ESXi setup. Back then, the whole USB controller was PCIe Passed to my TrueNAS VM. This time I'll try just the USB device passthrough.

                          Any other solutions I could try for the USB controller PCIe passthrough?

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by olivierlambert

                            but how can it work without any problems on ESXi and not on XCP-ng?

                            Maybe because they are 2 completely different software?

                            1 Reply Last reply Reply Quote 0
                            • J Offline
                              jeremfg
                              last edited by

                              Hi,

                              I just wanted to report facing the same issue, and the exact same error message for the same BFD device number as OP.

                              In my case, I'm attempting to pass one of the USB 2.0 Controllers from my Supermicro X10SDV-TLN4F motherboard (featuing a xeon-d 1541 CPU). Since it is the same BFD number as OP, I wouldn't be suprised it's similar hardware.

                              00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
                              

                              After the device is hidden in grub.cfg it is correctly reported by xl pci-assignable-list. The error I get on VM start-up after assignation is:

                              Internal error: xenopsd internal error: Cannot_add(0000:00:1d.0, Xenctrlext.Unix_error(30, "1: Operation not permitted"))
                              

                              FYI, passing the entire USB Controller was my 2nd attempt at passing a USB Bluetooth dongle which I was unssuccessfull in acheiving due to similar issues as in https://xcp-ng.org/forum/topic/266/usb-passthrough-test-reports-in-7-5rc1.

                              The RMRR explanation makes a lot of sense to me as well. I wouldn't mind passing more BFD devices to the same VMs if it fixes the issue, but I haven't found a combination that works.

                              I was hoping someone extrapolated a way to make this works in XCP-ng from the work done in the Qubes community as mentionned by @hkparker.

                              1 Reply Last reply Reply Quote 0
                              • J Joe_dev referenced this topic on
                              • First post
                                Last post