XCP-ng 8.1 PCIe onboard usb controller passthrough does not work

Maggehagge

Hi

Sorry for starting a new thread but i dont know if the old one was in the correct area for people to see.

I have XCP-ng 8.1 beta installed and im trying to passthrough my onboard usb controller on my gigabyte X79-UD5 and vm cannot be started.

This is the error message i receive: "Failed","Starting VM 'test'
Internal error: xenopsd internal error: Cannot_add(0000:00:1d.0, Xenctrlext.Unix_error(30, "1: Operation not permitted"))
Time: 00:00:08","xcp-ng-vkdehkmr","Feb 3, 2020 1:09 PM"

I know this worked fine in Xenserver 6.5 SP1. i have tested 8.0 but have problems even running default VMs on it and i believe this passthrough blocking issue is a thing that came in Xenserver 7 or something has i temporarily switched to proxmox around that time for this reason IIRC.

I believe there is some rule blocking this since it is usually not stable? maybe like USB passthrough usb-policy blocking some usb class devices.

Thanks in advance

olivierlambert

Do you have all hardware virt options enabled?

Maggehagge

All that exist in my UEFI/BIOS yes ofc. VT-d and so on. im running an xeon E5-1660v1 btw

stormi

Does this page help in any way? https://github.com/xcp-ng/xcp/wiki/USB-Pass-Through

Maggehagge

No, i tried that and its doesnt work for keyboard and mouse even with the usb policy change, my keyboard and mouse shows up in the gui and VMs start fine with no errors but they still control dom0 and not the guests.

it does not matter because the topic is actually about pcie passthrough that refuse to passthrough my onboard moterboard usb. i think there is some configuration that citrix has implemented.

hkparker

I'm experiencing the exact same issue on 8.1 stable. I'm able to passthrough and use any PCIe devices except USB controllers. There must be something preventing USB controllers from being able to be assigned (even though the PCI address shows up in xl pci-assignable-list after it's added to the pciback kernel parameter), but I'm not sure where to go looking for it.

hkparker

I realize this is old but I figured out what's going on here. Has to do with RMRR: https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf

Check out your /var/log/xen/hypervisor.log and you should see something like this:

[2020-08-22 10:09:03] (XEN) [  297.542134] [VT-D] It's disallowed to assign 0000:08:00.0 with shared RMRR at 7ba77000 for Dom32753.
[2020-08-22 10:09:03] (XEN) [  297.542136] d[IO]: assign (0000:08:00.0) failed (-1)

The redhat paper goes into good detail about what's going on here, but the tl;dr is that your BIOS has access to memory segments of USB devices for PS/2 emulation and therefore DMA isn't allowed in those regions.

Additional reading: https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt

Lots of people have been trying to figure this one out, especially in the Qubes community. Here's some discussions from all over the place that I found:

https://discussions.citrix.com/topic/378774-xenserver-7-usb-passthrough-on-xen-46/
http://xen.1045712.n5.nabble.com/shared-RMRR-issue-td5736717.html
https://groups.google.com/g/qubes-users/c/gS1XTal8XYs?pli=1
https://bugs.xenserver.org/browse/XSO-720
https://discussions.citrix.com/topic/381091-how-to-pass-rdm_policyrelaxed-through-xe-vm-param-set-in-xenserver-70/
https://groups.google.com/g/qubes-users/c/5x6iU28GadQ
https://lists.xen.org/archives/html/xen-users/2017-05/msg00053.html
http://xen.1045712.n5.nabble.com/quot-It-s-disallowed-to-assign-lt-device-gt-with-shared-RMRR-at-lt-address-gt-for-Dom-lt-N-gt-quot-a6-td5733137.html
https://github.com/QubesOS/qubes-issues/issues/1544

In my case appending ,rdm_policy=relaxed to my other-config:pci value didn't change anything, though some tickets indicated that had to be passed another way. I wasn't sure where I was supposed to set pci_strictreset, as all the examples were qubes-specific tooling.

Some of these discussions, as well as this part of xen's wiki give me the impression that maybe there's sub devices in my BDF address that need to be passed too, but if there are I couldn't find them with lspci. I haven't found any way to see what devices share an RMRR space other than one of the guys in one of those google groups discussing manually reading the hex of a dump from some tool.

Ahhh, oh well. This isn't mission critical for me, so I'm likely going to give up here. Hopefully this information provides helpful context for anyone running into this in the future, and if someone does manage to get PCI passthrough of a USB controller working, please do share your hardware and configuration.

Rudd-O created this issue in QubesOS/qubes-issues

closed fail to assign pci device at start of usbvm #1544

olivierlambert

That's a really interesting feedback, thanks a lot @hkparker for sharing this with us (it might be even interesting to write a recap on that into our official doc, what do you think?)

hkparker

Yeah, I'd be happy to write up a condensed version of this for the docs. Is that something I can PR somewhere?

olivierlambert

Using the link on the bottom of the doc will create a PR

darabontors

Hi,

I'm experiencing the same thing on 8.2 stable. The thing is, I could do the exact same passthrough via ESXi 6.7. I understand the DMA part of it, but how can it work without any problems on ESXi and not on XCP-ng? In my case, I tried USB device passthrough of a UPS to my OPNsense VM, but it didn't pick it up. I'll try to pass it to a TrueNAS VM which worked on my old ESXi setup. Back then, the whole USB controller was PCIe Passed to my TrueNAS VM. This time I'll try just the USB device passthrough.

Any other solutions I could try for the USB controller PCIe passthrough?

olivierlambert

but how can it work without any problems on ESXi and not on XCP-ng?

Maybe because they are 2 completely different software?

jeremfg

Hi,

I just wanted to report facing the same issue, and the exact same error message for the same BFD device number as OP.

In my case, I'm attempting to pass one of the USB 2.0 Controllers from my Supermicro X10SDV-TLN4F motherboard (featuing a xeon-d 1541 CPU). Since it is the same BFD number as OP, I wouldn't be suprised it's similar hardware.

00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)

After the device is hidden in grub.cfg it is correctly reported by xl pci-assignable-list. The error I get on VM start-up after assignation is:

Internal error: xenopsd internal error: Cannot_add(0000:00:1d.0, Xenctrlext.Unix_error(30, "1: Operation not permitted"))

FYI, passing the entire USB Controller was my 2nd attempt at passing a USB Bluetooth dongle which I was unssuccessfull in acheiving due to similar issues as in https://xcp-ng.org/forum/topic/266/usb-passthrough-test-reports-in-7-5rc1.

The RMRR explanation makes a lot of sense to me as well. I wouldn't mind passing more BFD devices to the same VMs if it fixes the issue, but I haven't found a combination that works.

I was hoping someone extrapolated a way to make this works in XCP-ng from the work done in the Qubes community as mentionned by @hkparker.