I realize this is old but I figured out what's going on here. Has to do with RMRR: https://access.redhat.com/sites/default/files/attachments/rmrr-wp1.pdf
Check out your /var/log/xen/hypervisor.log and you should see something like this:
[2020-08-22 10:09:03] (XEN) [ 297.542134] [VT-D] It's disallowed to assign 0000:08:00.0 with shared RMRR at 7ba77000 for Dom32753.
[2020-08-22 10:09:03] (XEN) [ 297.542136] d[IO]: assign (0000:08:00.0) failed (-1)
The redhat paper goes into good detail about what's going on here, but the tl;dr is that your BIOS has access to memory segments of USB devices for PS/2 emulation and therefore DMA isn't allowed in those regions.
Additional reading: https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt
Lots of people have been trying to figure this one out, especially in the Qubes community. Here's some discussions from all over the place that I found:
https://discussions.citrix.com/topic/378774-xenserver-7-usb-passthrough-on-xen-46/
http://xen.1045712.n5.nabble.com/shared-RMRR-issue-td5736717.html
https://groups.google.com/g/qubes-users/c/gS1XTal8XYs?pli=1
https://bugs.xenserver.org/browse/XSO-720
https://discussions.citrix.com/topic/381091-how-to-pass-rdm_policyrelaxed-through-xe-vm-param-set-in-xenserver-70/
https://groups.google.com/g/qubes-users/c/5x6iU28GadQ
https://lists.xen.org/archives/html/xen-users/2017-05/msg00053.html
http://xen.1045712.n5.nabble.com/quot-It-s-disallowed-to-assign-lt-device-gt-with-shared-RMRR-at-lt-address-gt-for-Dom-lt-N-gt-quot-a6-td5733137.html
https://github.com/QubesOS/qubes-issues/issues/1544
In my case appending ,rdm_policy=relaxed to my other-config:pci value didn't change anything, though some tickets indicated that had to be passed another way. I wasn't sure where I was supposed to set pci_strictreset, as all the examples were qubes-specific tooling.
Some of these discussions, as well as this part of xen's wiki give me the impression that maybe there's sub devices in my BDF address that need to be passed too, but if there are I couldn't find them with lspci. I haven't found any way to see what devices share an RMRR space other than one of the guys in one of those google groups discussing manually reading the hex of a dump from some tool.
Ahhh, oh well. This isn't mission critical for me, so I'm likely going to give up here. Hopefully this information provides helpful context for anyone running into this in the future, and if someone does manage to get PCI passthrough of a USB controller working, please do share your hardware and configuration.
