Just to close the loop on this for anyone who might run into the same issue.
I picked up a StarTech.com PEXUSB3S7 7 Port PCI Express USB 3.0 Card and passed the whole card through and it works great.
getting the same Inference speed in Frigate as I do using Proxmox.

Thank you all with your assistance with this, I truly appreciate it!
I have a cheap Inateck card that I picked up on amazon, which I was not able to get working correctly in either XCP-NG or Proxmox.
Does anyone have a recommendation for a USB 3.0 card that is known to work?
If not, I can do some searching, but figured I would ask.
Thanks!

@olivierlambert I believe so, thats when it would show up in this list?

[12:36 xcp-ng-test ~]# xl pci-assignable-list
0000:00:14.0

To be complete, these are the commands that I ran to add that adapter to my VM:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:00:14.0)"
xl pci-assignable-list  (to make sure it was there)
reboot
xe vm-param-set other-config:pci=0/0000:00:14.0 uuid=54836e04-7bee-9f34-16f7-51c7bf2fb96a

Here is its listing in lspci:

00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)

@olivierlambert
Sorry for the delay.
Shortly after posting this, I installed Proxmox for testing. In the same configuration, I was able to pass through just the USB TPU, and the speed was reported as 5000M. Though in other systems the Coral's Inference speed was around 10ms, and in this configuration It was about 18ms.
I decided to pass through the USB controller, and was able to get my Inference speed down to 8ms.
I then decided to install XCP-NG 8.3R2, and do the same.
First thing I noticed, none of the USB controllers showed up in the list to have them passed though. So I passed the USB controller though manually from the command line, then associated it with the VM. I am currently unable to start the VM due to this error:
INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:00:14.0, Xenctrlext.Unix_error(30, "1: Operation not permitted")))

Is that similar to this issue?
https://xcp-ng.org/forum/topic/2583/xcp-ng-8-1-pcie-onboard-usb-controller-passthrough-does-not-work

Please let me know if there is any hope, or if you need any additional information.
Thank you,

Joe

Good morning,
I'm testing XCP-ng 8.3r2 to get Frigate working to start out with.
I have a Debian VM running, which runs Docker, which has a Frigate container inside that.
I am passing through a Google Coral USB TPU to the Debian VM.
Initially I noticed that my inference speeds are very high (120ms), when they should be around 10ms.
I see that the XCP-ng host is seeing the USB Bus speeds correctly: (redacted unnecessary lines)

[17:26 xcp-ng-test ~]# lsusb --tree
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M
[10:02 xcp-ng-test ~]# lsusb
Bus 002 Device 003: ID 18d1:9302 Google Inc.

But on the Debian Host I'm not seeing the 5000M speed:

eppo@docker-test:~$ lsusb --tree
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=usbfs, 480M
eppo@docker-test:~$ lsusb
Bus 002 Device 002: ID 18d1:9302 Google Inc.

Anything I should be looking at to ensure the USB gets passed though correctly?
Any help would be greatly appreciated.
Thanks,

Joe