USB Passthrough speed issue

Joe_dev

Good morning,
I'm testing XCP-ng 8.3r2 to get Frigate working to start out with.
I have a Debian VM running, which runs Docker, which has a Frigate container inside that.
I am passing through a Google Coral USB TPU to the Debian VM.
Initially I noticed that my inference speeds are very high (120ms), when they should be around 10ms.
I see that the XCP-ng host is seeing the USB Bus speeds correctly: (redacted unnecessary lines)

[17:26 xcp-ng-test ~]# lsusb --tree
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 3: Dev 3, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M
[10:02 xcp-ng-test ~]# lsusb
Bus 002 Device 003: ID 18d1:9302 Google Inc.

But on the Debian Host I'm not seeing the 5000M speed:

eppo@docker-test:~$ lsusb --tree
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=usbfs, 480M
eppo@docker-test:~$ lsusb
Bus 002 Device 002: ID 18d1:9302 Google Inc.

Anything I should be looking at to ensure the USB gets passed though correctly?
Any help would be greatly appreciated.
Thanks,

Joe

olivierlambert

Have you tried to pass the USB controller directly? I have no idea about the USB passthrough speed, I'll ask around.

Joe_dev

@olivierlambert
Sorry for the delay.
Shortly after posting this, I installed Proxmox for testing. In the same configuration, I was able to pass through just the USB TPU, and the speed was reported as 5000M. Though in other systems the Coral's Inference speed was around 10ms, and in this configuration It was about 18ms.
I decided to pass through the USB controller, and was able to get my Inference speed down to 8ms.
I then decided to install XCP-NG 8.3R2, and do the same.
First thing I noticed, none of the USB controllers showed up in the list to have them passed though. So I passed the USB controller though manually from the command line, then associated it with the VM. I am currently unable to start the VM due to this error:
INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:00:14.0, Xenctrlext.Unix_error(30, "1: Operation not permitted")))

Is that similar to this issue?
https://xcp-ng.org/forum/topic/2583/xcp-ng-8-1-pcie-onboard-usb-controller-passthrough-does-not-work

Please let me know if there is any hope, or if you need any additional information.
Thank you,

Joe

olivierlambert

Have you double checked your PCI passthrough config is OK? (the device should be invisible in the Dom0)

Joe_dev

@olivierlambert I believe so, thats when it would show up in this list?

[12:36 xcp-ng-test ~]# xl pci-assignable-list
0000:00:14.0

To be complete, these are the commands that I ran to add that adapter to my VM:

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:00:14.0)"
xl pci-assignable-list  (to make sure it was there)
reboot
xe vm-param-set other-config:pci=0/0000:00:14.0 uuid=54836e04-7bee-9f34-16f7-51c7bf2fb96a

Here is its listing in lspci:

00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)

olivierlambert

I wonder if the USB chip is within the CPU SoC or not (and this could cause issues).
@Teddy-Astie those commands sounds fine by me, do you spot anything wrong?

TS79

Hi @Joe_dev

I'm no expert on this, but one thing I noticed is that on your host, the driver shows as "xhci" (which is the USB 3.x controller interface standard), whereas in the VM the driver is showing as "ehci" (which is the USB 2.x interface) and therefore has the 480Mbps bandwidth limitation.

Sadly I've got little to no experience with USB or PCIe passthrough (haven't had to use them yet), but hopefully this can point someone in the right direction to troubleshoot. My guess is, either the VM's OS has a limitation, or there's a VM setting at XCP-ng level, or the XCP-ng host itself has a setting or device passthrough challenge. Sorry I can't be more helpful.

EDIT: I just plugged a USB 3.0 storage device into my XCP-ng host, went to the VM settings and added a VUSB device from the PUSB. Within Xen Orchestra, the USB device shows as 5000Mbps speed, but within the VM it shows as 480Mbps. VM's OS is Ubuntu 22.04.5. I'm going to test on a newer Ubuntu 24.04 version - will reply on this thread.

TS79

Tried on Ubuntu 24.04 with kernel 6, same result: host and XCP/XOA show the device as xhci 5000Mbps, guest VM shows the device as ehci 480Mbps.

I don't have any time to fiddle, and some of this seems hardware specific, but some brief searching on keywords with Linux, EHCI, XHCI, etc. seem to mention a few possibilities: kernel driver being loaded wrong, guest EUFI vs BIOS (where BIOS initialises the device as USB 2.0), host BIOS (disabling legacy USB support - but this may endanger keyboard/mouse inputs), or host IOMMU settings.

olivierlambert

It's maybe due to the USB passthrough itself, we probably need to ask internally.

TS79

@olivierlambert Any idea who can be tagged here to ask internally (I assume you meant someone from the Vates team)

olivierlambert

Yes you assumed correctly and right now IDK who could take a look at this as it's a bit less a priority vs other stuff. Maybe we could ask @andyhhp if he's aware about the specifics of USB passthrough mechanism.

nuts23

I encountered this yesterday as well. It seems as though this is a Xen issue: https://wiki.xenproject.org/wiki/Xen_USB_Passthrough

USB will only pass-through as USB 2.0, you will need to pass-through PCI for the entire USB controller, unfortunately that wasn't available for my on-board USB controller, so I think you have to have a secondary USB controller either on-board or as a add-in card.

olivierlambert

I'm adding this in our "list of things" to check.

Joe_dev

Thank you all with your assistance with this, I truly appreciate it!
I have a cheap Inateck card that I picked up on amazon, which I was not able to get working correctly in either XCP-NG or Proxmox.
Does anyone have a recommendation for a USB 3.0 card that is known to work?
If not, I can do some searching, but figured I would ask.
Thanks!

Joe_dev

Just to close the loop on this for anyone who might run into the same issue.
I picked up a StarTech.com PEXUSB3S7 7 Port PCI Express USB 3.0 Card and passed the whole card through and it works great.
getting the same Inference speed in Frigate as I do using Proxmox.

olivierlambert

Thanks for your feedback. In the meantime, I will plan a "spike" to investigate the status of USB speed on passthrough.