I encountered this yesterday as well. It seems as though this is a Xen issue: https://wiki.xenproject.org/wiki/Xen_USB_Passthrough
USB will only pass-through as USB 2.0, you will need to pass-through PCI for the entire USB controller, unfortunately that wasn't available for my on-board USB controller, so I think you have to have a secondary USB controller either on-board or as a add-in card.