Have you tried to pass the USB controller directly? I have no idea about the USB passthrough speed, I'll ask around.