Just to close the loop on this for anyone who might run into the same issue.
I picked up a StarTech.com PEXUSB3S7 7 Port PCI Express USB 3.0 Card and passed the whole card through and it works great.
getting the same Inference speed in Frigate as I do using Proxmox.