@MajorTom thanks that could be a solution. In the end I moved my single-machine Homelab to Proxmox, because I have way better control in this specific use-case - I can see IOMMU groupings and figure out which USB controller I can also passthrough. And this works fine, I'm posting this from this VM.
XCPng isn't bad, it´s just not suited to my very specific use case.
Cheers,
Clément