@MajorTom thanks that could be a solution. In the end I moved my single-machine Homelab to Proxmox, because I have way better control in this specific use-case - I can see IOMMU groupings and figure out which USB controller I can also passthrough. And this works fine, I'm posting this from this VM.
XCPng isn't bad, it´s just not suited to my very specific use case.
Cheers,
Clément
around that (e.g. for KVM you can hide the fact that it's running inside a VM, and I believe there are similar solutions for Xen).