https://xcp-ng.org/docs/compute.html#vgpu
This is what you need, Pass through. Not necessarily VGPU ($400 per seat).
But! (Windows)
Follow these steps:
The link above.
Do not connect the GPU to the VM you have.
restart while holding shift so you can start with "safe mode + Networking"
Before your VM gets to BIOS force Shut down.
Attach RTX video Card
Start "safe mode + Networking"
Download and install NVIDIA drivers.
Reboot
You will have to do this to every VM that you need that RTX card on. Hasta la victoria siempre