There is still no guarantee that even modified drivers will work, however, due to the way NVIDIA drivers function. There are a lot of people, who have spent a lot of time, to get it to work - with some success; but, everything I've read points to this being something you'll have to screw around with and "maintain" regularly to keep it working. If you're expecting it to be a smooth, easy, and "permanent" process, I would recommend not even starting to try it on Xen (Server or XCP-NG).
If you REALLY need to do this with VMs, KVM and unRAID both have built-in capabilities to do this - which seem to work a bit better than others. vmWare is supposed to have a similar feature; but, I've not gotten it to work. I will tell you, that in messing around with this on all the hypervisors, even on KVM and unRAID, some card\driver combinations just will not work, and, you'll honestly waste more time screwing around with it than just spinning up a dedicated box or grabbing an AMD card. I'm not an AMD guy, and I didn't choose that route myself; but, they at least do not have this "limitation" in their driver stack.
Don't think NVIDIA will come around and fix it, either - their forums are rife with this topic; and, they've pretty much said "we don't have any plans to fix this bug [that we actually introduced on purpose]".