@olivierlambert
I got it working!
Pic while transcoding with plex

Plex info:

Detected:

For future reference if anyone finds this with google:
I had to " sudo chmod -R 777 /dev/dri" inside the Ubuntu VM, otherwise it didnt work.
I'm using binhex plex docker image, I have to add the device to the container. It's really easy with portainer:
The "/dev/dri/" part is the important

PS: @olivierlambert there is a typo in the xcp docs, a space is missing in Step 5:
https://docs.xcp-ng.org/compute/#5-put-this-pci-device-into-your-vm

This copy/pastes as:
xe vm-param-set other-config:pci=0/0000:04:01.0uuid=<vm uuid>
It should be:
xe vm-param-set other-config:pci=0/0000:04:01.0 uuid=<vm uuid>