I would run it in Scale if i were you. Ive read about people's set ups like this and there hasn't been any issues with it but It has been argued that virtualization of your storage underneath a hypervisor is dangerous. I don't know what type of pitfalls you might encounter with this but it will surely make recovery from a disaster a little more complicated. And with Scale being able to run VM's you can have the GPU passed through to your Win11 machine.
I presume you are transcoding in your Jellyfin? I use Plex and I pass through a GPU to it (but thats in esxi). Can you pass through your GPU to multiple devices to use for that? I know there is vGPU but I don't know if it can be used for transcoding. It would be interesting to try though