GPU share to more Windows VMs on same XCP-NG node
-
How to run more Windows VM (4+) on same server.
Which GPU card you suggest to use and how to configure XCP-NG for this work. Passthrough is one option (the easy one), but you need separate GPU cards 1xGPU for each Windows VM.
Problem is because XEON server CPU don't have iGPU included.
Do you have any experience with Nvidia, AMD or Intel GPU cards? -
You would need a gpu that support SR-IOV. From NVIDIA that means a non consumer gpu.
Edit the below is Ai output.
GPU SR-IOV support varies significantly by vendor and architecture, with Intel offering the most extensive hardware-based SR-IOV for consumer and data center graphics, while NVIDIA and AMD rely heavily on proprietary drivers or specific enterprise hardware.
Intel Graphics support is the most widespread for virtualization, with 12th Gen (Alder Lake), 13th Gen (Raptor Lake), and 14th Gen (Raptor Lake Refresh) Core processors supporting SR-IOV, as do the Intel
Data Center GPU Flex Series and Intel Arc
Pro B-Series (requires driver version 32.0.101.8306 or newer). Older generations (6th through 10th Gen) primarily support GVT-g (software-based mediation) rather than hardware SR-IOV, while the Intel Core Ultra Series 1 (Meteor Lake) and Series 3 (Panther Lake) do not support SR-IOV.NVIDIA supports SR-IOV primarily through its proprietary vGPU and MIG (Multi-Instance GPU) features on enterprise-grade hardware, including the A100, A40, A30, RTX A-series, and Tesla lines. While open-source drivers like Nouveau exist, NVIDIA's proprietary Mdev driver is the standard method for enabling SR-IOV and mediation, often managed via tools like sriov-manage in environments like Harvester or OpenStack.
AMD SR-IOV support is limited to older FirePro and Radeon Pro cards (e.g., W7100, S7150, V520) using the deprecated GIM or MxGPU drivers. Support for modern Navi and RDNA architecture consumer GPUs is currently unclear or non-existent in open-source ecosystems, with AMD reportedly focusing SR-IOV capabilities on exclusive enterprise contracts rather than consumer hardware.
Vendor Architecture SR-IOV Support Status Key Hardware / Notes
Intel 12th-14th Gen Core Yes (Hardware) Iris Xe, Data Center Flex, Arc Pro B-Series
Intel 6th-10th Gen Core No (Software/GVT-g) HD Graphics 5500–630, UHD 620/630
Intel Core Ultra (Series 1/3) No Meteor Lake, Panther Lake
NVIDIA Ampere/Hopper/Ada Yes (Proprietary) A100, A40, RTX A6000, Tesla T4 (via vGPU/MIG)
NVIDIA Maxwell/Pascal/Turing Yes (Proprietary) Tesla P100, T4, Quadro RTX (via vGPU)
AMD Tonga/Vega/Navi Limited/Deprecated FirePro S7150, W7100 (GIM driver); Modern support unclear -
You would need a gpu that support SR-IOV. From NVIDIA that means a non consumer gpu.
Edit the below is Ai output.
True... But would XCP-NG support for it not also be needed?
E.g. i would assume the Intel SR-IOV features would require a DOM0 kernel at minimum 6.17 and even newer for some cards?
-
@acebmxer
Thank you for a lot of technical stuff.
We will try in next days with Intel B50 card and will report how to work. -
Unfortunately I cant answer those specific questions bout xcp-ng. Maybe someone from vates can confirm.
it wasn't to long ago nvidia prevent users from using passing though consumer cards into vms.
-
@Aleksander There are plenty of examples on this forum.
We have used Nvidia T4, it works very well. -
Hi Aleksander,
I noticed you were on tid 11869 back in February, where @fatek shared his working setup on a Tesla P4 with the NVIDIA-GRID-XenServer-8 driver, and @tjkreidl ran through the cards that have been reported to work (Tesla M6/M10/M60, P4/P6/P40/P100, V100, T4, A2/A10/A16/A40, RTX A5000/A6000/6000/8000). You closed that thread with "I will try to virtualize GPU to Windows VMs." I'm curious what happened: did you give it a try and hit something specific, or is this a fresh use case where the constraints are different?The reason I'm asking rather than starting from scratch: fatek was pretty clear that the path is YMMV (XenServer is the supported hypervisor for those drivers, XCP-ng isn't officially), and the XCP-ng vGPU docs confirm it. So if you tried and hit a wall, that's the conversation worth having here. If you tried and it worked but you're now scaling to more Windows VMs and want to know if the same setup holds at 4+, that's a different conversation. On the supported-on-paper side, the same docs page says AMD MxGPU is "trivial using industry standard" if AMD hardware is on the table.
I don't have hands-on with any of this myself, but it'd help to know which problem you're trying to solve now versus February.
Thanks!
-
We have few Microsoft Terminal server in our XCP-NG VMs.
When remote users run for example Chrome rendering is very slow - Chrome can use GPU, but XEON server processors don't have this future. At the moment we use Nvidia cards with 8GB RAM in pass-true configuration. I am looking for solution how to use just one more powerful GPU and share it with VM on this XCP-NG node. That is what I am looking for and when XCP-NG will have solution (without additional Nvidia license), we will be very happy. I will be happy with Nvidia, AMD or Inter GPU, that will property works in VMs. -
@Aleksander I fear that all these -- even now the RTX cards from NVIDIA -- generally require licensing when run on servers, whether for pass-through or assigned to individual VMs. As well, installing an NVIDIA license server is then necessary. They have become more strict about this over the past several years, as RTX cards used to be exempt.
There are, however, some exceptions, but it's complicated and confusing! For details, consult NVIDIA documents such as this one: https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/index.html
P.S. Unfortunately, since retirement a few years ago, I no longer have any hands-on equipment at my disposal. -
@tjkreidl
I found that some Intel PRO GPU card have SRV-IO without any additional license, but this functionality is supported in Linux kernel 6.12 and up. I hope in new XCP-NG will have 6.12+ and support GPU sharing.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login