GPU support and Nvidia Grid vGPU

tjkreidl

@msupport Please write up all the steps involved, as this would be very useful documentation for anyone else wanting to accomplish this. Many have delayed switching to XCP-ng because of not being able to make use of NVIDIA GPUs.

olivierlambert

It works on 8.2 already even if it's not official at all

msupport

Installation instructions XCP-NG (RC1) Nvidia M10 | A16 GPU

install XCP-NG 8.3 RC1
download XenServer Driver Nvidia 17.1 (NVIDIA-GRID-XenServer-8-550.54.16-550.54.15-551.78)
unzip driver and copy host driver (NVIDIA-vGPU-xenserver-8-550.54.16.x86_64.iso) I used winscp to copy the driver to the tmp directory.
download XenServer iso file (https://www.xenserver.com/downloads | XenServer8_2024-06-03.iso)
copy the file (vgpu-7.4.13-1.xs8.x86_64.rpm) in the packages directory ! Do not use CitrixHypervisor-8.2.0-install-cd file vgpu-7.4.8-1.x86_64
unpack file vgpu-7.4.13-1.xs8.x86_64
copy the file \usr\lib64\xen\bin\vgpu (size 129KB) to \usr\lib64\xen\bin\ on your XCP-NG host (chmod 755)
(putty) /tmp/ xe-install-supplemental-pack NVIDIA-vGPU-xenserver-8-550.54.16.x86_64.iso
reboot
install guest driver on the VM client (551.78_grid_win10_win11_server2022_dch_64bit_international.exe)
token file from Nvidia (C:\Program Files\Nvidia Corporation\vGPU Licensing\ClientConfigToken*.tok)

Nvidia drivers 17.2 and 17.3 do not work yet (Guest driver crashes)
I will stay tuned and inform you about new findings

Have fun

msupport

@olivierlambert
Thanks for the hint, that helped me a lot

olivierlambert

Thank you very much!

tjkreidl

@msupport Many thanks for your write-up! Have you experienced any issues communicating with the NVIDIA license server?

fatek

I also used instructions from @msupport

https://xcp-ng.org/forum/topic/8987/vgpu-nvidia-tesla-p4-xcp-ng-8-3-beta-2?_=1721015408249

msupport

@tjkreidl
Nvidia licence server works perfectly so far

Tristis Oris

i can't find where to get this new nvidia driver. Tesla V100.

upd
looks the only way is license portal https://nvid.nvidia.com/. sad.

msupport

@Tristis-Oris
The driver version 17.1 worked for me, 17.2 and 17.3 crashed the Windows drivers
https://we.tl/t-VozEeV8TFB

Tristis Oris

@msupport thank you. Will try to play with it.

msupport

@Tristis-Oris
The download will be available for 3 days...

mgformula1

i have followed the documentation however for some reason the VMs wont power on with a vGPU profile attached. We are testing with NVidia M10 GPUs. I'm using xcp-ng 8.3 and NVidia host driver 17.1, also tried 17.2 and 17.3

This is the error

{
  "id": "0m369opsm",
  "properties": {
    "method": "vm.start",
    "params": {
      "id": "b8c94655-5801-21ee-7eb0-788a58b57736",
      "bypassMacAddressesCheck": false,
      "force": false
    },
    "name": "API call: vm.start",
    "userId": "2c8c735d-5369-4a91-8433-b9f94e6eb394",
    "type": "api.call"
  },
  "start": 1730921023894,
  "status": "failure",
  "updatedAt": 1730921066138,
  "end": 1730921066138,
  "result": {
    "code": "FAILED_TO_START_EMULATOR",
    "params": [
      "OpaqueRef:f3f7d9f6-9dc7-772e-ddfe-c1ed19f1aeff",
      "vgpu",
      "Device.Dm.start_vgpu: emulator failed to start for domain 1"
    ],
    "call": {
      "method": "VM.start",
      "params": [
        "OpaqueRef:f3f7d9f6-9dc7-772e-ddfe-c1ed19f1aeff",
        false,
        false
      ]
    },
    "message": "FAILED_TO_START_EMULATOR(OpaqueRef:f3f7d9f6-9dc7-772e-ddfe-c1ed19f1aeff, vgpu, Device.Dm.start_vgpu: emulator failed to start for domain 1)",
    "name": "XapiError",
    "stack": "XapiError: FAILED_TO_START_EMULATOR(OpaqueRef:f3f7d9f6-9dc7-772e-ddfe-c1ed19f1aeff, vgpu, Device.Dm.start_vgpu: emulator failed to start for domain 1)\n    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)\n    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21\n    at runNextTicks (node:internal/process/task_queues:60:5)\n    at processImmediate (node:internal/timers:447:9)\n    at process.callbackTrampoline (node:internal/async_hooks:128:17)"

host output

mgformula1

i also performed a clean install of 8.2, same power on issue when a vGPU profile is attached.

timemaster5

@msupport said in GPU support and Nvidia Grid vGPU:

@Tristis-Oris
The driver version 17.1 worked for me, 17.2 and 17.3 crashed the Windows drivers
https://we.tl/t-VozEeV8TFB

May I ask for a re-up? For some reason, I don't see version 17 in my Nvidia Licensing Portal The latest is 16

Tristis Oris

@timemaster5 https://file.io/8aVNWXPX4uAT

timemaster5

@Tristis-Oris thanks a lot, but is says: The transfer you requested has been deleted. sorry for troubles

Tristis Oris

@timemaster5 omg. i suppose it removed by legal reasons. i can upload somewhere else, but have no idea.

timemaster5

@Tristis-Oris sharing my google drive link via chat... should be easy to upload it there

ATSandrearicci

Let me update the procedure with my experience:

install XCP-NG 8.3
download XenServer Driver Nvidia 16.9 (latest supported by y M60)
unzip driver and copy host driver (NVIDIA-vGPU-xenserver-8-535.230.02.x86_64.iso) I used winscp to copy the driver to the tmp directory.
download XenServer iso file (https://www.xenserver.com/downloads | XenServer8_2024-12-09.iso)
copy the file (vgpu-7.4.16-1.xs8.x86_64.rpm) in the packages directory
yum localinstall vgpu-7.4.16-1.xs8.x86_64.rpm
yum localinstall NVIDIA-vGPU-xenserver-8-535.230.02.x86_64.rpm
reboot
install guest driver on the VM client (539.19_grid_win10_win11_server2019_server2022_dch_64bit_international.exe)

I preferred to use yum localinstall to be able to remove or update packages faster.

I have a question regarding vGPU, why i have profiles where i can have up to 4 heads? I don't have any option to add more than one display and i don't even understand how can i use them.
Thanks for the procedure, i'm a total newbie with vGPu.