GPU support and Nvidia Grid vGPU

dangerd

Has there been any changes to support GPU\vGPU with Nvidia Grid and or similar?

olivierlambert

Hi,

Short answer: no. You can't do it with XCP-ng, except if you decide to hack things around by replacing some binaries cherry picked from the XenServer ISO. Then it will work. Something we obviously can't do nor redistribute since it's not open source

EspenU

@olivierlambert Do you think it's possible for that support to be added, or are there proprietary code limitations that prevent it?
What would it take to solve that limitation?

We have currently delivered a project which which uses vmware, and the current temporary licenses end in September (could probably be extended a little).
In our cluster we have a total of 10x Nvidia A2, and 3x Nvidia L40 cards.
I would love to move to xcp-ng, as it really seems perfect for us, but the lack of Nvidia vgpu support stops us.

msupport

@EspenU
I have completed the migration step from Vmware to XCP-NG. I am using Nvidia M10 and Nvidia A16. I had to invest some time to get the Nvidia cards up and running. What I can reveal is that so far they do not work with xcp-ng release candidate 1

EspenU

@msupport said in GPU support and Nvidia Grid vGPU:

@EspenU
I have completed the migration step from Vmware to XCP-NG. I am using Nvidia M10 and Nvidia A16. I had to invest some time to get the Nvidia cards up and running. What I can reveal is that so far they do not work with xcp-ng release candidate 1

I've seen your post from March 14. where you got it working on 8.3.
So you're saying that something has broken between then and RC1?
That's not good .
I have it running in a test environment on 8.2 and since it was working fine there I was hoping it would still work in 8.3. sigh...

olivierlambert

8.3 isn't even officially released I don't see any reason why it would break between beta 2 and RC1, but if you can confirm it, we could investigate. There's no rush to get out of 8.2, it's still an LTS

msupport

@EspenU
#*** Here are my insights for the nvidia drivers and xcp-ng 8.3
#*** Install xcp-ng beta2
#*** then start updates

yum update kernel device-mapper guest-templates-json guest-templates-json-data-linux intel-microcode openssh python2-scapy amd-microcode cisco* libblkid libcgroup libcgroup-tools libcurl libmount libuuid curl util-linux edk2 forkexecd fuse-libs gdisk guest-templates-json-data-other guest-templates-json-data-windows intel-ice python-fasteners python2-defusedxml python2-xapi-storage nss-sysinit nss-tools nspr nss nss-softokn nss-softokn-freebl nss-tools   nss-util   kernel-livepatch logrotate mellanox-mlnxen message-switch openssl* openvswitch swtpm* tzdata microsemi-smartpqi vendor-drivers sudo newt qlogic-qla2xxx qlogic-fastlinq gpumon sm vhd-tool vcputune

* Nvidia Driver must install before this operation below, because the driver installation is not compatible with phython3 **

yum remove xcp-python-libs-2.3.5-1.1.xcpng8.3.noarch
yum update ncurses-compat-libs python3-fasteners python3-pyudev python3-scapy python3-xcp-libs python36-future    
yum update net-snmp yum update net-snmp-agent-libs net-snmp-libs
yum update xapi-storage-script
yum update xs-openssl-libs
yum update xapi-nbd
yum remove net-snmp
yum net-snmp-libs net-snmp-agent-libs net-snmp
yum update xcp-ng-plymouth-theme
yum update xen-crashdump-analyser

*** Driver don't work anymore with one of these 40 Updates,because the updates are dependent:

Name	Description	Version	Release	Size	
blktap	blktap user space utilities	3.54.9	1.1.xcpng8.3	305.47 KiB	
kexec-tools	kexec/kdump userspace tools	2.0.15	20.xcpng8.3	67.8 KiB	
ncurses	Ncurses support utilities	6.4	3.xcpng8.3	394.83 KiB	
ncurses-base	Descriptions of common terminals	6.4	3.xcpng8.3	57.81 KiB	
ncurses-libs	Ncurses libraries	6.4	3.xcpng8.3	312.17 KiB	
qemu	qemu-dm device model	4.2.1	5.2.9.xcpng8.3	15.57 MiB	
rrdd-plugins	RRDD metrics plugin	24.16.0	1.2.xcpng8.3	4.29 MiB	
setup	A set of system configuration and setup files	2.8.71	9.1.xcpng8.3	169.24 KiB	
sm-cli	CLI for xapi toolstack storage managers	24.16.0	1.2.xcpng8.3	1.53 MiB	
squeezed	Memory ballooning daemon for the xapi toolstack	24.16.0	1.2.xcpng8.3	1.54 MiB	
varstored	EFI Variable Storage Daemon	1.2.0	2.3.xcpng8.3	46.55 KiB	
varstored-guard	Deprivileged XAPI socket Daemon for EFI variable storage	24.16.0	1.2.xcpng8.3	4.3 MiB	
varstored-tools	Tools for manipulating a guest's EFI variables offline	1.2.0	2.3.xcpng8.3	58.66 KiB	
vncterm	vncterm tty to vnc utility	10.2.1	2.xcpng8.3	43.94 KiB	
wsproxy	Websockets proxy for VNC traffic	24.16.0	1.2.xcpng8.3	932.78 KiB	
xapi-core	The xapi toolstack	24.16.0	1.2.xcpng8.3	24.55 MiB	
xapi-rrd2csv	A tool to output RRD values in CSV format	24.16.0	1.2.xcpng8.3	2.61 MiB	
xapi-tests	Toolstack test programs	24.16.0	1.2.xcpng8.3	6.25 MiB	
xapi-xe	The xapi toolstack CLI	24.16.0	1.2.xcpng8.3	1.13 MiB	
xcp-clipboardd	Daemon to share a virtualized Windows clipboard	1.0.3	8.xcpng8.3	22.53 KiB	
xcp-featured	XCP-ng feature daemon	1.1.7	2.xcpng8.3	1.25 MiB	
xcp-networkd	Simple host network management service for the xapi toolstack	24.16.0	1.2.xcpng8.3	4.15 MiB	
xcp-ng-release	XCP-ng release file	8.3.0	24	112.56 KiB	
xcp-ng-release-config	XCP-ng configuration	8.3.0	24	49.93 KiB	
xcp-ng-release-presets	XCP-ng presets file	8.3.0	24	18.44 KiB	
xcp-ng-xapi-plugins	XAPI additional plugins for XCP-ng	1.10.0	1.xcpng8.3	46.17 KiB	
xcp-rrdd	Statistics gathering daemon for the xapi toolstack	24.16.0	1.2.xcpng8.3	3.14 MiB	
xen-dom0-libs	Xen Hypervisor Domain 0 libraries	4.17.4	3.xcpng8.3	691.85 KiB	
xen-dom0-tools	Xen Hypervisor Domain 0 tools	4.17.4	3.xcpng8.3	1.9 MiB	
xen-hypervisor	The Xen Hypervisor	4.17.4	3.xcpng8.3	2.34 MiB	
xen-libs	Xen Hypervisor general libraries	4.17.4	3.xcpng8.3	54.05 KiB	
xen-livepatch	Live patches for Xen	2.0	1.xcpng8.3	2.91 KiB	
xen-tools	Xen Hypervisor general tools	4.17.4	3.xcpng8.3	35.66 KiB	
xenopsd	Simple VM manager	24.16.0	1.2.xcpng8.3	1.17 MiB	
xenopsd-cli	CLI for xenopsd, the xapi toolstack domain manager	24.16.0	1.2.xcpng8.3	1.61 MiB	
xenopsd-xc	Xenopsd using xc	24.16.0	1.2.xcpng8.3	4.61 MiB	
xenserver-hwdata	Additional hardware identification and configuration data	20240411	1.xcpng8.3	284.41 KiB	
xenserver-status-report	A program that generates status reports for a XenServer host	2.0.3	1.xcpng8.3	33.24 KiB	
xo-lite	Xen Orchestra Lite	0.2.3	1.xcpng8.3	816.19 KiB	
xsconsole	XCP-ng Host Configuration Console	11.0.2	1.1.xcpng8.3	304.44 KiB

olivierlambert

Have you took the drivers for XS8, which is equivalent from XCP-ng 8.3?

EspenU

@olivierlambert Would that mean that the vgpu binary should be taken from XS8 as well when using XCP-ng 8.3?
During testing I tried using that binary in XCP-ng 8.2, and it didn't work (VMs would no boot). I had to use the one from Citrix Hypervisor 8.2.

msupport

@olivierlambert
I have tested the Nvidia XenServer version 17.0.

olivierlambert

I don't know all the versions, but I can tell that:

XCP-ng 8.2 == XS 8.2
XCP-ng 8.3 == XS 8

So be sure to use the right/matching binary first

msupport

@olivierlambert
I have found the solution. I will test the whole thing again tomorrow with a clean installation with rc1.

olivierlambert

Oh great! Keep us posted!!

tjkreidl

@msupport Please write up all the steps involved, as this would be very useful documentation for anyone else wanting to accomplish this. Many have delayed switching to XCP-ng because of not being able to make use of NVIDIA GPUs.

olivierlambert

It works on 8.2 already even if it's not official at all

msupport

Installation instructions XCP-NG (RC1) Nvidia M10 | A16 GPU

install XCP-NG 8.3 RC1
download XenServer Driver Nvidia 17.1 (NVIDIA-GRID-XenServer-8-550.54.16-550.54.15-551.78)
unzip driver and copy host driver (NVIDIA-vGPU-xenserver-8-550.54.16.x86_64.iso) I used winscp to copy the driver to the tmp directory.
download XenServer iso file (https://www.xenserver.com/downloads | XenServer8_2024-06-03.iso)
copy the file (vgpu-7.4.13-1.xs8.x86_64.rpm) in the packages directory ! Do not use CitrixHypervisor-8.2.0-install-cd file vgpu-7.4.8-1.x86_64
unpack file vgpu-7.4.13-1.xs8.x86_64
copy the file \usr\lib64\xen\bin\vgpu (size 129KB) to \usr\lib64\xen\bin\ on your XCP-NG host (chmod 755)
(putty) /tmp/ xe-install-supplemental-pack NVIDIA-vGPU-xenserver-8-550.54.16.x86_64.iso
reboot
install guest driver on the VM client (551.78_grid_win10_win11_server2022_dch_64bit_international.exe)
token file from Nvidia (C:\Program Files\Nvidia Corporation\vGPU Licensing\ClientConfigToken*.tok)

Nvidia drivers 17.2 and 17.3 do not work yet (Guest driver crashes)
I will stay tuned and inform you about new findings

Have fun

msupport

@olivierlambert
Thanks for the hint, that helped me a lot

olivierlambert

Thank you very much!

tjkreidl

@msupport Many thanks for your write-up! Have you experienced any issues communicating with the NVIDIA license server?

fatek

I also used instructions from @msupport

https://xcp-ng.org/forum/topic/8987/vgpu-nvidia-tesla-p4-xcp-ng-8-3-beta-2?_=1721015408249

GPU support and Nvidia Grid vGPU

*** Nvidia Driver must install before this operation below, because the driver installation is not compatible with phython3 ****

*** Driver don't work anymore with one of these 40 Updates,because the updates are dependent:

* Nvidia Driver must install before this operation below, because the driver installation is not compatible with phython3 **