Steps I have done to make NVIDIA vGPU works:

1. Install XCP-ng 8.2.1
2. Install all update yum update
3. reboot
4. Download NVIDIA vGPU drivers for XenServer 8.2 from NVIDIA site. Version NVIDIA-GRID-CitrixHypervisor-8.2-510.108.03-513.91
5. Unzip and install rpm from Host-Drivers
6. reboot again
7. Download free CitrixHypervisor-8.2.0-install-cd.iso from Citrix site
8. Open CitrixHypervisor-8.2.0-install-cd.iso with 7-zip, then unzip vgpu binary file from Packages->vgpu....rpm->vgpu....cpio->.->usr->lib64->xen->bin
9. Upload vgpu to XCP-ng host to /usr/lib64/xen/bin and made it executable chmod +x /usr/lib64/xen/bin/vgpu
10. Deployed VM with vGPU and it started without any problems

So I did not make any modifications with emu-manager.

My test server is far away from me and it will take some time to download the windows ISO to this test location. Then I will check how it works in the guest OS and report back here.

Hello!

I'm system and network engineer.
I have started work with xenserver since 2013 (I think it was opensource 6.2 version).
In 2015 I started deploing my own control panel for hosting based on xenserver.

After Citrix closed free version I started to move to XCP-ng.

Now I'm playing with xcp-ng + GPU, testing performance in different scenarios and sharing on this forum I believe that knowledge should be open so that everyone can move forward faster.

Some time later I faced with problem that VMs (Linux and Windows) can't correctly start GPU (ADM MxGPU).
In windows device manager there was error #43.
I have solved this error without host reboot by reloading gim module.

Hope this will help somebody else.

rmmod gim gim_api
modprobe gim gim_api

+1 to @Andrew setup

@wyatt-made I need few days to test it.
Will report later here

Finally after a week I found the solution!
There is no problem with emu-manager.
XCP does not contain necessary package vgpu.
I copied vgpu from Citrix ISO and now it is alive! : )

Live migration is not possible in this case, because of lack of free space.

You have to shutdown VM, then click this button and chose another host

There is no implementation in XAPI to "copy" VM to remote host (maybe XO has it own implementaion I'm not sure).
Copy works only localy.
You can only migrate it to remote host.

Disclaimer.
Everything you do on your own risk.

But you can do it manually by export/copy VDI from one host to another.
If you do not need "live" copy while VM is running, just stop VM, get all VBDs xe vbd-list vm-name-lable={VM-name}.
You will see all VBDs and VDIs of VM.
Go to certain SR cd /var/run/sr-mount/{SR-UUID} and copy VDI to new host's SR (you can use scp).
On destination host:

1. Get new VDI UUID vhd-util read -p -n {VDI-UUID}.vhd
2. Rename file using this UUID
   
3. Rescan SR on destination host.
4. Create VM, attach VDI.

If VDI stored on "shared" SR (iSCSI, NFS, ceph rbd and etc...), firts you have to export VDI by command xe vdi-export uuid={VDI-UUID} filename={VM-Name_xvda}.vhd format=vhd --progress
After success export, copy to destination host, read, rename and etc....

If you need "live" migration you can do it with minimal down time via snapshot.

1. Make snapshot with uniq name.
2. List VBDs xe vbd-list vm-name-lable={VM-name}.
3. Export all VDIs xe vdi-export uuid={VDI-UUID} filename={VM-Name_xvda}.vhd format=vhd --progress (repeat for all VDIs).
4. Copy to destination host
5. Use vhd-util to get new UUID and rename VDI vhd-util read -p -n {VDI-UUID}.vhd
6. Shutdown VM. Do not delete VM before it will not start successfully on "new" host!
7. Get snapshot list xe snapshot-list
8. Get disks in snapshot xe snapshot-disk-list uuid={snapshot_UUID}
9. Export diff xe vdi-export uuid={main_VDI_UUID_from_step_3} filename=2920_delta_xvda.vhd format=vhd base={snapshot_VDI_UUID_from_step_8)} --progress
10. Copy diff to destination host
11. Import diff xe vdi-import uuid=VDI_UUID_from_step_5 filename=2920_delta_xvda.vhd format=vhd --progress

@mdraugh sure.

As workaround you can create simple table with MAC-IP-VM fields, and simple script to deploy new VM with first "free" MAC. Yes, it sounds like to develop your own XO, but I belive it should help alot.

It is not a problem at all.

You can always set MAC manually, or create VIF with certain MAC via XAPI or cli.

Link to XAPI https://xapi-project.github.io/xen-api/classes/vif.html
check create method.

Cli command
xe vif-create vm-uuid=<VM UUID> network-uuid=<NETWORK UUID> device=<ETHERNET INTERFACE NUMBER> mac=<MAC ADDRESS>
Device could be in range 0-15
MAC in format XX:XX:XX:XX:XX:XX

After you need "activate" new VIF by command
xe vif-plug uuid=<VIF UUID>
If VM does not running xe-guest-tools you have to switch off VM and power it on again to activate new VIF.

@Mark-C Thank you!
Could you tell please how you scale your storage with iSCSI? What hardware/software are you using for storage? Is it possible to add storage nodes on fly or you have to deploy new storag cluster every time when you grow?

@olivierlambert NFS and iSCSI have single point of failure. Yes, it is possible to deploy multipath iSCSI, but it is to complicated. I like CEPH RBD because it does not have single point of failure.
So I'm looking for something similar.

From my point of vew XOSTOR is good idea, but in some cases there is no need to use all nodes as xcp-ng hosts. For example you do not need large amount of RAM and fast modern CPU for storage cluster nodes.
I think the best solution in my case will be to deploy XOSTOR controller in xcp-ng cluster connected to separate storage cluster.

At first glance, I assume that it should be possible to connect storage cluster to xcp-ng with this command
linstor resource create node1 test1 --diskless

So the base idea is to use xcp-ng nodes for linstor-controllers/linstor-satellite and "storage" nodes as linstor-satellite only.

Hi!

I'm looking for new storage cluster for XCP-ng, because ceph RBD performance is very poor.
The main quetion now - is it possible to build XOSTOR (linstore) cluster separatly from xcp-ng and connect it over ethernet?
No inforamtion about such scenario in this article.
So I would like to have "compute" claster of xcp-ng nodes with fast local NVMe disks + and dedicated storage cluster with big amount of HDDs connected vie ethernet.

And second question is about scaling.
How this storage cluster could be scaled? Is it possible to add storage nodes online without interrupting clients (VMs)?

Thank you

Maybe netdata will cover everything?

There are no default alerts, but you can easily create them by yourself.

Also it is very easy to deploy "parent" netdata node and stream metrics to it from all hosts (maybe this part could be integrated to XOA free version? ).
You do not need netdata cloud account for this solution

@RS sorry, I have no idea how to fix it

You can always try to use XAPI, or xe cli tool

@RS Few years ago I faced the same errors on all xva backups and after that switch to vdi backups.
I did not find the root of problem, but I know how to "repair" broken backup.

Check xva backup with command vhd-util check -n name_of_backup.xva.
If you got something like Checksum : 0x0|0xffffffff (Bad!), the solution is:

1. Create VM or get server with modern Ubuntu. You will need x3 space of your backup size.
2. Copy broken backup
3. Install xva-img https://github.com/eriklax/xva-img

apt-get install libssl-dev g++ cmake
add-apt-repository ppa:ubuntu-toolchain-r/test && apt-get update && apt-get install -y gcc-7

#To compile execute following commands in xva-img folder

cmake ./
make install

4. Unpack backup

mkdir my-virtual-machine
tar -xf name_of_backup.xva.xva -C my-virtual-machine
chmod -R 755 my-virtual-machine

5. Create RAW disk. Replace Ref\:1 with your digits

xva-img -p disk-export my-virtual-machine/Ref\:1/ disk.raw

6. Install qemu-utils from standard repos apt install qemu-utils
7. Convert RAW to VHD qemu-img convert -f raw -O vpc disk.raw resotred.vhd
8. Copy VHD to xcp-ng SR cp resotred.vhd /run/sr-mount/{sr-uuid}/
9. Get UUID of your restored VHD. This command should be executed on xcp-ng server. vhd-util read -p -n resotred.vhd
10. Rename vhd
11. Rescan SR, rename VHD with empty name