Ok, update on the update issues:
I could not migrate VMs from the older backup server which was upgraded to 8.3 RC1 back to 8.2 install, so I'm forced with having to continue troubleshooting it.
Today I tried to start a CD install from efi shell and I finally got an error / output as to why install fails silently:

"This build of Xen requires NX support"

I guess I had it disabled in the BIOS, just wish the install would have output the error with a wait prompt or something, rather than a silent failure.

Edit: Confirmed; after enabling execution disable option within bios, installer booted without issues and the install is currently ongoing.

Ok, found a solution.

Seems there is nothing I could do via cli or xo-ce to see the virtual GPUs, had to install xcp-ng center, specifically from:
https://github.com/xcp-ng/xenadmin/files/13800584/XCP-NG-Center-2023-Release.zip

Using the center, I was able to assign the GPU to the VM and everything worked well from then on.
Now to figure out licensing....

Re: nVidia Tesla P4 for vgpu and Plex encoding

Thanks for various guides I was able to install the driver on xcp-ng 8.3
I've used NVIDIA-vGPU-CitrixHypervisor-8.2-570.124.03.x86_64.iso
and vgpu-7.4.16-1.xs8.x86_64.rpm (from xen server)

The driver seems to work and T4 is detected

# nvidia-smi
Thu Apr  3 17:18:25 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.03             Driver Version: 570.124.03     CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:02:00.0 Off |                    0 |
| N/A   86C    P0             42W /   70W |      13MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Unfortunately the card doesn't seem to be in vGPU mode:

# nvidia-smi vgpu -q
No supported devices in vGPU mode

I'm not sure if "Addressing Mode: Unknown Error" is anything to be concerned about, cannot find anything specific about that.

#nvidia-smi -q
==============NVSMI LOG==============

Timestamp                                 : Thu Apr  3 16:53:35 2025
Driver Version                            : 570.124.03
CUDA Version                              : Not Found

Attached GPUs                             : 1
GPU 00000000:02:00.0
    Product Name                          : Tesla T4
    Product Brand                         : NVIDIA
    Product Architecture                  : Turing
    Display Mode                          : Enabled
    Display Active                        : Disabled
    Persistence Mode                      : Disabled
    Addressing Mode                       : Unknown Error
...
    GPU Virtualization Mode
        Virtualization Mode               : None
        Host VGPU Mode                    : N/A
        vGPU Heterogeneous Mode           : N/A
...

I also see that vGPU / sr-iov (as least theoretically) supported:

# lspci -v -s 02:00.0
02:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
        Subsystem: NVIDIA Corporation Device 12a2
        Physical Slot: 6
        Flags: bus master, fast devsel, latency 0, IRQ 32
        Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 383fc0000000 (64-bit, prefetchable) [size=256M]
        Memory at 383ff0000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] #00 [0080]
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Capabilities: [bb0] #15
        Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: nvidia
        Kernel modules: nvidia

I do have SR-IOV enabled in BIOS, but having it disabled didn't seem to change anything.

Those with nvidia vGPUs, how have you ended up getting xcp-ng to enable the feature?

Ok, found a solution.

Seems there is nothing I could do via cli or xo-ce to see the virtual GPUs, had to install xcp-ng center, specifically from:
https://github.com/xcp-ng/xenadmin/files/13800584/XCP-NG-Center-2023-Release.zip

Using the center, I was able to assign the GPU to the VM and everything worked well from then on.
Now to figure out licensing....

Re: nVidia Tesla P4 for vgpu and Plex encoding

Thanks for various guides I was able to install the driver on xcp-ng 8.3
I've used NVIDIA-vGPU-CitrixHypervisor-8.2-570.124.03.x86_64.iso
and vgpu-7.4.16-1.xs8.x86_64.rpm (from xen server)

The driver seems to work and T4 is detected

# nvidia-smi
Thu Apr  3 17:18:25 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.03             Driver Version: 570.124.03     CUDA Version: N/A      |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:02:00.0 Off |                    0 |
| N/A   86C    P0             42W /   70W |      13MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Unfortunately the card doesn't seem to be in vGPU mode:

# nvidia-smi vgpu -q
No supported devices in vGPU mode

I'm not sure if "Addressing Mode: Unknown Error" is anything to be concerned about, cannot find anything specific about that.

#nvidia-smi -q
==============NVSMI LOG==============

Timestamp                                 : Thu Apr  3 16:53:35 2025
Driver Version                            : 570.124.03
CUDA Version                              : Not Found

Attached GPUs                             : 1
GPU 00000000:02:00.0
    Product Name                          : Tesla T4
    Product Brand                         : NVIDIA
    Product Architecture                  : Turing
    Display Mode                          : Enabled
    Display Active                        : Disabled
    Persistence Mode                      : Disabled
    Addressing Mode                       : Unknown Error
...
    GPU Virtualization Mode
        Virtualization Mode               : None
        Host VGPU Mode                    : N/A
        vGPU Heterogeneous Mode           : N/A
...

I also see that vGPU / sr-iov (as least theoretically) supported:

# lspci -v -s 02:00.0
02:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)
        Subsystem: NVIDIA Corporation Device 12a2
        Physical Slot: 6
        Flags: bus master, fast devsel, latency 0, IRQ 32
        Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
        Memory at 383fc0000000 (64-bit, prefetchable) [size=256M]
        Memory at 383ff0000000 (64-bit, prefetchable) [size=32M]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] #00 [0080]
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [c8] MSI-X: Enable- Count=6 Masked-
        Capabilities: [100] Virtual Channel
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Capabilities: [bb0] #15
        Capabilities: [bcc] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [c14] Alternative Routing-ID Interpretation (ARI)
        Kernel driver in use: nvidia
        Kernel modules: nvidia

I do have SR-IOV enabled in BIOS, but having it disabled didn't seem to change anything.

Those with nvidia vGPUs, how have you ended up getting xcp-ng to enable the feature?

@yann Good catch. I tend to use net install when available, as it is faster than emulated iso on supermicro motherboards. When trying beta 2 install, I did use the network source by using xcp-ng-8.3.0-beta2-netinstall.iso.
To test things further, I used my backup server (Supermicro X10DRT-H, with E5-2630 V3) and disabled execute disable bit. Verified that 8.3 RC1 install gives the same error "Xen requires NX support" and ran beta 2 install from local media. This time 8.3 beta 2 was installed without issues and after a reboot I verified that execute was still disabled in the bios.

When using RC1 installer:

Beta 2 installer with local media this time:

Forgot to mention that I did try installing RC1 via legacy bios onto the system (installer would work in that case), but I could not get the system to recognize boot disk (NVME) after the install (installer itself saw and could use said disk).
Ended up discovering issue with EFI installer prior to fixing the legacy path as I've not spent much time on that since legacy bios would not be supported down the road.

@yann Never got that far, 8.3 beta 2 would not boot after the install.
Difference with RC1 is that the beta 2 installer itself did not check, or could re-enable NX support, but the actual boot of beta 2 would fail similar to RC1 installer (likely due to NX support being disabled).
So I could install, but not boot, the beta 2 onto the system. My previous posts don't provide much detail, but here is how it went: https://xcp-ng.org/forum/post/80208

@andyhhp thanks, seems issue was most visible within RC1 install. 8.2 worked well with NX disabled (I guess by activating it on it's own), beta 2 would install, but not boot (have not tried beta 1), and RC1 would fail seemingly silently.

The efi path is not to activate the NX setting, it was just me getting to the cd directory with the bootx64.efi launcher for the installer. Running it this way, rather than allowing BIOS to boot into CD directly is what allowed me to see the output of the installer prior to exit: "This build of Xen requires NX support". Normally the next boot option would take over and this text would not be seen.

My setup is Supermicro X10SRM-TF motherboard with E5-2699 V4 processor, latest bios, UEFI boot.
I don't even remember when I disabled NX support, perhaps couple of years ago when playing around with increasing single core performance.

@olivierlambert Thank you. Install went smoothly and I'm in the process of transferring all of the VMs from the backup server back onto my main workhorse.

I just wish I remembered an old troubleshooting tip:
When something odd is going on, reset the bios to defaults (or something along those lines).

I did figure out a way to run the installer from efi shell; never done that before. Wish I thought of it sooner as well (no telling how many times I've rebooted this system trying to troubleshoot things.

Ok, update on the update issues:
I could not migrate VMs from the older backup server which was upgraded to 8.3 RC1 back to 8.2 install, so I'm forced with having to continue troubleshooting it.
Today I tried to start a CD install from efi shell and I finally got an error / output as to why install fails silently:

"This build of Xen requires NX support"

I guess I had it disabled in the BIOS, just wish the install would have output the error with a wait prompt or something, rather than a silent failure.

Edit: Confirmed; after enabling execution disable option within bios, installer booted without issues and the install is currently ongoing.

I've tried to do a "clean" beta 2 install, same issue (efi shell).
I've changed bios to legacy boot and was able to run a RC1 installer (nice bootup graphic and all), but it warns you that such method will not be supported:

So I ended up using said installer to recover 8.2 install from backup and I'm back up and running at 8.2 (via EFI). Not sure what part of my config could cause issues with 8.3.

Decided to try updating to beta 2 and... install went smooth, but system now refuses to boot.
Seems issue is similar to RC1 install. Boot fails silently, similar to install on RC1, and next boot option is executed. But while the RC1 failed into a working 8.2 install, 8.3 boot fails into EFI shell.