@coolsport00 Sorry about the VMW need for the Cisco product. SOunds like you have a number of constraints, finances being I'm sure one of them! At least you have time on your hands and the means to experiment. You may, als, end up with a number of different platforms to meet your needs. We ran both Sun Microsystems and Red Hat Linux and Microsoft WIndows servers, each taking on specific duties. It's far from ideal and probably not very cost-effective, but you do what you have to to get stuff to work.

@coolsport00 While I was working at the university, we used RDS on Windows servers with RTX GPus with good results. The T4 I believe required NVIDIA licensing.
You have a few options: 1) Upgrade your servers if possible, 2) Add more hosts and GPUS (the used equipment market is one option, if you want to risk unsupported equipment),
3) If the GPU is the limitation and not the host, consider upgrading to a 5000 or 6000 series GPU. oOr NVIDIA GPUs to leverage the RTX vWS and CUDA capabilities, the latter's support being particularly important for Revit, AutoCaD, and other high-end graphics apps. We used P4 and T4 GPUs at the time with success under XenServer and with the high-end NVIDIA licensing (which unfortunately requires, as I'm sure you know, a separate VM running the licensing server).

In any case, you are fortunate to be able to test this all out way ahead of before you have to go live.

@coolsport00 Why not then just do the GPU passthrough for the entire host and just put the VMs on that host that you want to leverage the GPU? To get specific vGPU capabilities, my recollection is that you'd need to use a GPU that most likely requires licensing, at least as far as NVIDIA is concerned.

@coolsport00 See if any of this helps:

Here is the step-by-step guide to configure the NVIDIA GeForce RTX 3090 for full PCI passthrough to a virtual machine in XCP-ng.Step 1: Identify the GPU on the XCP-ng HostFirst, log in to your XCP-ng host via SSH or use the local console shell to find the exact hardware address of your RTX 3090.Run the following command to list your PCI devices and filter for NVIDIA:bashlspci | grep -i nvidia
Use code with caution.Look for the lines corresponding to your RTX 3090 and its Audio Controller. They will look similar to this:text01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Use code with caution.Take note of the PCI identifiers (e.g., 01:00.0 and 01:00.1). Note that your specific identifier numbers might vary based on your motherboard layout.Step 2: Hide the GPU from the Control Domain (Dom0)To prevent the main XCP-ng host system (Dom0) from seizing control of the graphics card, you must tell it to ignore those PCI IDs on boot.Instruct XCP-ng to assign the GPU components to the passthrough driver (xen-pciback) by executing this command (replace 01:00.0 and 01:00.1 with your actual IDs found in Step 1):bash/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(01:00.0)(01:00.1)"
Use code with caution.Reboot the physical server for this driver isolation configuration to take effect:bashreboot
Use code with caution.Step 3: Verify the IsolationOnce your host boots back up, check that the hardware isolation worked as intended.Run this command to verify that the xen-pciback driver has correctly claimed the card:bashxl pci-assignable-list
Use code with caution.You should see your GPU addresses printed in the terminal, confirming they are ready to be passed into a virtual environment:text0000:01:00.0
0000:01:00.1
Use code with caution.Step 4: Attach the GPU to your VM
You can attach the isolated card either visually using Xen Orchestra (XO) or straight from the command line interface.Option A: Using Xen Orchestra (Recommended)Navigate to the Xen Orchestra management interface.Go to the VMs tab and click on the target virtual machine (ensure it is powered down).Click on the Advanced tab of that VM.Scroll down to the PCI devices block.Select both the RTX 3090 graphics and audio IDs from the dropdown checklist and save your changes.Option B: Using the Command Line (CLI)Find the unique identifier (UUID) of your VM:bashxe vm-list name-label="Your_VM_Name"
Use code with caution.Assign the GPU to that VM UUID using the following command syntax:bashxe vm-param-set uuid=<VM_UUID> other-config:pci=0/0000:01:00.0,0/0000:01:00.1
Use code with caution.Step 5: Boot and Install DriversPower on your virtual machine. You can now download and install the official desktop NVIDIA GeForce Drivers directly inside the guest operating system. The installer will recognize the device normally, and you do not need to register a license server or install vGPU client software.

@Dezerd To check if there is perhaps a hung process that is interfering. I assume there is sufficient memory and storage space. Do any I/O disk errors show in /var/log/SMlog ? Just to clarify. are there VMs that do not show any such error condition?

@Dezerd What does "xe task-list" show as far as active processes?

@Pilow Yeah, I'd run iostat and look to see how th resources are being limited, I'd run something like "iostat -dtkx 10" so you get extended stats every 10 seconds during that replication process and look at the wait, queue states, etc. to see if that helps identify any bottlenecks.

@poddingue Thank you kindly! Honestly, whatever organizational structure you think is best is fine by me.

@poddingue I am confused is how your updated articles can be accessed somewhere on my main Github page. I only see them if I follow your "pull" link. What needs to be done to commit them?
Thank you!

@pierrebrunet Yes, that looks very good. Some of the images are stacked instead of side-by-side, but who cares?
Thanks ever so much again!

@poddingue Wow, fantastic! Your time and efforts are most appreciated. And I'd be more that willing to have the contents put on the docs.xcp-ng.org site. I'm just pleased that the information still can be found useful and will be able to serve the community for some time to come.

My only question is with these additions, is it possible to view the full articles similar to how the PDF and improved Word documents I hacked render that first article? I'm afraid I'm not savvy enough about Github to understand this. Many thanks again!

Folks, I made the effort to convert the first blog article from PDF to MS Word .docx format where I was able to much more easily edit it and clean it up. It's now there along with the messier PDF version on the Github site. When I get a chance. I plan to tackle the other two articles. Thank you for your patience. Note that to see the Word Doc, you have to download it as Github itself doesn't render it.

@Aleksander I fear that all these -- even now the RTX cards from NVIDIA -- generally require licensing when run on servers, whether for pass-through or assigned to individual VMs. As well, installing an NVIDIA license server is then necessary. They have become more strict about this over the past several years, as RTX cards used to be exempt.
There are, however, some exceptions, but it's complicated and confusing! For details, consult NVIDIA documents such as this one: https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/index.html
P.S. Unfortunately, since retirement a few years ago, I no longer have any hands-on equipment at my disposal.

@poddingue Good evening. Honestly, I'd be totally fine with your taking over a full copy of everything over to XCP-ng and augmenting it over time as new information becomes available. It'd stand a better change of longevity there than in my hands, and all I'd ask for is acknowledgement for the original work. There's no real need for me to keep a totally separate copy of the materials, which would only lead to confusion as things likely would diverse over time.
As mentioned above, there are more forum articles and such out there, and when I have time, I'll see if I can hunt some more of those down, as well.
I really appreciate what you folks are doing at XCP-ng and the time and effort you're putting into your products. Above all, the feedback and communication have been better than pretty much anyone else I've ever dealt with in the IT industry.
P.S. @johnnezero did a super summarizing job with his "Series Overview & Quick Reference" that includes some of the most useful commands in addition to a synopsis of each of the articles. Very impressive!

I tried adding the HTML file bundles and made a horrible mess, which took forever for me to clean up. The on-line instructions were not very helpful. What I saved as files seemed to contain also way more stuff than it should have, some seemingly unrelated to the original blog. Sigh. I'm no Github expert, that's for sure.

@poddingue said:

This is great to see, thank you for taking the time to rescue this; and thanks to @john.c for the recovery work and to @tjkreidl for writing it in the first place.
I went looking, and there is a small XCP-ng-specific piece on this in the official docs under NUMA affinity (https://docs.xcp-ng.org/compute#numa-affinity), but it's nothing like the depth of the Tale of Two Servers series, so having the originals archived is genuinely useful.
I won't pretend to judge how much of the 2019 BIOS and GPU-scheduler guidance still maps cleanly onto current hardware and XCP-ng versions; others here will know where it's aged and where it hasn't.
I'll make sure this is on our radar on the docs side, because it keeps coming up. Really appreciate you keeping this from disappearing.

Thank you kindly for your positive comments and appreciation. To me, it's amazing how some information can stay relevant for long periods of time, even given the rapid state of evolution in the technology sectors. I will try to get the full HTML docs uploaded soon, as well. There are a number of other XenServer articles I discovered a while back on a Polish server, and will see what else I can retrieve.
My original avocation for 15 years was that of an astronomer, so research is in my blood and diving into specific issues and doing extensive testing have always been a big part of my motivation to better understand as well as share knowledge.

@johnnezero The full HTML versions will render much better. The PDF conversion is less than perfect. iIll try to get those uploaded, as well.

@johnnezero Thanks, man! I hope you and others find it and the related other two articles useful.

@johnnezero Cool -- you guys probably have a way better idea of a proper layout and organization of the materials. Thank you!

@john.c They are under two branches ... do you not see them?
XenServer-Articles and XenServer-Forum-Threads -- they landed under the "Branches" section ...