T
Offline
Originally an astronomer for 15 years and later, an NAU employee in IT for 25+ years, most of which as a Team Lead. I was a Citrix CTP and NVIDIA NGCA for four years prior to retirement. Over 10 years' experience with XenServer/Citrix Hypervisor and close to that with NVIDIA GRID products. I was also a Red Hat Linux administrator and system programmer. Still trying to contribute what knowledge I have for the benefit of the IT community.
Hi, everyone. Nice to see this project turning into reality. I will try to spend time here as possible, which is hard with already being spread thinly. I've been a XenServer user for around a decade and am as interesting in learning as well as contributing whatever knowledge might be helpful to the community.
Best regards,
-=Tobias
And from the CLI:
This is great to see, thank you for taking the time to rescue this; and thanks to @john.c for the recovery work and to @tjkreidl for writing it in the first place.
I went looking, and there is a small XCP-ng-specific piece on this in the official docs under NUMA affinity (https://docs.xcp-ng.org/compute#numa-affinity), but it's nothing like the depth of the Tale of Two Servers series, so having the originals archived is genuinely useful.
I won't pretend to judge how much of the 2019 BIOS and GPU-scheduler guidance still maps cleanly onto current hardware and XCP-ng versions; others here will know where it's aged and where it hasn't.
I'll make sure this is on our radar on the docs side, because it keeps coming up. Really appreciate you keeping this from disappearing.
Thank you kindly for your positive comments and appreciation. To me, it's amazing how some information can stay relevant for long periods of time, even given the rapid state of evolution in the technology sectors. I will try to get the full HTML docs uploaded soon, as well. There are a number of other XenServer articles I discovered a while back on a Polish server, and will see what else I can retrieve.
My original avocation for 15 years was that of an astronomer, so research is in my blood and diving into specific issues and doing extensive testing have always been a big part of my motivation to better understand as well as share knowledge.
@robyt It depends on (1) licensing, if any, as some licenses go by cores vs. sockets, and (2) NUMA/VNUMA depending on how critical the performance is depending on how the VCPUs get allocated between sockets or on a single socket. Best way IMO is to try all and test with benchmarks. See, for example, this article and the previous two articles, as well as articles by Frank Denneman and others: https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/
@olivierlambert said in NUMA-impact - Xeon/Epyc - 1P vs 2P:
There is no universal answer (because it's mostly depending on your VM load and what do you expect). As usual, my advice is to keep it simple if you don't have a problem with it (ie: you are satisfied by the perf.). Even a default EPYC configuration will be likely always better than a Xeon one.
After that, if you want to go deeper and learn the details, it's OK, let me just ping @tjkreidl who did a remarkable job (if I remember correctly) on this very topic.
Thanks for the mention, @olivierlambert ! Here's a link to part 3, which contains links back to parts 1 and 2. Note that NUMA will affect EPYC processors differently as they changed the die configuration at one point with the number of cores. I'm open for any questions on this topic. 
https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/
@epretorious I would add that you have to be careful about overprovisioning when NUMA/vNUMA kicks in, that is when you allocate more VCPUs to exceed the number of physical CPUs of a bank of them as well as the associated physical memory (assume, for the sake of argument, you have two banks of physical CPUs and each has directly accessible to it one of two banks of memory) then things get inefficient because a CPU may need to go across to a different bank of memory to access data and there is additional overhead involved. See for example this article and the two preceding it:
https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/
-=Tobias
@MichaelCropper CPUs can be over-provisoned, but not memory. You can use DMC (dynamic memory control) to regulate how much memory a VM will actually use, but in total, you still cannot exceed the total amount of physical memory available on a server.
CPU over-provisioning is very common, especially if loads change significantly over time (day/night weekday/weekend, special event and holidays/regular days, etc.).
Watching the load with top and xentop will give you an idea about overall performance of dom0 and all VMs, respectively.
As to a VM powred off, it will use up neither memory nor CPU resources.
There are a lot of subtleties involved that would entail a much longer discussion, but hopefully this will help for starters. You can google a lot of information about memory and VCPU allocation; there is a lot of information out there.
@olivierlambert Agreed. The Citrix forum used to be very active, but especially since Citrix was taken over, https://community.citrix.com has had way less activity, sadly.
It's still gratifying that a lot of the functionality still is common to both platforms, although as XCP-ng evolves, there will be continually less commonality.
@Chrome Cheers -- always glad to help out. I put in many thousands of posts on the old Citrix XenServer site, and am happy to share whatever knowledge I still have, as long as it's still relevant! In a few years, it probably won't be, so carpe diem!
@Chrome Fantastic! Please mark my post as helpful if you found it as such. Was traveling much of today, hence the late response.
BTW, it's always good to make a backup and/or archive of your LVM configuration anytime you change it, as the restore option is the cleanest way to deal with connectivity issues if there is some sort of corruption. It's saved my rear end before, I can assure you!
Yeah, if the SSD drive got wiped, there's no option to get those back unless you made a backup somewhere of all that before you installed XCP-ng onto it.
BTW, another very useful command for LVM is "vgchange -ay" which will attempt to renew VG information if a VG seems missing or the like.
@coolsport00 Sorry about the VMW need for the Cisco product. Sounds like you have a number of constraints, finances being I'm sure one of them! At least you have time on your hands and the means to experiment. You may, als, end up with a number of different platforms to meet your needs. We ran both Sun Microsystems and Red Hat Linux and Microsoft Windows servers, each taking on specific duties. It's far from ideal and probably not very cost-effective, but you do what you have to to get stuff to work.
@coolsport00 While I was working at the university, we used RDS on Windows servers with RTX GPus with good results. The T4 I believe required NVIDIA licensing.
You have a few options: 1) Upgrade your servers if possible, 2) Add more hosts and GPUS (the used equipment market is one option, if you want to risk unsupported equipment),
3) If the GPU is the limitation and not the host, consider upgrading to a 5000 or 6000 series GPU. oOr NVIDIA GPUs to leverage the RTX vWS and CUDA capabilities, the latter's support being particularly important for Revit, AutoCaD, and other high-end graphics apps. We used P4 and T4 GPUs at the time with success under XenServer and with the high-end NVIDIA licensing (which unfortunately requires, as I'm sure you know, a separate VM running the licensing server).
In any case, you are fortunate to be able to test this all out way ahead of before you have to go live.
@coolsport00 Why not then just do the GPU passthrough for the entire host and just put the VMs on that host that you want to leverage the GPU? To get specific vGPU capabilities, my recollection is that you'd need to use a GPU that most likely requires licensing, at least as far as NVIDIA is concerned.
@coolsport00 See if any of this helps:
Here is the step-by-step guide to configure the NVIDIA GeForce RTX 3090 for full PCI passthrough to a virtual machine in XCP-ng.Step 1: Identify the GPU on the XCP-ng HostFirst, log in to your XCP-ng host via SSH or use the local console shell to find the exact hardware address of your RTX 3090.Run the following command to list your PCI devices and filter for NVIDIA:bashlspci | grep -i nvidia
Use code with caution.Look for the lines corresponding to your RTX 3090 and its Audio Controller. They will look similar to this:text01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
Use code with caution.Take note of the PCI identifiers (e.g., 01:00.0 and 01:00.1). Note that your specific identifier numbers might vary based on your motherboard layout.Step 2: Hide the GPU from the Control Domain (Dom0)To prevent the main XCP-ng host system (Dom0) from seizing control of the graphics card, you must tell it to ignore those PCI IDs on boot.Instruct XCP-ng to assign the GPU components to the passthrough driver (xen-pciback) by executing this command (replace 01:00.0 and 01:00.1 with your actual IDs found in Step 1):bash/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(01:00.0)(01:00.1)"
Use code with caution.Reboot the physical server for this driver isolation configuration to take effect:bashreboot
Use code with caution.Step 3: Verify the IsolationOnce your host boots back up, check that the hardware isolation worked as intended.Run this command to verify that the xen-pciback driver has correctly claimed the card:bashxl pci-assignable-list
Use code with caution.You should see your GPU addresses printed in the terminal, confirming they are ready to be passed into a virtual environment:text0000:01:00.0
0000:01:00.1
Use code with caution.Step 4: Attach the GPU to your VM
You can attach the isolated card either visually using Xen Orchestra (XO) or straight from the command line interface.Option A: Using Xen Orchestra (Recommended)Navigate to the Xen Orchestra management interface.Go to the VMs tab and click on the target virtual machine (ensure it is powered down).Click on the Advanced tab of that VM.Scroll down to the PCI devices block.Select both the RTX 3090 graphics and audio IDs from the dropdown checklist and save your changes.Option B: Using the Command Line (CLI)Find the unique identifier (UUID) of your VM:bashxe vm-list name-label="Your_VM_Name"
Use code with caution.Assign the GPU to that VM UUID using the following command syntax:bashxe vm-param-set uuid=<VM_UUID> other-config:pci=0/0000:01:00.0,0/0000:01:00.1
Use code with caution.Step 5: Boot and Install DriversPower on your virtual machine. You can now download and install the official desktop NVIDIA GeForce Drivers directly inside the guest operating system. The installer will recognize the device normally, and you do not need to register a license server or install vGPU client software.
@Dezerd To check if there is perhaps a hung process that is interfering. I assume there is sufficient memory and storage space. Do any I/O disk errors show in /var/log/SMlog ? Just to clarify. are there VMs that do not show any such error condition?
@Dezerd What does "xe task-list" show as far as active processes?
@Pilow Yeah, I'd run iostat and look to see how th resources are being limited, I'd run something like "iostat -dtkx 10" so you get extended stats every 10 seconds during that replication process and look at the wait, queue states, etc. to see if that helps identify any bottlenecks.
@poddingue Thank you kindly! Honestly, whatever organizational structure you think is best is fine by me.
@poddingue I am confused is how your updated articles can be accessed somewhere on my main Github page. I only see them if I follow your "pull" link. What needs to be done to commit them?
Thank you!
@pierrebrunet Yes, that looks very good. Some of the images are stacked instead of side-by-side, but who cares?
Thanks ever so much again!
