Offline
Originally an astronomer for 15 years and later, an NAU employee in IT for 25+ years, most of which as a Team Lead. I was a Citrix CTP and NVIDIA NGCA for four years prior to retirement. Over 10 years' experience with XenServer/Citrix Hypervisor and close to that with NVIDIA GRID products. I was also a Red Hat Linux administrator and system programmer. Still trying to contribute what knowledge I have for the benefit of the IT community.
Hi, everyone. Nice to see this project turning into reality. I will try to spend time here as possible, which is hard with already being spread thinly. I've been a XenServer user for around a decade and am as interesting in learning as well as contributing whatever knowledge might be helpful to the community.
Best regards,
-=Tobias
And from the CLI:
This is great to see, thank you for taking the time to rescue this; and thanks to @john.c for the recovery work and to @tjkreidl for writing it in the first place.
I went looking, and there is a small XCP-ng-specific piece on this in the official docs under NUMA affinity (https://docs.xcp-ng.org/compute#numa-affinity), but it's nothing like the depth of the Tale of Two Servers series, so having the originals archived is genuinely useful.
I won't pretend to judge how much of the 2019 BIOS and GPU-scheduler guidance still maps cleanly onto current hardware and XCP-ng versions; others here will know where it's aged and where it hasn't.
I'll make sure this is on our radar on the docs side, because it keeps coming up. Really appreciate you keeping this from disappearing.
Thank you kindly for your positive comments and appreciation. To me, it's amazing how some information can stay relevant for long periods of time, even given the rapid state of evolution in the technology sectors. I will try to get the full HTML docs uploaded soon, as well. There are a number of other XenServer articles I discovered a while back on a Polish server, and will see what else I can retrieve.
My original avocation for 15 years was that of an astronomer, so research is in my blood and diving into specific issues and doing extensive testing have always been a big part of my motivation to better understand as well as share knowledge.
@robyt It depends on (1) licensing, if any, as some licenses go by cores vs. sockets, and (2) NUMA/VNUMA depending on how critical the performance is depending on how the VCPUs get allocated between sockets or on a single socket. Best way IMO is to try all and test with benchmarks. See, for example, this article and the previous two articles, as well as articles by Frank Denneman and others: https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/
@olivierlambert said in NUMA-impact - Xeon/Epyc - 1P vs 2P:
There is no universal answer (because it's mostly depending on your VM load and what do you expect). As usual, my advice is to keep it simple if you don't have a problem with it (ie: you are satisfied by the perf.). Even a default EPYC configuration will be likely always better than a Xeon one.
After that, if you want to go deeper and learn the details, it's OK, let me just ping @tjkreidl who did a remarkable job (if I remember correctly) on this very topic.
Thanks for the mention, @olivierlambert ! Here's a link to part 3, which contains links back to parts 1 and 2. Note that NUMA will affect EPYC processors differently as they changed the die configuration at one point with the number of cores. I'm open for any questions on this topic. 
https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/
@epretorious I would add that you have to be careful about overprovisioning when NUMA/vNUMA kicks in, that is when you allocate more VCPUs to exceed the number of physical CPUs of a bank of them as well as the associated physical memory (assume, for the sake of argument, you have two banks of physical CPUs and each has directly accessible to it one of two banks of memory) then things get inefficient because a CPU may need to go across to a different bank of memory to access data and there is additional overhead involved. See for example this article and the two preceding it:
https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/
-=Tobias
@MichaelCropper CPUs can be over-provisoned, but not memory. You can use DMC (dynamic memory control) to regulate how much memory a VM will actually use, but in total, you still cannot exceed the total amount of physical memory available on a server.
CPU over-provisioning is very common, especially if loads change significantly over time (day/night weekday/weekend, special event and holidays/regular days, etc.).
Watching the load with top and xentop will give you an idea about overall performance of dom0 and all VMs, respectively.
As to a VM powred off, it will use up neither memory nor CPU resources.
There are a lot of subtleties involved that would entail a much longer discussion, but hopefully this will help for starters. You can google a lot of information about memory and VCPU allocation; there is a lot of information out there.
@olivierlambert Agreed. The Citrix forum used to be very active, but especially since Citrix was taken over, https://community.citrix.com has had way less activity, sadly.
It's still gratifying that a lot of the functionality still is common to both platforms, although as XCP-ng evolves, there will be continually less commonality.
@Chrome Cheers -- always glad to help out. I put in many thousands of posts on the old Citrix XenServer site, and am happy to share whatever knowledge I still have, as long as it's still relevant! In a few years, it probably won't be, so carpe diem!
@Chrome Fantastic! Please mark my post as helpful if you found it as such. Was traveling much of today, hence the late response.
BTW, it's always good to make a backup and/or archive of your LVM configuration anytime you change it, as the restore option is the cleanest way to deal with connectivity issues if there is some sort of corruption. It's saved my rear end before, I can assure you!
Yeah, if the SSD drive got wiped, there's no option to get those back unless you made a backup somewhere of all that before you installed XCP-ng onto it.
BTW, another very useful command for LVM is "vgchange -ay" which will attempt to renew VG information if a VG seems missing or the like.
@Pilow Yeah, I'd run iostat and look to see how th resources are being limited, I'd run something like "iostat -dtkx 10" so you get extended stats every 10 seconds during that replication process and look at the wait, queue states, etc. to see if that helps identify any bottlenecks.
@poddingue Thank you kindly! Honestly, whatever organizational structure you think is best is fine by me.
@poddingue I am confused is how your updated articles can be accessed somewhere on my main Github page. I only see them if I follow your "pull" link. What needs to be done to commit them?
Thank you!
@pierrebrunet Yes, that looks very good. Some of the images are stacked instead of side-by-side, but who cares?
Thanks ever so much again!
@poddingue Wow, fantastic! Your time and efforts are most appreciated. And I'd be more that willing to have the contents put on the docs.xcp-ng.org site. I'm just pleased that the information still can be found useful and will be able to serve the community for some time to come.
My only question is with these additions, is it possible to view the full articles similar to how the PDF and improved Word documents I hacked render that first article? I'm afraid I'm not savvy enough about Github to understand this. Many thanks again!
Folks, I made the effort to convert the first blog article from PDF to MS Word .docx format where I was able to much more easily edit it and clean it up. It's now there along with the messier PDF version on the Github site. When I get a chance. I plan to tackle the other two articles. Thank you for your patience. Note that to see the Word Doc, you have to download it as Github itself doesn't render it.
@Aleksander I fear that all these -- even now the RTX cards from NVIDIA -- generally require licensing when run on servers, whether for pass-through or assigned to individual VMs. As well, installing an NVIDIA license server is then necessary. They have become more strict about this over the past several years, as RTX cards used to be exempt.
There are, however, some exceptions, but it's complicated and confusing! For details, consult NVIDIA documents such as this one: https://docs.nvidia.com/vgpu/latest/grid-licensing-user-guide/index.html
P.S. Unfortunately, since retirement a few years ago, I no longer have any hands-on equipment at my disposal.
@poddingue Good evening. Honestly, I'd be totally fine with your taking over a full copy of everything over to XCP-ng and augmenting it over time as new information becomes available. It'd stand a better change of longevity there than in my hands, and all I'd ask for is acknowledgement for the original work. There's no real need for me to keep a totally separate copy of the materials, which would only lead to confusion as things likely would diverse over time.
As mentioned above, there are more forum articles and such out there, and when I have time, I'll see if I can hunt some more of those down, as well.
I really appreciate what you folks are doing at XCP-ng and the time and effort you're putting into your products. Above all, the feedback and communication have been better than pretty much anyone else I've ever dealt with in the IT industry.
P.S. @johnnezero did a super summarizing job with his "Series Overview & Quick Reference" that includes some of the most useful commands in addition to a synopsis of each of the articles. Very impressive!
I tried adding the HTML file bundles and made a horrible mess, which took forever for me to clean up. The on-line instructions were not very helpful. What I saved as files seemed to contain also way more stuff than it should have, some seemingly unrelated to the original blog. Sigh. I'm no Github expert, that's for sure.
This is great to see, thank you for taking the time to rescue this; and thanks to @john.c for the recovery work and to @tjkreidl for writing it in the first place.
I went looking, and there is a small XCP-ng-specific piece on this in the official docs under NUMA affinity (https://docs.xcp-ng.org/compute#numa-affinity), but it's nothing like the depth of the Tale of Two Servers series, so having the originals archived is genuinely useful.
I won't pretend to judge how much of the 2019 BIOS and GPU-scheduler guidance still maps cleanly onto current hardware and XCP-ng versions; others here will know where it's aged and where it hasn't.
I'll make sure this is on our radar on the docs side, because it keeps coming up. Really appreciate you keeping this from disappearing.
Thank you kindly for your positive comments and appreciation. To me, it's amazing how some information can stay relevant for long periods of time, even given the rapid state of evolution in the technology sectors. I will try to get the full HTML docs uploaded soon, as well. There are a number of other XenServer articles I discovered a while back on a Polish server, and will see what else I can retrieve.
My original avocation for 15 years was that of an astronomer, so research is in my blood and diving into specific issues and doing extensive testing have always been a big part of my motivation to better understand as well as share knowledge.
