@dhiraj26683 Sorry for late reply, I'm interested if you solve this issue? I have similar issue on one node in cluster, the other working well. When memory issue happened last time I moved all VMs on 'healty' node and on problematic node I applied: # free && sync && echo 3 > /proc/sys/vm/drop_caches && free After few seconds I was prompted with login (all the time server was responsive so I think it is not restarted). I know that doesn't solve problem of not releasing cache, but I didn't have to restart server, at least - maybe So if you have something better, please let me know!

@Danp oh yes you are right . . . boot is 500GB and SR is 4TB, though I removed the hardware mirror and am just using the 2 4TB drives so I have a SR of close to 8TB. I think my problem has been that the hardware raid was not working well with XCP, and I also was trying to use some older samsung SSDs (also 500GB) both of which did fail a long test in the system diagnostics. So I have removed both of them from the machine and will see how things work now (after yet another reinstall because I was not having luck getting XCP to see the second 4TB HDD after removing the HW mirror).

@Danp how can u resolve from this?

Late to the game, but this might help someone... I don't recommend this in a production environment, but it would be fine for e.g. a home lab. I built qemu from source on another host (in my case, on a laptop running Ubuntu Mate): ./configure --static make ... and then sftp'd the generated qemu-img executable to the XCP-ng host (I didn't care about any other outputs from the build process, some of which failed anyway likely due to the ancient version of Ubuntu Mate running on my laptop). Building with --static avoided any issues with missing libraries on the XCP-ng host, at least for qemu-img.

Thank you for that @olivierlambert I've tried with the Dell R710 and R730, without any luck - I did a lot more extensive tests with the R710, which led to the frustration of not "trying as hard" with the R730, but perhaps there are some "tricks" that I missed? I tried to enable NV with modprobe and the like (I know that's vague, but its a LOT of nuance to describe for the "what and how"). Since the R710 is fairly ancient, I would typically assume its "less capable" than the R730 for NV with XCP-ng, but then sometimes I run into oddities where the R710 actually outperforms the R730 "without additional configuration" (such as with my other post/comment I'm actively exploring/troubleshooting - https://xcp-ng.org/forum/post/89035 ) - Then there is also the incredibly tedious and annoying mxGPU "slicing" features from the limited GPUs that feature was available for that the major of the internet has reported to NOT work for them in any modern Kernel of Linux, (though a few "edge cases" have been reported sucessfull). So, "maybe" older hardware has better / more mature drivers in some cases? That said, I've retired the R710, so the point is somewhat moot, anyway - but it serves as an interesting use case that I've learned a great deal from "conceptually". (TL;DR - its true what they say about "assuming" ) But, rather than start tinkering again and potentially create another 3+ day project, are there any specific "success stories" with the R730, specifically that you could recommend? or perhaps a specific model / CPU combination where NV has been reported by others as working with XCP-ng? Perhaps specific UEFI/BIOS settings or "kernel hacks" that "tend to have a higher reported success"?

It's very likely in Xen itself since a part of the power management is done there. Another topic to bring during the Xen Winter Meetup @TeddyAstie

@gudge25 ...forget about it... Mikrotik won't fix v6 and they have a proprietary OS, so you can't install normal stuff.

Great news

@Tristis-Oris sharing my google drive link via chat... should be easy to upload it there

@knightjoel I have not but glad you found something and I wasn't (at least then) crazy.

@jasonnix Yes, a vCPU means a virtual CPU, which is the assignment of a VM to a physical CPU core. Servers have sockets that contain physical CPUs, so it sounds like your system has four sockets, holding four physical CPUs. Each physical CPU can have multiple cores and in some cases, one thread per core or in others, two threads per core, but let's stick to the simpler case here. A configuration of 4 cores with 1 core per socket means each of the 4 vCPUs will reside on a core on four separate physical CPU sockets, so all four physical CPUs are accessed. This is in most cases not ldeal as in many servers with 4 physical CPUs, the memory banks are split between pairs of CPUs, two on one bank, two on the other. Having VMs cross over physical CPU memory bank boundaries is generally inefficient and should be avoided if possible. This is why NUMA (Non-Uniform Memory Access) and vNUMA become important in the configuration. And @gskger is correct that licensing can sometimes depend on the configuration. I should add that under some circumstances, configuring the CPUs for turbo mode can be an advantage. Suggested reading: my article on the effects of vCPU and GPU allocations in a three-part set of articles. In particular, Part 3 addresses NUMA and configuration issues and Part 2 discusses turbo mode. I hope this helps as it is initially quite confusing. https://community.citrix.com/citrix-community-articles/a-tale-of-two-servers-how-bios-settings-can-affect-your-apps-and-gpu-performance/ https://community.citrix.com/citrix-community-articles/a-tale-of-two-servers-part-2-how-not-only-bios-settings-but-also-gpu-settings-can-affect-your-apps-and-gpu-performance/ https://community.citrix.com/citrix-community-articles/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/

@olivierlambert Will do... that VM is running a massive rsync job at the moment, but once it's finished I'll be able to shut it down.

IIRC, just remove it (small cross) so it should use something default

@mattnj Thanks for the update, I didn't realize you could select the kernel in the ipxe boot, this might help me down the road.

@MathieuRA It does indeed Sir. Thank you very much!

@nomad What do you mean by the grand reconfiguration? Is there a new version that changes how things work?

@flakpyro Thanks, that will probably save me an hour this summer if I decide to upgrade my production system. By then it might all be fixed.

@StormMaster Thank you for sharing your findings! @stephane-m-dev Thank you for the update! Looking forward to testing the fix just not sure how to replicate other than what @StormMaster found.