xe command "hangs on start" when max / open files is high
I don't propose this needs "fixing", but it took some time to triage so sharing:
We are using
xapi-xe_1.249.19-2.3_amd64.deboutside of the xen server host. We found in some situations: any execution that invoked communication with a xen server, would pause / hang for ~10 mins:
via strace, we found that running
xecauses an enumeration over
ulimit -a <snip> open files (-n) 1073741816
on xcp-ng / xen host this appears to still be 1024. To remove this pause / hang, a possibility is to:
ulimit -n <your-choice>
<your-choice>is a value you deem sensible for your system.
Hope that saves someone some time
olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑💼
Thanks for the feedback That's another good reason to move forward on our REST API in XO
Looking forward to it: fyi, we are currently using this to facilitate the transfer of templates between clusters. We didn't find it possible to achieve that via xo-cli, though we're open to the possibility we failed to understand something
Can you show us how do you achieve that with
xeso we can check how/if it's possible to do it via XO?
we have more than one "destination" server, but simplified here for only one:
check if destination server has a template with name "x" present
xe -s <server> -u <user> -pw <pw> template-list name-label=x
if present, do nothing.
if absent on this destination, check if we have that template locally
# this is just a filesystem check where we are running
if absent locally, pull it from our source server
# not captured here: lookup template uuid for 'x'
xe -s <server> -u <user> -pw <pw> template-export template-uuid='uuid' filename='templatename'
push to destination
xe -s <server> -u <user> -pw <pw> vm-import filename='templatename'
Where we have multiple targets, loop over remaining destinations, and push when not detected as being present.
template-list template-export vm-import
Are what I think we need. I wasn't the most recent to look at using xo-cli for this, but we either didn't find them all, or somehow didn't understand how to use what we need
So all in all, the "functional" specification is to be sure to propagate templates between multiple pools, right? (and you are running this in a kind of CRON?)
Yes, for reasons that would pollute this thread, we're running a number of single host pools.
The broader picture is:
We drive https://github.com/ddelnano/packer-plugin-xenserver from a CI server to build a small number of base images / templates
A related CI job then replicates those to all destination pools.
I don't think I'm yet in a position to share full details publicly, but if you're interested I'm sure I could give you more visibility to what we've built so far, and our intended direction of travel.
Clearly, having a central API point might be easier for that. Let me add a link to this topic internally for a next conversation with XO team
marcungeschikts Vates 🪐 Project management ⏱️
@endi can you show us your current script so we can understand all the requirements?
psafont XAPI Vizier 🐪
There's ongoing effort to avoid opening file descriptors, to avoid this situation. I'll consult whether the current patches avoid this situation in template-export / vm-import