xe command "hangs on start" when max / open files is high

endi

I don't propose this needs "fixing", but it took some time to triage so sharing:

We are using xe from xapi-xe_1.249.19-2.3_amd64.deb outside of the xen server host. We found in some situations: any execution that invoked communication with a xen server, would pause / hang for ~10 mins:

via strace, we found that running xe causes an enumeration over

ulimit -a
<snip>
open files                          (-n) 1073741816

on xcp-ng / xen host this appears to still be 1024. To remove this pause / hang, a possibility is to:

ulimit -n <your-choice>

where <your-choice> is a value you deem sensible for your system.

Hope that saves someone some time
ewan

olivierlambert

Hi!

Thanks for the feedback That's another good reason to move forward on our REST API in XO

endi

@olivierlambert

Looking forward to it: fyi, we are currently using this to facilitate the transfer of templates between clusters. We didn't find it possible to achieve that via xo-cli, though we're open to the possibility we failed to understand something

olivierlambert

Can you show us how do you achieve that with xe so we can check how/if it's possible to do it via XO?

endi

hi @olivierlambert

we have more than one "destination" server, but simplified here for only one:

check if destination server has a template with name "x" present
xe -s <server> -u <user> -pw <pw> template-list name-label=x

if present, do nothing.

if absent on this destination, check if we have that template locally
# this is just a filesystem check where we are running

if absent locally, pull it from our source server
# not captured here: lookup template uuid for 'x'
xe -s <server> -u <user> -pw <pw> template-export template-uuid='uuid' filename='templatename'

push to destination
xe -s <server> -u <user> -pw <pw> vm-import filename='templatename'

Where we have multiple targets, loop over remaining destinations, and push when not detected as being present.

So tldr:

template-list
template-export
vm-import

Are what I think we need. I wasn't the most recent to look at using xo-cli for this, but we either didn't find them all, or somehow didn't understand how to use what we need

cheers
ewan

olivierlambert

So all in all, the "functional" specification is to be sure to propagate templates between multiple pools, right? (and you are running this in a kind of CRON?)

endi

@olivierlambert

Yes, for reasons that would pollute this thread, we're running a number of single host pools.

The broader picture is:

We drive https://github.com/ddelnano/packer-plugin-xenserver from a CI server to build a small number of base images / templates

A related CI job then replicates those to all destination pools.

I don't think I'm yet in a position to share full details publicly, but if you're interested I'm sure I could give you more visibility to what we've built so far, and our intended direction of travel.

regards
ewan

olivierlambert

Clearly, having a central API point might be easier for that. Let me add a link to this topic internally for a next conversation with XO team

marcungeschikts

olivierlambert

@endi can you show us your current script so we can understand all the requirements?

psafont

There's ongoing effort to avoid opening file descriptors, to avoid this situation. I'll consult whether the current patches avoid this situation in template-export / vm-import

https://github.com/xapi-project/xen-api/pull/4877

snwoods created this issue in xapi-project/xen-api

xe command "hangs on start" when max / open files is high

open CP-32622: avoid using select and instead use epoll #4877