xe command "hangs on start" when max / open files is high

[endi]

I don't propose this needs "fixing", but it took some time to triage so sharing:

We are using xe from xapi-xe_1.249.19-2.3_amd64.deb outside of the xen server host. We found in some situations: any execution that invoked communication with a xen server, would pause / hang for ~10 mins:

via strace, we found that running xe causes an enumeration over

ulimit -a
<snip>
open files                          (-n) 1073741816

on xcp-ng / xen host this appears to still be 1024. To remove this pause / hang, a possibility is to:

ulimit -n <your-choice>

where <your-choice> is a value you deem sensible for your system.

Hope that saves someone some time
ewan

[olivierlambert]

Hi!

Thanks for the feedback That's another good reason to move forward on our REST API in XO

[endi]

@olivierlambert

Looking forward to it: fyi, we are currently using this to facilitate the transfer of templates between clusters. We didn't find it possible to achieve that via xo-cli, though we're open to the possibility we failed to understand something

[olivierlambert]

Can you show us how do you achieve that with xe so we can check how/if it's possible to do it via XO?

[endi]

hi @olivierlambert

we have more than one "destination" server, but simplified here for only one:

check if destination server has a template with name "x" present
xe -s <server> -u <user> -pw <pw> template-list name-label=x

if present, do nothing.

if absent on this destination, check if we have that template locally
# this is just a filesystem check where we are running

if absent locally, pull it from our source server
# not captured here: lookup template uuid for 'x'
xe -s <server> -u <user> -pw <pw> template-export template-uuid='uuid' filename='templatename'

push to destination
xe -s <server> -u <user> -pw <pw> vm-import filename='templatename'

Where we have multiple targets, loop over remaining destinations, and push when not detected as being present.

So tldr:

template-list
template-export
vm-import

Are what I think we need. I wasn't the most recent to look at using xo-cli for this, but we either didn't find them all, or somehow didn't understand how to use what we need

cheers
ewan

[olivierlambert]

So all in all, the "functional" specification is to be sure to propagate templates between multiple pools, right? (and you are running this in a kind of CRON?)

[endi]

@olivierlambert

Yes, for reasons that would pollute this thread, we're running a number of single host pools.

The broader picture is:

We drive https://github.com/ddelnano/packer-plugin-xenserver from a CI server to build a small number of base images / templates

A related CI job then replicates those to all destination pools.

I don't think I'm yet in a position to share full details publicly, but if you're interested I'm sure I could give you more visibility to what we've built so far, and our intended direction of travel.

regards
ewan

[olivierlambert]

Clearly, having a central API point might be easier for that. Let me add a link to this topic internally for a next conversation with XO team

[marcungeschikts]

[olivierlambert]

@endi can you show us your current script so we can understand all the requirements?

[psafont]

There's ongoing effort to avoid opening file descriptors, to avoid this situation. I'll consult whether the current patches avoid this situation in template-export / vm-import

https://github.com/xapi-project/xen-api/pull/4877

snwoods opened this pull request in xapi-project/xen-api

closed CP-32622: avoid using select and instead use epoll #4877