XCP-ng

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups

    xe command "hangs on start" when max / open files is high

    Compute
    4
    11
    205
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      endi last edited by

      I don't propose this needs "fixing", but it took some time to triage so sharing:

      We are using xe from xapi-xe_1.249.19-2.3_amd64.deb outside of the xen server host. We found in some situations: any execution that invoked communication with a xen server, would pause / hang for ~10 mins:

      via strace, we found that running xe causes an enumeration over

      ulimit -a
      <snip>
      open files                          (-n) 1073741816
      

      on xcp-ng / xen host this appears to still be 1024. To remove this pause / hang, a possibility is to:

      ulimit -n <your-choice>
      

      where <your-choice> is a value you deem sensible for your system.

      Hope that saves someone some time 😉
      ewan

      1 Reply Last reply Reply Quote 0
      • olivierlambert
        olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by olivierlambert

        Hi!

        Thanks for the feedback 🙂 That's another good reason to move forward on our REST API in XO 🙂

        E 1 Reply Last reply Reply Quote 0
        • E
          endi @olivierlambert last edited by

          @olivierlambert

          Looking forward to it: fyi, we are currently using this to facilitate the transfer of templates between clusters. We didn't find it possible to achieve that via xo-cli, though we're open to the possibility we failed to understand something 🙂

          1 Reply Last reply Reply Quote 0
          • olivierlambert
            olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

            Can you show us how do you achieve that with xe so we can check how/if it's possible to do it via XO?

            E 1 Reply Last reply Reply Quote 0
            • E
              endi @olivierlambert last edited by

              hi @olivierlambert

              we have more than one "destination" server, but simplified here for only one:

              check if destination server has a template with name "x" present
              xe -s <server> -u <user> -pw <pw> template-list name-label=x

              if present, do nothing.

              if absent on this destination, check if we have that template locally
              # this is just a filesystem check where we are running

              if absent locally, pull it from our source server
              # not captured here: lookup template uuid for 'x'
              xe -s <server> -u <user> -pw <pw> template-export template-uuid='uuid' filename='templatename'

              push to destination
              xe -s <server> -u <user> -pw <pw> vm-import filename='templatename'

              Where we have multiple targets, loop over remaining destinations, and push when not detected as being present.

              So tldr:

              template-list
              template-export
              vm-import
              

              Are what I think we need. I wasn't the most recent to look at using xo-cli for this, but we either didn't find them all, or somehow didn't understand how to use what we need

              cheers
              ewan

              1 Reply Last reply Reply Quote 0
              • olivierlambert
                olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

                So all in all, the "functional" specification is to be sure to propagate templates between multiple pools, right? (and you are running this in a kind of CRON?)

                E 1 Reply Last reply Reply Quote 0
                • E
                  endi @olivierlambert last edited by

                  @olivierlambert

                  Yes, for reasons that would pollute this thread, we're running a number of single host pools.

                  The broader picture is:

                  We drive https://github.com/ddelnano/packer-plugin-xenserver from a CI server to build a small number of base images / templates

                  A related CI job then replicates those to all destination pools.

                  I don't think I'm yet in a position to share full details publicly, but if you're interested I'm sure I could give you more visibility to what we've built so far, and our intended direction of travel.

                  regards
                  ewan

                  1 Reply Last reply Reply Quote 0
                  • olivierlambert
                    olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

                    Clearly, having a central API point might be easier for that. Let me add a link to this topic internally for a next conversation with XO team 🙂

                    marcungeschikts 1 Reply Last reply Reply Quote 0
                    • marcungeschikts
                      marcungeschikts Vates 🪐 Project management ⏱️ @olivierlambert last edited by

                      👌

                      1 Reply Last reply Reply Quote 1
                      • olivierlambert
                        olivierlambert Vates 🪐 Co-Founder🦸 CEO 🧑‍💼 last edited by

                        @endi can you show us your current script so we can understand all the requirements?

                        psafont 1 Reply Last reply Reply Quote 0
                        • psafont
                          psafont XAPI Vizier 🐪 @olivierlambert last edited by

                          There's ongoing effort to avoid opening file descriptors, to avoid this situation. I'll consult whether the current patches avoid this situation in template-export / vm-import

                          https://github.com/xapi-project/xen-api/pull/4877

                          snwoods created this issue in xapi-project/xen-api

                          open CP-32622: avoid using select and instead use epoll #4877

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post