XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    xe command "hangs on start" when max / open files is high

    Scheduled Pinned Locked Moved Compute
    11 Posts 4 Posters 876 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E Offline
      endi
      last edited by

      I don't propose this needs "fixing", but it took some time to triage so sharing:

      We are using xe from xapi-xe_1.249.19-2.3_amd64.deb outside of the xen server host. We found in some situations: any execution that invoked communication with a xen server, would pause / hang for ~10 mins:

      via strace, we found that running xe causes an enumeration over

      ulimit -a
      <snip>
      open files                          (-n) 1073741816
      

      on xcp-ng / xen host this appears to still be 1024. To remove this pause / hang, a possibility is to:

      ulimit -n <your-choice>
      

      where <your-choice> is a value you deem sensible for your system.

      Hope that saves someone some time 😉
      ewan

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by olivierlambert

        Hi!

        Thanks for the feedback 🙂 That's another good reason to move forward on our REST API in XO 🙂

        E 1 Reply Last reply Reply Quote 0
        • E Offline
          endi @olivierlambert
          last edited by

          @olivierlambert

          Looking forward to it: fyi, we are currently using this to facilitate the transfer of templates between clusters. We didn't find it possible to achieve that via xo-cli, though we're open to the possibility we failed to understand something 🙂

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Can you show us how do you achieve that with xe so we can check how/if it's possible to do it via XO?

            E 1 Reply Last reply Reply Quote 0
            • E Offline
              endi @olivierlambert
              last edited by

              hi @olivierlambert

              we have more than one "destination" server, but simplified here for only one:

              check if destination server has a template with name "x" present
              xe -s <server> -u <user> -pw <pw> template-list name-label=x

              if present, do nothing.

              if absent on this destination, check if we have that template locally
              # this is just a filesystem check where we are running

              if absent locally, pull it from our source server
              # not captured here: lookup template uuid for 'x'
              xe -s <server> -u <user> -pw <pw> template-export template-uuid='uuid' filename='templatename'

              push to destination
              xe -s <server> -u <user> -pw <pw> vm-import filename='templatename'

              Where we have multiple targets, loop over remaining destinations, and push when not detected as being present.

              So tldr:

              template-list
              template-export
              vm-import
              

              Are what I think we need. I wasn't the most recent to look at using xo-cli for this, but we either didn't find them all, or somehow didn't understand how to use what we need

              cheers
              ewan

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                So all in all, the "functional" specification is to be sure to propagate templates between multiple pools, right? (and you are running this in a kind of CRON?)

                E 1 Reply Last reply Reply Quote 0
                • E Offline
                  endi @olivierlambert
                  last edited by

                  @olivierlambert

                  Yes, for reasons that would pollute this thread, we're running a number of single host pools.

                  The broader picture is:

                  We drive https://github.com/ddelnano/packer-plugin-xenserver from a CI server to build a small number of base images / templates

                  A related CI job then replicates those to all destination pools.

                  I don't think I'm yet in a position to share full details publicly, but if you're interested I'm sure I could give you more visibility to what we've built so far, and our intended direction of travel.

                  regards
                  ewan

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    Clearly, having a central API point might be easier for that. Let me add a link to this topic internally for a next conversation with XO team 🙂

                    marcungeschiktsM 1 Reply Last reply Reply Quote 0
                    • marcungeschiktsM Offline
                      marcungeschikts Vates 🪐 Project mgmt @olivierlambert
                      last edited by

                      👌

                      1 Reply Last reply Reply Quote 1
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        @endi can you show us your current script so we can understand all the requirements?

                        psafontP 1 Reply Last reply Reply Quote 0
                        • psafontP Offline
                          psafont @olivierlambert
                          last edited by

                          There's ongoing effort to avoid opening file descriptors, to avoid this situation. I'll consult whether the current patches avoid this situation in template-export / vm-import

                          https://github.com/xapi-project/xen-api/pull/4877

                          snwoods opened this pull request in xapi-project/xen-api

                          closed CP-32622: avoid using select and instead use epoll #4877

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post