XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XOSTOR getting an SR_BACKEND_FAILURE because it is trying to connect to a random IP

    Scheduled Pinned Locked Moved Unsolved XOSTOR
    5 Posts 4 Posters 56 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      ccooke
      last edited by

      HI,

      I have a test cluster which has exhibited a repeated failure pattern. After a week or two of operation, all actions on the XOSTOR SR fail with messages like this:

      SR_BACKEND_FAILURE_47(, The SR is not available [opterr=Error: Unable to connect to linstor://82.147.85.101: [Errno 111] Connection refused], )

      Note that this IP address is not part of my cluster. Once the error message appears, it will complain about the same IP address until I fix the issue (usually by some fairly drastic cluster reboots and/or recreating the SR). This has now happened three times, with a different apparently-random IP each time.

      Obviously this looks very suspicious. Thankfully there's no serious data on this cluster at the moment, and I have extensive offsite backups if needed. I can keep it running like this for a little while if anyone is able to do some debugging.

      Hopefully this is just a very odd bug, of course, but I'd like to be sure. Especially because it is now recurring.

      Thanks,

      poddingueP ronan-aR 2 Replies Last reply
      Reply Quote 0
      • poddingueP poddingue marked this topic as a question
      • poddingueP
        poddingue Vates 🪐 @ccooke
        last edited by

        Do your hosts have public IP addresses on their management interface? I'm asking because the XOSTOR docs carry a warning about that setup (here) : according to them, the LINSTOR satellites listen on port 3366 without authentication, and an outside party can connect to them and get in the way of the real controller.

        If yours are reachable from outside, I think a different address each time would fit that, but I haven't checked it against a setup like yours. Thanks for offering to keep the cluster in that state for a bit. I'd point @Team-Storage at it before you reboot anything, since they're much better placed than me to say what's worth collecting.

        1 Reply Last reply
        Reply Quote 0
        • olivierlambertO
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Ping @ronan-a we discussed this recently.

          1 Reply Last reply
          Reply Quote 0
          • ronan-aR
            ronan-a Vates 🪐 XCP-ng Team @ccooke
            last edited by

            @ccooke As said by @poddingue , your network is not isolated. You must change that to protect your pool. We are aware of this issue and we will change this behavior in a future release to block LINSTOR controller and satellite access from the outside (probably in December). Also we will use TLS to add another security layer.

            C 1 Reply Last reply
            Reply Quote 0
            • C
              ccooke @ronan-a
              last edited by

              @ronan-a said:

              @ccooke As said by @poddingue , your network is not isolated. You must change that to protect your pool. We are aware of this issue and we will change this behavior in a future release to block LINSTOR controller and satellite access from the outside (probably in December). Also we will use TLS to add another security layer.

              Right. I missed that warning - I was going from https://xcp-ng.org/forum/topic/5361/xostor-hyperconvergence-preview which doesn't look to link to the XOSTOR documentation with the warning.

              Does this need a full reinstall effectively, or can I recover it?

              I see the warning is you can't have hypervisors with any public IPs - that will need a little re-engineering for my small test cluster. I know you can change the IP addresses for linstor after installing xostor - does that resolve the issue, or is there no way to secure it when the hypervisor is on a public IP? Could I not just add some firewall rules for now?

              Thanks for the reply.

              1 Reply Last reply
              Reply Quote 0

              Hello! It looks like you're interested in this conversation, but you don't have an account yet.

              Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

              With your input, this post could be even better 💗

              Register Login
              • First post
                Last post