XOSTOR getting an SR_BACKEND_FAILURE because it is trying to connect to a random IP
-
HI,
I have a test cluster which has exhibited a repeated failure pattern. After a week or two of operation, all actions on the XOSTOR SR fail with messages like this:
SR_BACKEND_FAILURE_47(, The SR is not available [opterr=Error: Unable to connect to linstor://82.147.85.101: [Errno 111] Connection refused], )Note that this IP address is not part of my cluster. Once the error message appears, it will complain about the same IP address until I fix the issue (usually by some fairly drastic cluster reboots and/or recreating the SR). This has now happened three times, with a different apparently-random IP each time.
Obviously this looks very suspicious. Thankfully there's no serious data on this cluster at the moment, and I have extensive offsite backups if needed. I can keep it running like this for a little while if anyone is able to do some debugging.
Hopefully this is just a very odd bug, of course, but I'd like to be sure. Especially because it is now recurring.
Thanks,
-
P poddingue marked this topic as a question
-
Do your hosts have public IP addresses on their management interface? I'm asking because the XOSTOR docs carry a warning about that setup (here) : according to them, the LINSTOR satellites listen on port 3366 without authentication, and an outside party can connect to them and get in the way of the real controller.
If yours are reachable from outside, I think a different address each time would fit that, but I haven't checked it against a setup like yours. Thanks for offering to keep the cluster in that state for a bit. I'd point @Team-Storage at it before you reboot anything, since they're much better placed than me to say what's worth collecting.
-
Ping @ronan-a we discussed this recently.
-
@ccooke As said by @poddingue , your network is not isolated. You must change that to protect your pool. We are aware of this issue and we will change this behavior in a future release to block LINSTOR controller and satellite access from the outside (probably in December). Also we will use TLS to add another security layer.
-
@ccooke As said by @poddingue , your network is not isolated. You must change that to protect your pool. We are aware of this issue and we will change this behavior in a future release to block LINSTOR controller and satellite access from the outside (probably in December). Also we will use TLS to add another security layer.
Right. I missed that warning - I was going from https://xcp-ng.org/forum/topic/5361/xostor-hyperconvergence-preview which doesn't look to link to the XOSTOR documentation with the warning.
Does this need a full reinstall effectively, or can I recover it?
I see the warning is you can't have hypervisors with any public IPs - that will need a little re-engineering for my small test cluster. I know you can change the IP addresses for linstor after installing xostor - does that resolve the issue, or is there no way to secure it when the hypervisor is on a public IP? Could I not just add some firewall rules for now?
Thanks for the reply.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login