XOSTOR hyperconvergence preview

snk33

@ronan-a said in XOSTOR hyperconvergence preview:

@peter_webbird We've already had feedback on CBT and LINSTOR/DRBD, we don't necessarily recommend enabling it. We have a blocking dev card regarding a bug with LVM lvchange command that may fail on CBT volumes used by a XOSTOR SR. We also have other issues related to migration with CBT.

Is the problem still occurring on latest XCP-ng / XOSTOR version ? Not being able to use CBT on XOSTOR is a big issue for backup/replication.

snk33

Any update from someone ? Not having the ability to enable CBT on XOSTOR is really complicated to make replication / DR scenarios.

At least to know if this is work in progress or not. If it's not we'll need to pause our PoC and post-pone production schedule waiting for significant update.

ronan-a

@snk33 In fact we have limitations with CBT and XOSTOR. And this won't be resolved until we get back into SMAPIv3.

Not being able to use CBT on XOSTOR is a big issue for backup/replication.

Using it is more likely to cause problems than improve your situation. What bothers you about not using CBT here?

snk33

@ronan-a we need to have CBT to use 3rd party backup & replication solution such as Veeam.

On the other hand, XOSTOR with VHD disk might be ok using Xen Orchestra backup & replication features but it needs more testing on our end to make sure it works good enough.
One big question is about the snapshot chain. Coming from VMware world where keeping snapshots may impact I/O performances, we're not very comfortable with it.

Also, we need to find solution for low-RPO replication. On our VMware infrastructure we use Zerto DR for this but XOA being more like Veeam using snapshots, low-RPO will be tough to achieve (unless making snapshots doesn't affect performances at all ?).

Maelstrom96

Is there a chance something is broken with XOSTOR when adding a new node? I've added one and tried to ran the addHost command, but whatever I do, the SM logs are filled with

Nov  3 18:45:01 van3-pprod-xen05 SM: [7577][MainThread] Got exception: Unable to find controller uri.... Retry number: 8
Nov  3 18:45:05 van3-pprod-xen05 SM: [4767][MainThread] Got exception: Unable to find controller uri.... Retry number: 16
Nov  3 18:45:23 van3-pprod-xen05 SM: [5313][MainThread] Got exception: Unable to find controller uri.... Retry number: 15
Nov  3 18:45:23 van3-pprod-xen05 SM: [9900][MainThread] Got exception: Unable to find controller uri.... Retry number: 3

Networking seems all good, but the node is not able to be enabled because of that.

Danp

@Maelstrom96 Can you provide more context?

XOA or XO from sources? Commit / version?
Are the XCP-ng hosts fully patched?
Did you follow this procedure when adding the new host?
etc

dthenot

@Maelstrom96 Hello,

There is indeed an identified bug with addHost that we are investigating at the moment.

Maelstrom96

@dthenot Do we have a fix yet?

dthenot

@Maelstrom96 Not yet, but you could revert https://github.com/xcp-ng/sm/pull/121 while adding the host and then go back to the current version. It's in progress on our side.

Maelstrom96

@dthenot

For anyone stuck, here is what I ended up doing:

I've manually hardcoded the controller_uri in /opt/xensource/sm/linstorvolumemanager.py on the new host, restarted the toolstack using xe-toolstack-restart and ran the xe host-call-plugin host-uuid=NEW_HOST_UUID plugin=linstor-manager fn=addHost args:groupName=linstor_group/thin_device command.

def _get_controller_uri() -> str:
    # TODO: Check that an IP address from the current pool is returned.
    #addresses = _get_controller_addresses()
    return "linstor://10.25.254.100"
    #return "linstor://" + addresses[0] if addresses else ""

After that, you should see the addHost command return True. It should be safe to revert the change to the _get_controller_uri() function and restart to make sure everything starts properly.

[10:06 van3-pprod-xen01 ~]# linstor --controllers 10.25.254.100 node l
╭───────────────────────────────────────────────────────────────────╮
┊ Node             ┊ NodeType ┊ Addresses                  ┊ State  ┊
╞═══════════════════════════════════════════════════════════════════╡
┊ van3-pprod-xen01 ┊ COMBINED ┊ 10.25.254.100:3366 (PLAIN) ┊ Online ┊
┊ van3-pprod-xen02 ┊ COMBINED ┊ 10.25.254.101:3366 (PLAIN) ┊ Online ┊
┊ van3-pprod-xen03 ┊ COMBINED ┊ 10.25.254.102:3366 (PLAIN) ┊ Online ┊
┊ van3-pprod-xen04 ┊ COMBINED ┊ 10.25.254.103:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────────────────╯
[10:06 van3-pprod-xen01 ~]#  xe host-call-plugin host-uuid=f9d94f6c-e826-452f-b9b9-638550b209cf plugin=linstor-manager fn=addHost args:groupName=linstor_group/thin_device
True
[10:08 van3-pprod-xen01 ~]# linstor --controllers 10.25.254.100 node l                                            ╭───────────────────────────────────────────────────────────────────╮
┊ Node             ┊ NodeType ┊ Addresses                  ┊ State  ┊
╞═══════════════════════════════════════════════════════════════════╡
┊ van3-pprod-xen01 ┊ COMBINED ┊ 10.25.254.100:3366 (PLAIN) ┊ Online ┊
┊ van3-pprod-xen02 ┊ COMBINED ┊ 10.25.254.101:3366 (PLAIN) ┊ Online ┊
┊ van3-pprod-xen03 ┊ COMBINED ┊ 10.25.254.102:3366 (PLAIN) ┊ Online ┊
┊ van3-pprod-xen04 ┊ COMBINED ┊ 10.25.254.103:3366 (PLAIN) ┊ Online ┊
┊ van3-pprod-xen05 ┊ COMBINED ┊ 10.25.254.104:3366 (PLAIN) ┊ Online ┊
╰───────────────────────────────────────────────────────────────────╯