@ha_tu_su
After using gparted to wiping out all disks, sr-create command works as expected to create XOSTOR.

I am experimenting with xcp-ng as a viable option to replace my company's 3 node clusters (2 main nodes + 1 witness node) at customer sites. The hardware we are using has 2 10Gig NICs and rest are 1Gig.

Since XOSTOR is based on LINSTOR I experimented with implementing LINSTOR HA cluster on 3 nodes on ProxMox first. Although mesh network was an option, I wanted to explore other options of implementing a storage network without a switch, since mesh network is only 'community-supported' at the moment. I ended up doing the following:

1. Only 2 nodes contribute to the storage pool. 3rd node is just there to maintain quorum for hypervisor and for LINSTOR.
2. All 3 nodes are connected to each other via a 1Gig Ethernet switch. This is the management network for the cluster.
3. 2 main nodes are directly connected to each other via a 10Gig link. This is the storage network. Note that the witness node is not connected to storage network.
4. Create a loopback interface on witness node which has an IP in the storage network subnet.
5. Enable ipv_forward=1 on all 3 nodes.
6. Add static routes to 2 main hosts like following:
   ip route add <witness_node_storage_int_ip>/32 dev <main_host_mgmt_int_name>
7. Add static route to witness node like following:
   ip route add <main_node1_storage_int_ip>/32 dev <witness_host_mgmt_int_name>
   ip route add <main_node2_storage_int_ip>/32 dev <witness_host_mgmt_int_name>
8. After this all 3 nodes can talk to each other on storage subnet. LINSTOR traffic to and from witness node will use the management network. Since this traffic would not be much, it will not hamper other traffic on management network.

Now I want to do a similar implementation in xcp-ng and XOSTOR. Proxmox was basically a learning ground to iron out issues and get a grasp on LINSTOR concepts. So now the questions are:

1. Is the above method doable on xcp-ng?
2. Is it advisable to do 3 node storage network without switches this way?
3. Any issues with enterprise support if someone does this?

Thanks.

@Danp
Thanks for the info. I will retest after next update to XOA and let you know. Thanks for all the wonderful work that you guys are doing.

Cheers!!

@olivierlambert
Thanks for the info. No hurry. Better to do it right than fast.