@Chr57
Storage between the servers is not shared as a large file system like Gluster. Right?
So for example, each 4 hosts has 2TB storage then the max HD space is 2TB
As @stormi mentioned, this depends on your replication factor. It works like this for your example:
Total Space = (No of Hosts x Storage ) / Replication Factor
(Assuming you have same storage on all nodes)
Eg. replication factor is 2, then :
Total Space = (4 x 2)/2 = 4 TB
Note:
What you have to keep in mind though is it also depends on each SSD you have so say if you put 1.5 TB SSD & 0.5 TB SSD, then although you have 2TB on each node, but if you create a VM with 1TB space, you will not be able to create another VM with 1TB since there will not be enough contiguous space. What it means is that XOSTOR will not split the VM disk on two separate drives in case of JBOD.
In case of Raid 0 at bios level, you may be able to get away with this but Raid 0 is not recommended.
Is the NIC speed of the storage network important? Is 2x40G on each server for this overkill?
The question is generic and it actually depends on your workload and SSD speed (Gen4 or Gen5 or if you have an old Gen1). At the outset 2x40G seems to be more than enough for most applications. If you have a an old Gen1 SSD or SATA SSD, then you might not even reach the full bandwidth in case of 2x40GB (practically speaking).
What software raid on the NVME disks is recommended?
For going production with Nvme SSDs, I would not recommend RAID at all ! JBOD would work just fine.(Assuming your are using generic applications)