Idea/Dev test: HowTo use GlusterFS as SR on XCP-ng

olivierlambert

We got an ext4-ng driver already working. But benchmarks are… not good so far. So now the goal is to investigate to find the bottleneck. Oddly enough, even while using what we thing the same datapath than SMAPIv1 (yes you can do that), it's still slower.

In the end, we must really understand exactly how this whole stuff works, so we could improve it ourselves (there is no public master branch for some repo, thanks Citrix…)

edit: we are using the datapath coded by Citrix, so they should suffer the same problem than us for GFS2. I don't understand how this could be "production ready", but that's another story.

geek-baba

@olivierlambert just checking in, the wiki still says its not production ready...

@borzel how is your implementation performing over last few months...

olivierlambert

Depends on what you expect about production ready. Eg this driver is used since 2y in XOSAN without issues. But we support XOSAN ourselves, not the driver alone.

geek-baba

@olivierlambert any update on this topic, almost over an year now so wondering, is it safe to use now?

Or I stumbled upon ceph implementation using SMAPIv3, I believe this a better option than glusterFS?

https://xcp-ng.org/forum/topic/1151/ceph-qemu-dp-in-xcp-ng-7-6/12

jmccoy555

@geek-baba or a CephFS option!

geek-baba

@jmccoy555 that does not meet the need I have, I have a K8s cluster and everything works fine except some apps require block storage and should be fast, I have a rook-ceph cluster running in the k8s cluster and I was looking to move it out of that so I don't have to worry about it during k8s cluster upgrades/migrations.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-July/002815.html

jmccoy555

@geek-baba Fair enough, just another option. I don't like block storage, always favoured NFS over iSCSI, i think the speed trade off is worth the transparency.

geek-baba

@jmccoy555 I don't like block storage either, all my VMs are running off a NVMe NAS over 10Gig network, all my k8s app use dynamically provisioned NFS PV/PVC, unfortunately some of the apps today are not cloud native and need block storage for their inbuilt database. I looked at CephFS and it looks like another option to NFS though, will test the performance at some point...

r1

@geek-baba Are you looking to expose RBD over iSCSI and using that as SR for your XCP-NG hosts? If you don't need SR, I think your rbd client in your APP VMs should work fine.

May be we are missing the larger picture of your implementation.

olivierlambert

@geek-baba said in Idea/Dev test: HowTo use GlusterFS as SR on XCP-ng:

@olivierlambert any update on this topic, almost over an year now so wondering, is it safe to use now?

It was always "safe". Gluster packages are now available directly in XCP-ng 8.1

geek-baba

@r1 I have multiple debian vm's that are running as slave nodes for my k8s cluster, for most app the config and data is on a NFS share, very few app require a block storage due to the internal database locking requirements. Fundamentally apps can start on any of the k8s node when restarted, so if an app need access to block storage, it should be available to each node. Another requirement is that it needs to be as fast as possible, so a gluster or ceph running over SSD's attached would do the job. Currently I am running ceph-rook within k8s and migrating it is a headache and hence evaluating other options.

geek-baba

@olivierlambert thats great to hear! Is there a guide that I can look into?

olivierlambert

Install Gluster client packages and create a SR based on Gluster type. Do you have any Gluster knowledge or you start from scratch?

geek-baba

@olivierlambert I have been following it and installed it on k8s cluster using heketi, dont know much more than that, my goal is to create replica of each SSD exposed as SR to VM's.

olivierlambert

I'm not sure to get it: do you have an external Gluster cluster setup outside your XCP-ng host? Or do you want to use each host local SSD as a Gluster resource? (hyperconvergence)

geek-baba

@olivierlambert hyperconvergence - thats the goal.

olivierlambert

I don't know your planning about servers (in VMs? in the dom0?).

We are about to integrate Ceph and Gluster drivers soon in the official repo, so it will be even easier to setup.

geek-baba

@olivierlambert in the VM's, and yes I am looking for something that is supported and do not break when upgrading, I have alternatives now that I can use but having a hyperconvergence storage like XOSAN is the best way to handle it.

olivierlambert

After the driver and yum packages will be integrated, an upgrade with yum will preserve everything.

r1

@geek-baba said in Idea/Dev test: HowTo use GlusterFS as SR on XCP-ng:

have multiple debian vm's that are running as slave nodes for my k8s cluster

So you have VMs on your compute hosts running your apps as well as storage? Did you pass-through your SSD disks to these VMs? If yes, then I think it's straightforward to install Gluster or Ceph client utilities on Dom0 and then attach SR created in those VMs.

We will let you know once these are yum installable. BTW, you could also have your Gluster or Ceph cluster out of your compute hosts and use it with those utilities.