Native Ceph RBD SM driver for XCP-ng
-
Hello,
I am using CEPH for a very long time, I started back in the times of old Citrix Xen Server (I think version 6 maybe?)
I was using RBD for a long time in a hacky way via LVM with a shared attribute, which had major issues in latest xcp-ng, then I migrated to CephFS (which is relatively stable, but has its own issues related to the nature of CephFS - reduced performance, dependency on the MDS etc.).
I finally decided to move outside of my comfort zone and try and write my own SM driver for actual RBD and after some testing move it to my production cluster. I know there is already another github project wrote by another guy, that is unmaintained for many years and has various versions. I didn't want to bother trying to understand how that one is implemented - I already know how to use RBD with CEPH, I just needed to put it into a SM driver. So I made my own.
I will see how it goes, there are drawbacks already to that "native" RBD over CephFS - while IO performance is superior, the "meta" performance (creating disks, snapshots, deleting disks, rescanning SR) is actually slower because it relies on native CEPH APIs and doesn't just use very fast low-level "file access" of CephFS. But I still think it could be a valuable addition to people who need raw IO performance.
My driver is fully open source and available here - I currently target XCP-ng 8.2 and SMAPIv2, because that's what I use on my production cluster which I am primarily making this for. But eventually I will try to test this with XCP-ng 8.3 and when SMAPIv3 is finally ready, I might port it there as well.
Here is the code: https://github.com/benapetr/CephRBDSR/blob/master/CephRBDSR.py
There is also an installation script that makes the installation of the SM driver pretty simple, may need a tweak as I am not sure if manually adding SM modules to /etc/xapi.conf is really a good idea
Please note it's a work in progress, it's unfinished, and some features probably don't work yet.
What is already tested and works:
- SR creation, unplug, plug, rescan, stat
- Basic RBD / VDI manipulation - creating VDI, deleting VDI, openning VDI / mapping VDI, copying VDI
It's really just managing RBD block devices for you and uses aio to map VDIs to them
What is not tested yet
- Snapshots
- Cluster behaviour
I only recommend for use on dev xcp-ng environments at this moment. I think within a month I might have a fully tested version 1.0
Feedback and suggestions welcome!
-
Hi,
Thanks for the contribution. What are you calling SMAPIv2 exactly? FYI, XCP-ng 8.2 is EOL this month, so it's better to build everything for 8.3
edit: adding @Team-Storage in the loop
-
@olivierlambert by SMAPIv2 I mean the predecessor of SMAPIv3 (eg. the old school SM driver, same as the CephFSSR driver) or maybe that's called SMAPIv1? IDK I am genuinely lost in these conventions and have a hard time finding any relevant docs
-
I took a very quick look (so I might be wrong) but it seems to be a SMAPIv1 driver.
You might be better to start working directly on a SMAPIv3 driver. I would also advise to start here: https://xapi-project.github.io/xapi-storage/#introduction
You can also take a look at our first draft SMAPIv3 plugin implementation: https://github.com/xcp-ng/xcp-ng-xapi-storage/tree/master/plugins/volume/org.xen.xapi.storage.zfs-vol
-
@olivierlambert that documentation is really interesting, those diagrams are full of examples of accessing Ceph and RBD via librados, which is exactly what I am doing here LOL
Did you design those diagram based on some existing driver? It seems someone in your team already had to study Ceph concepts if it's in your very example documentation. Does the RBD driver already exists somewhere in your lab?
-
