CEPH FS Storage Driver



  • [Edited to remove huge quote]

    WOW @r1
    This sound really good.

    CephFS is the one with the worst performance.
    But at least if it works it's better than nothing.
    This also doesn't seems so disruptive but a small patch.
    So I guess, less changes, less issues with the standard installation of XCP.

    good job!
    I'll start to test it at beginning of march.
    Count on me.



  • Thanks @maxcuttins. Let me know how it goes..



  • RBD is faster because of client side caching.. Ceph FS does not do it but can be made faster by enabling fscache. Once you have your numbers we will retest with fscache 🙂



  • Ok I have mine in a very restricted environment and do not have any logins defined for the ceph access.
    So saying that if I don't put any user authentication in the options will it still work? I had issues with your plugin around that same problem.
    I'm also still on 7.5 will that be an issue and my ceph is mimic.



  • Can you show your mount command by doing some example mount? e.g. #mount.ceph add1:remotepath localpath /mnt? Does this work as it is?

    The advanced options are necessary for now to discriminate between NFS and Ceph. But I think once I know your working command, I will be able to generalize it.

    BTW - I hope you installed the ceph.ko as mentioned in earlier steps.



  • @r1 I'm in planning phase right now since these boxes are actually production and I'll have to figure how to do this without it being a possible danger to my production enviro. The box I have that needs it the worst I still haven't upgraded from xenserver 6.5. I need to setup a fully dev box for this. I've got plans for upgrading my infrastructure for the ceph and virtual public network that this would help tremendously.



  • Sadly XS 6.5 uses 3.10 kernel which does not support Ceph outright and you would most likely to update this host to a recent XCP version.



  • @r1 Yeah I just haven't had time yet since it has one of our heaviest used virtuals on it and not good to upset the masses lol.


  • XCP-ng Center Team

    @scboley said in CEPH FS Storage Driver:

    and not good to upset the masses

    this is the reason I worked at night the last two weeks at work 🌃



  • @r1 sudo mount -t ceph sanadmin.nams.net:6789: / /mnt/nfsmigrate



  • @scboley nice. It should work straight as mentioned in 2nd post of this thread.



  • Ok,

    i start my adventure but I get soon an issue.

    This command install the nautilus release Repo:

    yum install centos-release-ceph-nautilus --enablerepo=extras
    

    and worked fine.

    This one should installed ceph-common but cannot find many dependencies

    yum install ceph-common
    Loaded plugins: fastestmirror
    centos-ceph-nautilus                                                                                                                                | 2.9 kB  00:00:00
    centos-ceph-nautilus/7/x86_64/primary_db                                                                                                            |  62 kB  00:00:00
    Loading mirror speeds from cached hostfile
     * centos-ceph-nautilus: mirrors.prometeus.net
    Resolving Dependencies
    --> Running transaction check
    ---> Package ceph-common.x86_64 2:14.2.0-1.el7 will be installed
    --> Processing Dependency: python-rgw = 2:14.2.0-1.el7 for package: 2:ceph-common-14.2.0-1.el7.x86_64
    
    ... [CUT] ...
    
    Error: Package: 2:librgw2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1()(64bit)
    Error: Package: leveldb-1.12.0-5.el7.1.x86_64 (centos-ceph-nautilus)
               Requires: libsnappy.so.1()(64bit)
    Error: Package: 2:libcephfs2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librdmacm.so.1()(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: liblz4.so.1()(64bit)
    Error: Package: 2:libradosstriper1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1()(64bit)
    Error: Package: 2:librbd1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1()(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: python-requests
    Error: Package: 2:librgw2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librdmacm.so.1()(64bit)
    Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librdmacm.so.1()(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1()(64bit)
    Error: Package: 2:libcephfs2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1()(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libfuse.so.2()(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libsnappy.so.1()(64bit)
    Error: Package: 2:librbd1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librdmacm.so.1()(64bit)
    Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librdmacm.so.1(RDMACM_1.0)(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librabbitmq.so.4()(64bit)
    Error: Package: 2:libradosstriper1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librdmacm.so.1()(64bit)
    Error: Package: 2:librgw2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librabbitmq.so.4()(64bit)
    Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1()(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libtcmalloc.so.4()(64bit)
    Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1(IBVERBS_1.0)(64bit)
    Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: libibverbs.so.1(IBVERBS_1.1)(64bit)
    Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
               Requires: librdmacm.so.1()(64bit)
     You could try using --skip-broken to work around the problem
     You could try running: rpm -Va --nofiles --nodigest
    
    

    Of course I'm using a higher version of Ceph than your post.
    Any hint anyway?



  • Ok, solved.
    Of course Ceph need Epel and Base repo enabled.

    This worked:

    yum install epel-release -y --enablerepo=extras
    yum install ceph-common --enablerepo='centos-ceph-nautilus' --enablerepo='epel' --enablerepo='base'
    


  • @r1 said in CEPH FS Storage Driver:

    patch -p0 < ceph.patch

    This went not so smooth, reporting "unexpected end":

    patch -p0 < ceph.patch
    patching file /opt/xensource/sm/nfs.py
    patching file /opt/xensource/sm/NFSSR.py
    patch unexpectedly ends in middle of line
    Hunk #3 succeeded at 197 with fuzz 1.
    


  • Hi @maxcuttins - what's your XCP-NG version and can you share checksum for your NFSSR.py file?


Log in to reply