CEPH FS Storage Driver


  • XCP-ng Team

    Hi @maxcuttins - what's your XCP-NG version and can you share checksum for your NFSSR.py file?



  • Hi,

    I also play around with xcp-ng and ceph and just tried to use CephFS as a storage repos. I am using XCP-ng 8.0.0 with Ceph Nautilus. Before I tried to use lvm on top of a rbd which basically works but performance is really bad this way.

    So I downloaded your patch and ran it on my test 8.0.0 xenserver, all Hunks succeeded. ceph is also installed, modprobe ceph works. Next I tried to add a new cephFs repository just as you described it above.

    In the end I see an error box pooping up in XencCenter: "SM has thrown a generic python exception". And thats all.

    In /var/log/SMlog I see the log attached below.

    Thanks
    Rainer

    /var/log/SMLog:

    Sep 26 09:42:54 rzinstal4 SM: [18075] lock: opening lock file /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
    Sep 26 09:42:54 rzinstal4 SM: [18075] lock: acquired /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
    Sep 26 09:42:54 rzinstal4 SM: [18075] sr_create {'sr_uuid': 'd63d0d49-522d-6160-a1eb-a9d5f34cec1e', 'subtask_of': 'DummyRef:|55cbc4ef-c6e9-4dba-aa5f-8f56b2473443|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:0c03f516-7ed5-48ac-895b-51833177aed5', 'session_ref': 'OpaqueRef:d472846e-c6aa-467d-ac8b-0f401fbc698c', 'device_config': {'server': 'ip_of_mon', 'SRmaster': 'true', 'serverpath': '/sds', 'options': 'name=admin,secretfile=/etc/ceph/admin.secret'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:9fb35d75-a45c-4ce4-b9f4-8ed8072ad6ad'}
    Sep 26 09:42:54 rzinstal4 SM: [18075] ['/usr/sbin/rpcinfo', '-p', 'ip_of_mon']
    Sep 26 09:42:54 rzinstal4 SM: [18075] FAILED in util.pread: (rc 1) stdout: '', stderr: 'rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused
    Sep 26 09:42:54 rzinstal4 SM: [18075] '
    Sep 26 09:42:54 rzinstal4 SM: [18075] Unable to obtain list of valid nfs versions
    Sep 26 09:42:54 rzinstal4 SM: [18075] lock: released /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
    Sep 26 09:42:54 rzinstal4 SM: [18075] ***** generic exception: sr_create: EXCEPTION <type 'exceptions.TypeError'>, not all arguments converted during string formatting
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
    Sep 26 09:42:54 rzinstal4 SM: [18075]     return self._run_locked(sr)
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
    Sep 26 09:42:54 rzinstal4 SM: [18075]     rv = self._run(sr, target)
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
    Sep 26 09:42:54 rzinstal4 SM: [18075]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/NFSSR", line 216, in create
    Sep 26 09:42:54 rzinstal4 SM: [18075]     raise exn
    Sep 26 09:42:54 rzinstal4 SM: [18075]
    Sep 26 09:42:54 rzinstal4 SM: [18075] ***** NFS VHD: EXCEPTION <type 'exceptions.TypeError'>, not all arguments converted during string formatting
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 372, in run
    Sep 26 09:42:54 rzinstal4 SM: [18075]     ret = cmd.run(sr)
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
    Sep 26 09:42:54 rzinstal4 SM: [18075]     return self._run_locked(sr)
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
    Sep 26 09:42:54 rzinstal4 SM: [18075]     rv = self._run(sr, target)
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
    Sep 26 09:42:54 rzinstal4 SM: [18075]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
    Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/NFSSR", line 216, in create
    Sep 26 09:42:54 rzinstal4 SM: [18075]     raise exn
    Sep 26 09:42:54 rzinstal4 SM: [18075]
    Sep 26 09:42:54 rzinstal4 SM: [18075] lock: closed /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
    

  • XCP-ng Team

    Hi @Rainer and welcome 🙂

    Please use Markdown syntax for console/code blocks:

    ```
    my test
    ```
    


  • Hello oliverlambert,
    thanks for the hint. I edited my post correcting the formatting.


  • XCP-ng Team

    @Rainer Please share your NFSSR.py to check. Are you able to manually mount the SR using #mount.ceph addr1,addr2,addr3,addr4:remotepath localpath?



  • Hello,

    thanks for your answer. I can mount the the CephFS filesystem on the xenserver this way:

    mount.ceph  1.2.3.4:/base /mnt -o "name=admin,secretfile=/etc/ceph/admin.secret"
    

    where 1.2.3.4 is the IP of the active ceph monitor.

    In xencenters New-SR-dialog I put the "1.2.3.4:/base" part in the "Share Name" input field and the options name=admin,secretfile=/etc/ceph/admin.secret in the "Advanced Options" field .

    Below I attached my patched /opt/xensource/sm/NFSSR.py file:
    [0_1569574207746_NFSSR.py](Uploading 100%)

    NFSSR.py.txt


  • XCP-ng Team

    Just so you know - the original patch was for XCP-NG 7.6. I'm checking it for XCP-NG 8.


  • XCP-ng Team

    Can you share /opt/xensource/sm/nfs.py as well... it was also supposed to be patched.


  • XCP-ng Team

    Updated new patch for XCP-NG 8.


  • XCP-ng Team

    Note that all are hacks are won't survive any upgrade and aren't supported 😉 But it's fine for a proof of concept.


  • XCP-ng Team

    Agree - unless we have enough interest from community to pack it or make it maintainable, it will stay a hack.

    Also mentioning the command to apply patch # patch -d/ -p1 < cephfs-8.patch

    Edit : updated patch strip level



  • Hello and thanks for the new patch. I understand that this is only a kind of hack or better a proof of concept. I do not want to use this for a productive system but only wanted to give it a try with nautilus and see what performance I can expect if this hack hopefully develops to a supported integration of Ceph.

    I hoped that the patch might work for XCP-ng 8.0 as well but it seems it does not that easily. So I could also install a xenserver 7.6 and use the original patch and test how it works for me. I do not want you to invest too much work in the 8.0 adoption. So if you think testing the patch 8.0 does not really help you in any way, then please do not hesitate and tell me. Else if you are as well interested to test if it works with 8.0 here is what I found out:

    The new patch does not work completely, there is a "patch unexpectedly ends in middle of line" -warning. On a fresh installed system:

    # cd /
    # patch --verbose -p0  < /cephfs-8.patch
    mm...  Looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |--- /opt/xensource/sm/nfs.py   20199-27 16:45:11.918853933 +0530
    |+++ /opt/xensource/sm/nfs.py   20199-27 16:49:26.910379515 +0530
    --------------------------
    patching file /opt/xensource/sm/nfs.py
    Using Plan A...
    Hunk #1 succeeded at 137.
    Hunk #2 succeeded at 159.
    Hmm...  The next patch looks like a unified diff to me...
    The text leading up to this was:
    --------------------------
    |--- /opt/xensource/sm/NFSSR.py 20199-27 16:44:56.437557255 +0530
    |+++ /opt/xensource/sm/NFSSR.py 20199-27 17:07:45.651188670 +0530
    --------------------------
    patching file /opt/xensource/sm/NFSSR.py
    Using Plan A...
    Hunk #1 succeeded at 137.
    Hunk #2 succeeded at 160.
    patch unexpectedly ends in middle of line
    Hunk #3 succeeded at 197 with fuzz 1.
    done
    

    I can do a ceph.mount just like described before but when I instead try to create a new Ceph-FS-SR named "Ceph" in xencenter 8.0.0 I still get the error that SM has thrown a generic python exception.

    The SMlog error log indicates that creating a NFS repos is tried with the given parameters and not a CephFS one:

    ...
    Sep 30 13:03:05 rzinstal4 SM: [975] sr_create {'sr_uuid': '79cc55db-ee9f-9cc0-8d94-8cdb2915363b', 'subtask_of': 'DummyRef:|6456510f-b2dd-4dcd-b6ab-eb5eb40ceb9c|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:72898e54-20b2-4046-9825-94553457fa60', 'session_ref': 'OpaqueRef:8a0819af-8b91-4086-a9cf-04ae37dc8e79', 'device_config': {'server': '1.2.3.4', 'SRmaster': 'true', 'serverpath': '/base', 'options': 'name=admin,secretfile=/etc/ceph/admin.secret'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:91c592f6-47a0-40d6-8a33-7fe01831145f'}
    Sep 30 13:03:05 rzinstal4 SM: [975] ['/usr/sbin/rpcinfo', '-p', '1.2.3.4']
    Sep 30 13:03:05 rzinstal4 SM: [975] FAILED in util.pread: (rc 1) stdout: '', stderr: 'rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused
    Sep 30 13:03:05 rzinstal4 SM: [975] '
    Sep 30 13:03:05 rzinstal4 SM: [975] Unable to obtain list of valid nfs versions
    ...
    

    I attached my /opt/xensource/sm/nfs.py file as well.

    Thanks a lot

    Rainer
    nfs.py.txt


  • XCP-ng Team

    @r1 said in CEPH FS Storage Driver:

    patch -d/ -p0 < cephfs-8.patch

    Please try # patch -d/ -p0 < cephfs-8.patch


  • XCP-ng Team

    I see that there is a call to check_server_tcp which is expected to return True but not doing so in my patch. I'll modify it.


  • XCP-ng Team

    The same patch is updated. Kindly download again and try with # patch -d/ -p1 < cephfs-8.patch.



  • Thanks for the new patch. Applying the patch does not report any errors or warnings. When I try to create a "NFS" SR I still get the generic python error message from xencenter however the error message in /var/log/SMlog has changed:

    Oct  1 10:17:52 xenserver8 SM: [4975] lock: opening lock file /var/lock/sm/76ad6fee-a7a8-34e6-bcce-a53f69a677a9/sr
    Oct  1 10:17:52 xenserver8 SM: [4975] lock: acquired /var/lock/sm/76ad6fee-a7a8-34e6-bcce-a53f69a677a9/sr
    Oct  1 10:17:52 xenserver8 SM: [4975] sr_create {'sr_uuid': '76ad6fee-a7a8-34e6-bcce-a53f69a677a9', 'subtask_of': 'DummyRef:|c98cee36-24ff-4575-b7
    43-3a84cd93ed17|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:72898e54-20b2-4046-9825-94553457fa60', 'session_ref': 'OpaqueRef:175293cb-a791-
    44be-bd5f-a4cb911adddb', 'device_config': {'server': '1.2.3.4', 'SRmaster': 'true', 'serverpath': '/base', 'options': 'name=admin,secretfil
    e=/etc/ceph/admin.secret'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:a36f0371-1dc7-4685-a208-24e04a1c1e08'}
    Oct  1 10:17:52 xenserver8 SM: [4975] lock: released /var/lock/sm/76ad6fee-a7a8-34e6-bcce-a53f69a677a9/sr
    Oct  1 10:17:52 xenserver8 SM: [4975] ***** generic exception: sr_create: EXCEPTION <type 'exceptions.NameError'>, global name 'useroptions' is no
    t defined
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
    Oct  1 10:17:52 xenserver8 SM: [4975]     return self._run_locked(sr)
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
    Oct  1 10:17:52 xenserver8 SM: [4975]     rv = self._run(sr, target)
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
    Oct  1 10:17:52 xenserver8 SM: [4975]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/NFSSR", line 216, in create
    Oct  1 10:17:52 xenserver8 SM: [4975]     raise exn
    Oct  1 10:17:52 xenserver8 SM: [4975]
    Oct  1 10:17:52 xenserver8 SM: [4975] ***** NFS VHD: EXCEPTION <type 'exceptions.NameError'>, global name 'useroptions' is not defined
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/SRCommand.py", line 372, in run
    Oct  1 10:17:52 xenserver8 SM: [4975]     return self._run_locked(sr)
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
    Oct  1 10:17:52 xenserver8 SM: [4975]     rv = self._run(sr, target)
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
    Oct  1 10:17:52 xenserver8 SM: [4975]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
    Oct  1 10:17:52 xenserver8 SM: [4975]   File "/opt/xensource/sm/NFSSR", line 216, in create
    Oct  1 10:17:52 xenserver8 SM: [4975]     raise exn
    Oct  1 10:17:52 xenserver8 SM: [4975]
    

  • XCP-ng Team

    Looks like variable declaration issue. I had it working for XCP-NG 7.6... Give me some time to fix this and make it working for 8.0



  • @r1 Is this still in development; would like to get it working with 8.1?

    Manually mounting the CephFS works, but the patch fails, probably not surprisingly; or is a GlusterFS approach a better option?

    With Ceph Octopus and the cephadm approach setting Ceph up is really easy, so maybe this will become a more popular feature....?


  • XCP-ng Team

    @jmccoy555 I did not continue this. Will definitely rejuvenate it once 8.1 is released. Hopefully next week.


  • XCP-ng Team

    @jmccoy555

    Notice: This is experimental and will be replaced by any updates to sm package

    Ok, so here is updated patch for XCP-NG 8.1

    # cd /
    # wget "https://gist.githubusercontent.com/rushikeshjadhav/ea8a6e15c3b5e7f6e61fe0cb873173d2/raw/85acc83ec95cba5db7674572b79c403106b48a48/ceph-8.1.patch"
    # patch -p0 < ceph-8.1.patch
    

    Apart from patch you would need to install ceph-common to obtain mount.ceph

    # yum install centos-release-ceph-nautilus --enablerepo=extras
    # yum install ceph-common
    

    There is no need of ceph.ko as now CephFS is part of kernel in 8.1.

    You can refer NFS SR (Ceph) creation screenshot in earlier post or use following
    # xe sr-create type=nfs device-config:server=10.10.10.10 device-config:serverpath=/ device-config:options=name=admin,secretfile=/etc/ceph/admin.secret name-label=CephFS

    To remove this and restore to normal state
    # yum reinstall sm

    Note: Keep secret in /etc/ceph/admin.secret with permission 600

    Edit: Hope to get this into a proper CephSR someday. Until then it must be considered for Proof of Concept only.
    Changed ceph release repo to nautilus.


Log in to reply
 

XCP-ng Pro Support

XCP-ng Pro Support