XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    CEPH FS Storage Driver

    Scheduled Pinned Locked Moved Development
    86 Posts 10 Posters 37.9k Views 8 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      scboley @r1
      last edited by

      @r1 I'm in planning phase right now since these boxes are actually production and I'll have to figure how to do this without it being a possible danger to my production enviro. The box I have that needs it the worst I still haven't upgraded from xenserver 6.5. I need to setup a fully dev box for this. I've got plans for upgrading my infrastructure for the ceph and virtual public network that this would help tremendously.

      1 Reply Last reply Reply Quote 0
      • R Offline
        r1 XCP-ng Team
        last edited by

        Sadly XS 6.5 uses 3.10 kernel which does not support Ceph outright and you would most likely to update this host to a recent XCP version.

        S 1 Reply Last reply Reply Quote 0
        • S Offline
          scboley @r1
          last edited by

          @r1 Yeah I just haven't had time yet since it has one of our heaviest used virtuals on it and not good to upset the masses lol.

          borzelB 1 Reply Last reply Reply Quote 0
          • borzelB Offline
            borzel XCP-ng Center Team @scboley
            last edited by borzel

            @scboley said in CEPH FS Storage Driver:

            and not good to upset the masses

            this is the reason I worked at night the last two weeks at work 🌃

            1 Reply Last reply Reply Quote 0
            • S Offline
              scboley @r1
              last edited by scboley

              @r1 sudo mount -t ceph sanadmin.nams.net:6789: / /mnt/nfsmigrate

              1 Reply Last reply Reply Quote 0
              • R Offline
                r1 XCP-ng Team
                last edited by

                @scboley nice. It should work straight as mentioned in 2nd post of this thread.

                1 Reply Last reply Reply Quote 0
                • M Offline
                  maxcuttins
                  last edited by maxcuttins

                  Ok,

                  i start my adventure but I get soon an issue.

                  This command install the nautilus release Repo:

                  yum install centos-release-ceph-nautilus --enablerepo=extras

                  and worked fine.

                  This one should installed ceph-common but cannot find many dependencies

                  yum install ceph-common
Loaded plugins: fastestmirror
centos-ceph-nautilus                                                                                                                                | 2.9 kB  00:00:00
centos-ceph-nautilus/7/x86_64/primary_db                                                                                                            |  62 kB  00:00:00
Loading mirror speeds from cached hostfile
 * centos-ceph-nautilus: mirrors.prometeus.net
Resolving Dependencies
--> Running transaction check
---> Package ceph-common.x86_64 2:14.2.0-1.el7 will be installed
--> Processing Dependency: python-rgw = 2:14.2.0-1.el7 for package: 2:ceph-common-14.2.0-1.el7.x86_64

... [CUT] ...

Error: Package: 2:librgw2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1()(64bit)
Error: Package: leveldb-1.12.0-5.el7.1.x86_64 (centos-ceph-nautilus)
           Requires: libsnappy.so.1()(64bit)
Error: Package: 2:libcephfs2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librdmacm.so.1()(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: liblz4.so.1()(64bit)
Error: Package: 2:libradosstriper1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1()(64bit)
Error: Package: 2:librbd1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1()(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: python-requests
Error: Package: 2:librgw2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librdmacm.so.1()(64bit)
Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librdmacm.so.1()(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1()(64bit)
Error: Package: 2:libcephfs2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1()(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libfuse.so.2()(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libsnappy.so.1()(64bit)
Error: Package: 2:librbd1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librdmacm.so.1()(64bit)
Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librdmacm.so.1(RDMACM_1.0)(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librabbitmq.so.4()(64bit)
Error: Package: 2:libradosstriper1-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librdmacm.so.1()(64bit)
Error: Package: 2:librgw2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librabbitmq.so.4()(64bit)
Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1()(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libtcmalloc.so.4()(64bit)
Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1(IBVERBS_1.0)(64bit)
Error: Package: 2:librados2-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: libibverbs.so.1(IBVERBS_1.1)(64bit)
Error: Package: 2:ceph-common-14.2.0-1.el7.x86_64 (centos-ceph-nautilus)
           Requires: librdmacm.so.1()(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

                  Of course I'm using a higher version of Ceph than your post.
                  Any hint anyway?

                  M 1 Reply Last reply Reply Quote 0
                  • M Offline
                    maxcuttins @maxcuttins
                    last edited by

                    Ok, solved.
                    Of course Ceph need Epel and Base repo enabled.

                    This worked:

                    yum install epel-release -y --enablerepo=extras
yum install ceph-common --enablerepo='centos-ceph-nautilus' --enablerepo='epel' --enablerepo='base'
                    1 Reply Last reply Reply Quote 0
                    • M Offline
                      maxcuttins
                      last edited by

                      @r1 said in CEPH FS Storage Driver:

                      patch -p0 < ceph.patch

                      This went not so smooth, reporting "unexpected end":

                      patch -p0 < ceph.patch
patching file /opt/xensource/sm/nfs.py
patching file /opt/xensource/sm/NFSSR.py
patch unexpectedly ends in middle of line
Hunk #3 succeeded at 197 with fuzz 1.
                      1 Reply Last reply Reply Quote 0
                      • R Offline
                        r1 XCP-ng Team
                        last edited by

                        Hi @maxcuttins - what's your XCP-NG version and can you share checksum for your NFSSR.py file?

                        1 Reply Last reply Reply Quote 0
                        • R Offline
                          Rainer
                          last edited by Rainer

                          Hi,

                          I also play around with xcp-ng and ceph and just tried to use CephFS as a storage repos. I am using XCP-ng 8.0.0 with Ceph Nautilus. Before I tried to use lvm on top of a rbd which basically works but performance is really bad this way.

                          So I downloaded your patch and ran it on my test 8.0.0 xenserver, all Hunks succeeded. ceph is also installed, modprobe ceph works. Next I tried to add a new cephFs repository just as you described it above.

                          In the end I see an error box pooping up in XencCenter: "SM has thrown a generic python exception". And thats all.

                          In /var/log/SMlog I see the log attached below.

                          Thanks
                          Rainer

                          /var/log/SMLog:

                          Sep 26 09:42:54 rzinstal4 SM: [18075] lock: opening lock file /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
Sep 26 09:42:54 rzinstal4 SM: [18075] lock: acquired /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
Sep 26 09:42:54 rzinstal4 SM: [18075] sr_create {'sr_uuid': 'd63d0d49-522d-6160-a1eb-a9d5f34cec1e', 'subtask_of': 'DummyRef:|55cbc4ef-c6e9-4dba-aa5f-8f56b2473443|SR.create', 'args': ['0'], 'host_ref': 'OpaqueRef:0c03f516-7ed5-48ac-895b-51833177aed5', 'session_ref': 'OpaqueRef:d472846e-c6aa-467d-ac8b-0f401fbc698c', 'device_config': {'server': 'ip_of_mon', 'SRmaster': 'true', 'serverpath': '/sds', 'options': 'name=admin,secretfile=/etc/ceph/admin.secret'}, 'command': 'sr_create', 'sr_ref': 'OpaqueRef:9fb35d75-a45c-4ce4-b9f4-8ed8072ad6ad'}
Sep 26 09:42:54 rzinstal4 SM: [18075] ['/usr/sbin/rpcinfo', '-p', 'ip_of_mon']
Sep 26 09:42:54 rzinstal4 SM: [18075] FAILED in util.pread: (rc 1) stdout: '', stderr: 'rpcinfo: can't contact portmapper: RPC: Remote system error - Connection refused
Sep 26 09:42:54 rzinstal4 SM: [18075] '
Sep 26 09:42:54 rzinstal4 SM: [18075] Unable to obtain list of valid nfs versions
Sep 26 09:42:54 rzinstal4 SM: [18075] lock: released /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
Sep 26 09:42:54 rzinstal4 SM: [18075] ***** generic exception: sr_create: EXCEPTION <type 'exceptions.TypeError'>, not all arguments converted during string formatting
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Sep 26 09:42:54 rzinstal4 SM: [18075]     return self._run_locked(sr)
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Sep 26 09:42:54 rzinstal4 SM: [18075]     rv = self._run(sr, target)
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
Sep 26 09:42:54 rzinstal4 SM: [18075]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/NFSSR", line 216, in create
Sep 26 09:42:54 rzinstal4 SM: [18075]     raise exn
Sep 26 09:42:54 rzinstal4 SM: [18075]
Sep 26 09:42:54 rzinstal4 SM: [18075] ***** NFS VHD: EXCEPTION <type 'exceptions.TypeError'>, not all arguments converted during string formatting
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 372, in run
Sep 26 09:42:54 rzinstal4 SM: [18075]     ret = cmd.run(sr)
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Sep 26 09:42:54 rzinstal4 SM: [18075]     return self._run_locked(sr)
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Sep 26 09:42:54 rzinstal4 SM: [18075]     rv = self._run(sr, target)
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/SRCommand.py", line 323, in _run
Sep 26 09:42:54 rzinstal4 SM: [18075]     return sr.create(self.params['sr_uuid'], long(self.params['args'][0]))
Sep 26 09:42:54 rzinstal4 SM: [18075]   File "/opt/xensource/sm/NFSSR", line 216, in create
Sep 26 09:42:54 rzinstal4 SM: [18075]     raise exn
Sep 26 09:42:54 rzinstal4 SM: [18075]
Sep 26 09:42:54 rzinstal4 SM: [18075] lock: closed /var/lock/sm/d63d0d49-522d-6160-a1eb-a9d5f34cec1e/sr
                          R 1 Reply Last reply Reply Quote 0
                          • olivierlambertO Online
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Hi @Rainer and welcome 🙂

                            Please use Markdown syntax for console/code blocks:

                            ```
my test
```
                            R 1 Reply Last reply Reply Quote 0
                            • R Offline
                              Rainer @olivierlambert
                              last edited by

                              Hello oliverlambert,
                              thanks for the hint. I edited my post correcting the formatting.

                              1 Reply Last reply Reply Quote 1
                              • R Offline
                                r1 XCP-ng Team @Rainer
                                last edited by

                                @Rainer Please share your NFSSR.py to check. Are you able to manually mount the SR using #mount.ceph addr1,addr2,addr3,addr4:remotepath localpath?

                                1 Reply Last reply Reply Quote 0
                                • R Offline
                                  Rainer
                                  last edited by

                                  Hello,

                                  thanks for your answer. I can mount the the CephFS filesystem on the xenserver this way:

                                  mount.ceph  1.2.3.4:/base /mnt -o "name=admin,secretfile=/etc/ceph/admin.secret"

                                  where 1.2.3.4 is the IP of the active ceph monitor.

                                  In xencenters New-SR-dialog I put the "1.2.3.4:/base" part in the "Share Name" input field and the options name=admin,secretfile=/etc/ceph/admin.secret in the "Advanced Options" field .

                                  Below I attached my patched /opt/xensource/sm/NFSSR.py file:
                                  [0_1569574207746_NFSSR.py](Uploading 100%)

                                  NFSSR.py.txt

                                  1 Reply Last reply Reply Quote 0
                                  • R Offline
                                    r1 XCP-ng Team
                                    last edited by

                                    Just so you know - the original patch was for XCP-NG 7.6. I'm checking it for XCP-NG 8.

                                    1 Reply Last reply Reply Quote 0
                                    • R Offline
                                      r1 XCP-ng Team
                                      last edited by

                                      Can you share /opt/xensource/sm/nfs.py as well... it was also supposed to be patched.

                                      1 Reply Last reply Reply Quote 0
                                      • R Offline
                                        r1 XCP-ng Team
                                        last edited by

                                        Updated new patch for XCP-NG 8.

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO Online
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          Note that all are hacks are won't survive any upgrade and aren't supported 😉 But it's fine for a proof of concept.

                                          1 Reply Last reply Reply Quote 0
                                          • R Offline
                                            r1 XCP-ng Team
                                            last edited by r1

                                            Agree - unless we have enough interest from community to pack it or make it maintainable, it will stay a hack.

                                            Also mentioning the command to apply patch # patch -d/ -p1 < cephfs-8.patch

                                            Edit : updated patch strip level

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post