<![CDATA[CEPH FS Storage Driver]]>As a side experiment - I could extract and test CEPH FS kernel module for XCP-NG 7.4+ and mount a multi-mon ceph cluster (luminous).

Once ceph.ko module is in place, XCP-NG can mount the ceph fs mount point similar to NFS.

e.g. In NFSSR we use #mount.nfs4 addr:remotepath localpath

while for CEPHSR we can use #mount.ceph addr1,addr2,addr3,addr4:remotepath localpath

I'm currently looking at NFSSR.py to create CEPHFSSR.py - and would share once ready, meanwhile if anyone wants to help me in this by testing ceph.ko, developing CEPHFSSR.py, kindly ping here. Like EXT4 and XFS, this too will have some cleanup issues..

If this works as expected, I'd request @olivierlambert and @borzel to look at possibilities of integrating CEPH FS option in XO/XC near NFS SR.

Note: This is completely different than RBDSR which is worked by @rposudnevskiy and uses rbd image protocol of ceph.

Bonus : If someone knows how nfs-ganesha fits in this - buzzz... e.g. Each CEPH FS node can have a NFS server while each XCP-NG host mounts all of them using NFS 4.1 (pNFS). Bypassing need of ceph.ko. We can also rebuild NFS module as CONFIG_NFSD_PNFS is not set by default.

Caveats : CEPH FS caching is not fully explored.

State: Ready. See post below for instructions on how to use.

]]>https://xcp-ng.org/forum/topic/1006/ceph-fs-storage-driverRSS for NodeTue, 10 Mar 2026 00:09:07 GMTSat, 23 Feb 2019 19:47:11 GMT60<![CDATA[Reply to CEPH FS Storage Driver on Wed, 23 Nov 2022 19:35:56 GMT]]>We always try to work with Citrix ("XenServer" division now), to push things upstream and manage to get it merged.

]]>https://xcp-ng.org/forum/post/55349https://xcp-ng.org/forum/post/55349Wed, 23 Nov 2022 19:35:56 GMT<![CDATA[Reply to CEPH FS Storage Driver on Wed, 23 Nov 2022 17:21:56 GMT]]>@olivierlambert so are you completely on your own release schedule now or are you still tied to citrix version releases? I've used you since 7.x versions and have had zero issues and still have one 6.5 I'm going to migrate a vm off of because it was on local storage and it's quite large and I didn't want to redo it because of the storage changes from 6 to 7 and then I'll have a pool with all 8.2 after I patch up my other 8.2 and then rebuild the 6.5 to 8.2.

]]>https://xcp-ng.org/forum/post/55347https://xcp-ng.org/forum/post/55347Wed, 23 Nov 2022 17:21:56 GMT<![CDATA[Reply to CEPH FS Storage Driver on Wed, 23 Nov 2022 17:09:56 GMT]]>Okay good to know. We hope to do an even better native integration on SMAPIv3, but the hardest part isn't on writing the driver itself, but to improve the SMAPIv3 itself to support what's missing (live storage migration and so on).

]]>https://xcp-ng.org/forum/post/55345https://xcp-ng.org/forum/post/55345Wed, 23 Nov 2022 17:09:56 GMT<![CDATA[Reply to CEPH FS Storage Driver on Wed, 23 Nov 2022 17:11:36 GMT]]>@olivierlambert that is right and its up and running using the driver you guys built and taking a good load no issues. Hats off to you and your team for making this work. I used nfs for exporting off my old ceph system but the importing is strictly on your native cephfs drivers. On my old one I was on xcp-ng 7.6 and had nfs sitting over the cephfs and just gigabit network and it worked but no live movement at all and now I can live migrate no problems with bonded dual 10gb fiber and single 10gb fiber to the hosts.

]]>https://xcp-ng.org/forum/post/55344https://xcp-ng.org/forum/post/55344Wed, 23 Nov 2022 17:11:36 GMT<![CDATA[Reply to CEPH FS Storage Driver on Wed, 23 Nov 2022 16:55:59 GMT]]>So just to be sure I get it, you have a dedicated Ceph storage on dedicated hardware and you wanted to connect to it via XCP-ng without using NFS or iSCSI, right?

]]>https://xcp-ng.org/forum/post/55342https://xcp-ng.org/forum/post/55342Wed, 23 Nov 2022 16:55:59 GMT<![CDATA[Reply to CEPH FS Storage Driver on Wed, 23 Nov 2022 16:41:48 GMT]]>@olivierlambert this is holding up quite well. I've pounded it good doing 3 exports and an import simultaneously and maintained about 60mb sec writes while doing reads and writes asynchronous as the imports and exports are on the same cephfs. Had an issue with the ceph repository moving to the pool when I created the pool and wound up hosing my pool manager trying to fix it but setting new pool manager and rebuilding the other and it has been flawless since. I've built the XO from source and it has come a long way from the last time I looked at it. We went with a small vendor out of canada called 45drives for the ceph hardware and deployment and they have a nice ansible derived delivery and config package if anyone is looking for a supported solution that is pretty slick.

]]>https://xcp-ng.org/forum/post/55339https://xcp-ng.org/forum/post/55339Wed, 23 Nov 2022 16:41:48 GMT<![CDATA[Reply to CEPH FS Storage Driver on Mon, 21 Nov 2022 16:25:39 GMT]]>@olivierlambert nevermind I fixed it, I had forgot to add the public side of my ceph network onto the new host and then it all scanned correctly. Thanks for being responsive and a little education as it always helps.

]]>https://xcp-ng.org/forum/post/55260https://xcp-ng.org/forum/post/55260Mon, 21 Nov 2022 16:25:39 GMT<![CDATA[Reply to CEPH FS Storage Driver on Mon, 21 Nov 2022 16:21:06 GMT]]>@olivierlambert

Nov 21 10:20:11 xcp4-1 SM: [30250] vhd=/var/run/sr-mount/51b80ad1-820d-c29a-1f9c-a50d6454f927/.vhd scan-error=-5 error-message='failure scanning target'
Nov 21 10:20:11 xcp4-1 SM: [30250] scan failed: -5
Nov 21 10:20:11 xcp4-1 SM: [30250] ', stderr: ''
Nov 21 10:20:12 xcp4-1 SM: [30250] ['/usr/bin/vhd-util', 'scan', '-f', '-m', '/var/run/sr-mount/51b80ad1-820d-c29a-1f9c-a50d6454f927/.vhd']
Nov 21 10:20:12 xcp4-1 SM: [30250] FAILED in util.pread: (rc 5) stdout: 'vhd=/var/run/sr-mount/51b80ad1-820d-c29a-1f9c-a50d6454f927 scan-error=2 error-message='failure
scanning target'
Nov 21 10:20:12 xcp4-1 SM: [30250] vhd=/var/run/sr-mount/51b80ad1-820d-c29a-1f9c-a50d6454f927/*.vhd scan-error=-5 error-message='failure scanning target'
Nov 21 10:20:12 xcp4-1 SM: [30250] scan failed: -5

]]>https://xcp-ng.org/forum/post/55259https://xcp-ng.org/forum/post/55259Mon, 21 Nov 2022 16:21:06 GMT<![CDATA[Reply to CEPH FS Storage Driver on Mon, 21 Nov 2022 16:04:50 GMT]]>@scboley Storage related errors will be in SMlog

]]>https://xcp-ng.org/forum/post/55257https://xcp-ng.org/forum/post/55257Mon, 21 Nov 2022 16:04:50 GMT<![CDATA[Reply to CEPH FS Storage Driver on Mon, 21 Nov 2022 16:02:34 GMT]]>@olivierlambert adding another host to the pool and it fails to connect to the ceph shared storage:

Nov 21 09:57:48 xcp4-1 xapi: [debug||116026 /var/lib/xcp/xapi|SR.scan R:05af02328263|helpers] Waiting for up to 12.902806 seconds before retrying...
Nov 21 09:57:59 xcp4-1 xapi: [debug||116027 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:79aefd48b34b created by task D:f32e5efdeec8
Nov 21 09:57:59 xcp4-1 xapi: [ info||116027 /var/lib/xcp/xapi|session.logout D:67032978d90c|xapi_session] Session.destroy trackid=c8a5d1fe7e932298b267edb677909a4b
Nov 21 09:57:59 xcp4-1 xapi: [debug||116028 /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.slave_login D:0366d884ee46 created by task D:f32e5efdeec8
Nov 21 09:57:59 xcp4-1 xapi: [ info||116028 /var/lib/xcp/xapi|session.slave_login D:b39585e0b07e|xapi_session] Session.create trackid=fc78c651286146c61742b0ca74212bb9 pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Nov 21 09:57:59 xcp4-1 xapi: [debug||116029 /var/lib/xcp/xapi||dummytaskhelper] task dis

Nov 21 09:59:34 xcp4-1 xapi: [ info||116009 HTTPS 192.168.254.101->|Async.PBD.plug R:631710626e67|xapi_session] Session.destroy trackid=726402fee499e51bb72de7fd054a93d0
Nov 21 09:59:34 xcp4-1 xapi: [debug||116009 HTTPS 192.168.254.101->|Async.PBD.plug R:631710626e67|message_forwarding] Unmarking SR after PBD.plug (task=OpaqueRef:63171062-6e67-4cbd-b3be-91bb534a94bf)
Nov 21 09:59:34 xcp4-1 xapi: [error||116009 ||backtrace] Async.PBD.plug R:631710626e67 failed with exception Server_error(SR_BACKEND_FAILURE_12, [ ; mount failed with return code 32; ])
Nov 21 09:59:34 xcp4-1 xapi: [error||116009 ||backtrace] Raised Server_error(SR_BACKEND_FAILURE_12, [ ; mount failed with return code 32; ])
Nov 21 09:59:34 xcp4-1 xapi: [error||116009 ||backtrace] 1/1 xapi Raised at file (Thread 116009 has no backtrace table. Was with_backtraces called?, line 0
Nov 21 09:59:34 xcp4-1 xapi: [error||116009 ||backtrace]

]]>https://xcp-ng.org/forum/post/55255https://xcp-ng.org/forum/post/55255Mon, 21 Nov 2022 16:02:34 GMT<![CDATA[Reply to CEPH FS Storage Driver on Mon, 14 Nov 2022 19:37:44 GMT]]>@scboley I figured it out finally. I used another key created by the cluster and got it to connect and mount the ceph.

]]>https://xcp-ng.org/forum/post/54951https://xcp-ng.org/forum/post/54951Mon, 14 Nov 2022 19:37:44 GMT<![CDATA[Reply to CEPH FS Storage Driver on Mon, 14 Nov 2022 19:16:09 GMT]]>Ok I've got this setup and I have a cluster serving the cephfs and here's my errors:
xe sr-create type=cephfs name-label=ceph device-config:server=172.30.254.23,172.30.254.24,172.30.254.25 device-config:serverport=6789 device-config:serverpath=/fsgw/xcpsr device-config:options=name=admin,secretfile=/etc/ceph/admin.secret
Error code: SR_BACKEND_FAILURE_111
Error parameters: , CephFS mount error [opterr=mount failed with return code 1],

]]>https://xcp-ng.org/forum/post/54948https://xcp-ng.org/forum/post/54948Mon, 14 Nov 2022 19:16:09 GMT<![CDATA[Reply to CEPH FS Storage Driver on Fri, 11 Nov 2022 11:31:51 GMT]]>We use an officially supported kernel (4.19 in LTS) and yes, sometimes we even backport stuff to it specifically for XCP-ng 🙂

A kernel isn't "linked" to a distro, it's all about the distro maintainers to choose which kernel they want. We do that for XCP-ng and XenServer (with Citrix).

In short: we make our own choices regarding Xen and the kernel, entirely outside CentOS project.

]]>https://xcp-ng.org/forum/post/54886https://xcp-ng.org/forum/post/54886Fri, 11 Nov 2022 11:31:51 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 21:20:03 GMT]]>@olivierlambert I know kernel.org maintains a lot of very new kernels for centos versions way newer than the locked and back ported mess the default kernels are so do you build off that base and change out the virtual parts and put them in your builds?

]]>https://xcp-ng.org/forum/post/54876https://xcp-ng.org/forum/post/54876Thu, 10 Nov 2022 21:20:03 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 20:33:23 GMT]]>We don't use any kernel from CentOS project (nor the Xen package). We only use "the rest".

So in order, it will be:

  • newer Xen version (easiest thing)
  • more recent kernel (some patches are needed at different places)
  • more recent user space/base distro (bigger work, but started already, like migrating all Python 2 stuff to Python 3!)

SMAPIv3 is done in parallel and with XS teams too 🙂

]]>https://xcp-ng.org/forum/post/54874https://xcp-ng.org/forum/post/54874Thu, 10 Nov 2022 20:33:23 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 19:29:00 GMT]]>@olivierlambert so what are your plans for going to a streams 8 version which would give the updated kernel platform and hopefully soon after SMAPIv3? IO throughput on 8 over 7 is vastly superior and not near as big as the 6 to 7 changes were.

]]>https://xcp-ng.org/forum/post/54869https://xcp-ng.org/forum/post/54869Thu, 10 Nov 2022 19:29:00 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 18:33:56 GMT]]>No, not really, see https://xcp-ng.org/blog/2020/12/17/centos-and-xcpng-future/ (so no biggie)

]]>https://xcp-ng.org/forum/post/54867https://xcp-ng.org/forum/post/54867Thu, 10 Nov 2022 18:33:56 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 18:24:31 GMT]]>@olivierlambert Ok I see even with 8.x you are still based on centos 7 when is it going up to 8 and I'd assume rocky would be the choice since the redhat streaming snafu cough cough.

]]>https://xcp-ng.org/forum/post/54866https://xcp-ng.org/forum/post/54866Thu, 10 Nov 2022 18:24:31 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 18:13:36 GMT]]>Very likely when the platform will be more modern (upgrading the kernel, platform and using SMAPIv3)

]]>https://xcp-ng.org/forum/post/54865https://xcp-ng.org/forum/post/54865Thu, 10 Nov 2022 18:13:36 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 18:08:47 GMT]]>@olivierlambert what are the plans to elevate it? I have a feeling its really starting to gain traction in the storage world.

]]>https://xcp-ng.org/forum/post/54864https://xcp-ng.org/forum/post/54864Thu, 10 Nov 2022 18:08:47 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 18:04:54 GMT]]>I don't think we updated anything on that aspect, since Ceph isn't a "main" supported SR 🙂

]]>https://xcp-ng.org/forum/post/54863https://xcp-ng.org/forum/post/54863Thu, 10 Nov 2022 18:04:54 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 17:51:52 GMT]]>Ok I see the package listed in the documentation is still nautilus has that been updated to any newer ceph versions as of yet? @olivierlambert

]]>https://xcp-ng.org/forum/post/54862https://xcp-ng.org/forum/post/54862Thu, 10 Nov 2022 17:51:52 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 17:33:21 GMT]]>@scboley Yes, it's mostly just a matter of doing yum update: https://xcp-ng.org/docs/updates.html

]]>https://xcp-ng.org/forum/post/54859https://xcp-ng.org/forum/post/54859Thu, 10 Nov 2022 17:33:21 GMT<![CDATA[Reply to CEPH FS Storage Driver on Thu, 10 Nov 2022 17:16:00 GMT]]>@jmccoy555 I'm talking about 8.2.1 and 8.2.2 and so forth. Is that a simple yum update on the system? I've just left it default version and never updated I was on 7.6 for a long time and just took it all to 8.2 with one straggler xenserver 6.5 still in production. I've loved the stability I've had with xcp-ng not even messing with it at all.

]]>https://xcp-ng.org/forum/post/54858https://xcp-ng.org/forum/post/54858Thu, 10 Nov 2022 17:16:00 GMT