Native Ceph RBD SM driver for XCP-ng

dicode-nl

For these interested and for further development:

https://github.com/dicode-nl/xcp-ng-ceph-rbd

@olivierlambert any interest in fixing this for SMAPIv3 for inclusion into XCP 9?

benapetr

@dicode-nl hello, very nice, I will definitely check it out!

dicode-nl

@benapetr @olivierlambert I've made a new release which includes SMAPIv1 improvements and a proper SMAPIv3 volume + datapath plugin.

https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases#release-v20260827

Let me know your thoughts and if there is anything you'll like to see added / changed / tested.

Next step for me is CBT and SXM.

dicode-nl

Did a lot of work this weekend and added more to the volume and datapath.
in native mode, everything is rbd native including CBT! still working on SXM
in tapdisk mode, everything is working with optimized rbd calls except SXM
in qemu mode everything works by using qemu-dp and also SXM in MSAPIv3

Huge improvements made, but qemu is the slowest mode due to extra nbt hop.
I only provided tapdisk and qemu modes to get more information on how CBT works and how SXM works. Right now I'm using that knowledge to provide SXM over native rbt. I have some good ideas but need to figure out how to make them work.

I've also made a backport to ceph v15.2 which runs on dom0 to also include the newer cephx authentication protocol. This is because the ceph dashboard cannot provide a diff between snaps (which is needed for CBT) but a pull request is pending to have this working in the future without the need for rbd userspace.

I'll update my git and packages later this week.

dicode-nl

Latest pre-release available:

https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases/tag/v20260901

This release includes a qemu backend mode which fully supports SXM over SMAPIv3! Both tapdisk and native mode still don't have full SXM but can be targets to transfer VDI's to.

Also patches ceph userspace tools are available to provide rbd command line tool. This is needed for CBT as upstream Ceph doesn't have a diff end-point in their API, awaiting pull request https://github.com/ceph/ceph/pull/71388. This is already wired into the RestBackend code so once that lands in Ceph this SR can run without the need for Ceph userspace tools in dom0.

GitHub updated with the latest commits and changes.

WIP: SXM on native rbd without qemu-dp for the native and tapdisk mode.

As always, use with caution. I did run a lot of test scenario's but please do test yourself and let me know your findings!

dicode-nl

New version:

https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases/tag/v20260903

This one includes native Ceph rbd SXM over SMAPIv3!

GitHub updated with the latest commits and changes.

As always, use with caution. I did run a lot of test scenario's but please do test yourself and let me
know your findings!

nick.lloyd

@dicode-nl Just got Ceph 15 working on Dom0 and will be testing this thought experiment. I commend your effort on getting this working! I've been begging for Vates to work on this as LINSTOR is not a "real" solution to the HCI problem with its current limitations (I have clients with 20+ vSAN nodes). I'll keep you updated

dicode-nl

@nick.lloyd if there is anything that isn't working just let me know. I do advise the SMAPIv3 solution at that implementation offers the full feature set!

nick.lloyd

@dicode-nl Just got it all working! Since prefer-crc isn't available in kernels prior to 5.x (can't remember the specific version), I ended up needing to use ms_mode: legacy to get VMs to boot on it. That being said, the only thing I don't see that would be a nice-to-have would be QCOW2 support. The fact that this is actually working is a miracle itself!

Got some initial testing done on my 2TB Micron 1100 SATA SSD pool. Not too bad of results so far. I'll do my bank of 3.2TB NVMe SSDs later, but I want to try and get this more automated/working as an XO(A) plugin first. Thank you for the incredible work you've done! Let me know if there's something you'd want me to test on this HCI cluster.

nick.lloyd

@dicode-nl Found one thing that isn't working. I cannot use the Ceph SR to enable HA on the hosts. Let me know what you need from me!

pool.enableHa
{
  "pool": "8493dccd-830e-70c3-4081-0f760250642f",
  "heartbeatSrs": [
    "9a468eaf-9d48-3888-88e4-65d4859a71d5"
  ],
  "configuration": {}
}
{
  "code": "SR_OPERATION_NOT_SUPPORTED",
  "params": [
    "OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07"
  ],
  "call": {
    "duration": 27,
    "method": "pool.enable_ha",
    "params": [
      "* session id *",
      [
        "OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07"
      ],
      {}
    ]
  },
  "message": "SR_OPERATION_NOT_SUPPORTED(OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07)",
  "name": "XapiError",
  "stack": "XapiError: SR_OPERATION_NOT_SUPPORTED(OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}