Native Ceph RBD SM driver for XCP-ng

nick.lloyd

@dicode-nl Just got Ceph 15 working on Dom0 and will be testing this thought experiment. I commend your effort on getting this working! I've been begging for Vates to work on this as LINSTOR is not a "real" solution to the HCI problem with its current limitations (I have clients with 20+ vSAN nodes). I'll keep you updated

dicode-nl

@nick.lloyd if there is anything that isn't working just let me know. I do advise the SMAPIv3 solution at that implementation offers the full feature set!

nick.lloyd

@dicode-nl Just got it all working! Since prefer-crc isn't available in kernels prior to 5.x (can't remember the specific version), I ended up needing to use ms_mode: legacy to get VMs to boot on it. That being said, the only thing I don't see that would be a nice-to-have would be QCOW2 support. The fact that this is actually working is a miracle itself!

Got some initial testing done on my 2TB Micron 1100 SATA SSD pool. Not too bad of results so far. I'll do my bank of 3.2TB NVMe SSDs later, but I want to try and get this more automated/working as an XO(A) plugin first. Thank you for the incredible work you've done! Let me know if there's something you'd want me to test on this HCI cluster.

nick.lloyd

@dicode-nl Found one thing that isn't working. I cannot use the Ceph SR to enable HA on the hosts. Let me know what you need from me!

pool.enableHa
{
  "pool": "8493dccd-830e-70c3-4081-0f760250642f",
  "heartbeatSrs": [
    "9a468eaf-9d48-3888-88e4-65d4859a71d5"
  ],
  "configuration": {}
}
{
  "code": "SR_OPERATION_NOT_SUPPORTED",
  "params": [
    "OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07"
  ],
  "call": {
    "duration": 27,
    "method": "pool.enable_ha",
    "params": [
      "* session id *",
      [
        "OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07"
      ],
      {}
    ]
  },
  "message": "SR_OPERATION_NOT_SUPPORTED(OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07)",
  "name": "XapiError",
  "stack": "XapiError: SR_OPERATION_NOT_SUPPORTED(OpaqueRef:f3997856-74b4-eb65-2964-bd327b181c07)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

dicode-nl

@nick.lloyd I guess you haven't installed the newer kernel modules then? Those would fix the new auth method and the msrg2 protocol.

Also, the SMAPIv3 is better then the SMAPIv1, then you don;t need qcow2. All snapshots, cbt etc are then ceph-native.

nick.lloyd

@dicode-nl I am using SMAPIv3 (rbd-vol). The limitation on the msrg2 protocol is XCP-ng's Dom0 being... ancient to put it gently. I have not tried to backport/install any kernel modules yet. I may try it, but honestly I'm happy with where this is at now. The biggest thing is the inability to host the heartbeat datastore for HA. The reason qcow2 would be nice is just for >2TB VMs (unless this is possible with VHD and I'm just being dumb). Not that I have any of those currently.

dicode-nl

@nick.lloyd I supply patched kernel modules which support the extra features needed for full rbd functionality. in that case when running in datapath blkback (default) no blktap/vhd/qcow2 is used, it is RAW rbd performance. snapshotting, cbt and SXM are handled natively with RBD features

this plugin has been developed with no userspace daemon / no overlay like vhd/qcow2 in mind, so you shouln't have the 2tb limit.

if you want to remain with the older kernel modules you can use the other datapath modes which run trough tapdisk / qemu-nbd and support the vhd/qcow2 formats

dicode-nl

@nick.lloyd I'll look into the HA support, need to dig into that whats needed to support that. keep you posted

nick.lloyd

@dicode-nl For some reason, I can't seem to get the Ceph Mon to achieve quorum with msrg2. I'll have to look into why that is because I do have the SMAPIv3 drivers/patches installed. I'll probably have to throw an LLM at it because I won't have the time with work and US holidays, but I'll do my best to keep up communication.

To be honest (and maybe this is me being stubborn to change), I definitely prefer the older modules as to keep the "traditional" way of managing the storage in XCP-ng.

dicode-nl

@nick.lloyd that's something out of this plugin, that's ceph native. most clusters should run on msgr2 native. which ceph versions are you running?