Native Ceph RBD SM driver for XCP-ng

henri9813

Hello,

Thanks for your work !

We have some hypervisors of tests at Gladhost, we can use them with pleasure to test your work on xcp-ng 8.3 !

Best regards

Maelstrom96

@benapetr Good news, seems like the latest version for SM will implement vdi_revert which should work nicely with Ceph RBD. The only thing now is to have the newer Ceph kernel drivers in XCP-ng since the current driver is very old... (v12 I believe, we're now at v20, soon v22)

henri9813

Hello @Maelstrom96.

This should be possible in the future version of XCP-NG built on top of alma 9

olivierlambert

It's more like "partially"/loosely based on Alma (10, not 9). And yes, the kernel will be far more recent, same for Xen and so on.

dicode-nl

@olivierlambert I backported recent libceph/rbd/krb5 support to the xcp 8,3 kernel to support the new ceph authx aes256k method and include more rbd image features. I've added the rpms which install these newer versions as alt and extra modules and can be removed again as well. although I did run test on it I would suggest people to take caution and do not test in PROD

https://imirc.nl/xcpng/krb5-module-1.0.0-1.xcpng8.3.x86_64.rpm
https://imirc.nl/xcpng/libceph-module-alt-1.0.0-1.xcpng8.3.x86_64.rpm
https://imirc.nl/xcpng/rbd-module-alt-1.0.0-1.xcpng8.3.x86_64.rpm

If anyone else is interested / has feedback please do let me know!

Kind regards
Tim

olivierlambert

Nice! Have you tested the snapshot, revert and so on?

dicode-nl

@olivierlambert busy writing/adapting de RBD SR and running tests. once I got the results and thins finalized I'll post and update with the RBDSR file (based on @benapetr)

dicode-nl

I've finished work on the RBD SR driver. Packaged as an RPM which makes sure the other kernel module drivers are required (otherwise the newer RBD features aren't available) and it doesn't require any rbd tooling on the dom0.

https://imirc.nl/xcpng/sm-additional-rbd-2.0-1.xcpng8.3.noarch.rpm

This needs a cephx key to have dom0 access the rbd images and a ceph mgr user/pw to manage the SR.

The driver creates a rbd namespace with the sr-uuid as it's name and the images are the vdi-uuid. so multiple sr's on the same ceph rbd pool are supported.

If leverages the xen-blkback dom0 to expose the vdi to the guest. No tapdisk, qcow2 or vhd involved. Uses native RBD snapshots, leveraging the new vdi_revert and uses GC to flatten and clean snapshots and images when required.

I'll be setting up a git repo with my sources for the kernel modules and sr driver for anyone to dive into the sources and perhaps help testing and improving this,

The next thing I'm diving into is SXM and SMAPIv3 version as I guess those are the missing parts to make this future proof.

dicode-nl

For these interested and for further development:

https://github.com/dicode-nl/xcp-ng-ceph-rbd

@olivierlambert any interest in fixing this for SMAPIv3 for inclusion into XCP 9?

benapetr

@dicode-nl hello, very nice, I will definitely check it out!