Native Ceph RBD SM driver for XCP-ng

psafont

@benapetr This is driven by hacky logic from 16 years ago:

on revert, unserialize the previous state, and update the VM record with its saved values. As we do not want to modify that each time we add a field in the datamodel, use some low-level database functions to iterate over the fields of a record. Not very nice as it makes some assumptions on the database layer, but seems to work allright and I don't think that database layer will change a lot in the future.

I think it might be a good idea to add a revert rpc call to the storage interface that xapi can call to, with a backup to use the current logic if necessary; xapi should be able to clean up the database afterwards. I'll ask other maintainers about this or possible alternatives, but since SMAPIv1 is considered deprecated, I doubt it will happen.

I have to say that SMAPIv3 was finally fixed upstream on June by Xenserver (migrations were finally done!) and XCP-ng should get the update that fixes it in the coming weeks. Given this, I would encourage you to take all the learnings you've acquired while doing the driver and porting it to SMAPIv3. SMAPIv1 just simply has too many problems, some of them are architectural, so in general xenserver and xcp-ng maintainers would like to see it finally go away.

for now I am still targetting XCP-ng 8.2 as that's what I use in production, and I haven't seen many SMAPIv3 drivers there.

8.2 is out of support for xenserver, and for xcp-ng yesterday was the last day it was supported, you really should update

benapetr

@psafont thanks for the reply, but isn't that 16 year old logic part of XAPI? I mean - this same hacky logic is present in SMAPIv3 isn't it?

I was going through SMAPIv3 docs and from SM driver perspective (feature-wise) it doesn't seem much different, it looks to me more like many cosmetic changes that make packaging and modularization easier (definitely a good thing), but don't really change any fundamental SM logic - the RPCs are all same as in SMAPIv1, even porting my own driver is probably going to be pretty trivial, it's just about splitting it into multiple files and add some wrappers around it, but it still won't solve my problem - the rollback RPC is just not there, so I would need to instead support this "rollback by making another snapshot of a snapshot" logic enforced by XAPI

psafont

@benapetr You're right. Unfortunately, there's no VDI revert that allows the revert to happen '. This is shown in the documentation: https://xapi-project.github.io/new-docs/toolstack/features/snapshots/index.html (see revert section)

There's an old proposal to do add this: https://xapi-project.github.io/new-docs/design/snapshot-revert/index.html

But the effort fizzed out because currently the imports do not set the snapshot_of correctly, and the operation needs to work even if the field is not set correctly, as it is now. (falling back to the current code seems sensible) https://github.com/xapi-project/xen-api/pull/2058

This needs some effort to get fixed, I'll set up some ticketing so it can be prioritized accordingly.

Maelstrom96

This post is deleted!

olivierlambert

The author of the main recent effort is basically the person who posted just before you

Maelstrom96

@olivierlambert you basically replied just after that I noticed that and deleted my message...

JamesG

Nothing fruitful to add....

But...

Oooof....

This will be somewhat messy to clean up. I'm rooting for you guys though!!

olivierlambert

There's some nice progress on @psafont's work regarding improved revert. I'm confident we'll get there

henri9813

Hello,

Thanks for your work !

We have some hypervisors of tests at Gladhost, we can use them with pleasure to test your work on xcp-ng 8.3 !

Best regards

Maelstrom96

@benapetr Good news, seems like the latest version for SM will implement vdi_revert which should work nicely with Ceph RBD. The only thing now is to have the newer Ceph kernel drivers in XCP-ng since the current driver is very old... (v12 I believe, we're now at v20, soon v22)

henri9813

Hello @Maelstrom96.

This should be possible in the future version of XCP-NG built on top of alma 9

olivierlambert

It's more like "partially"/loosely based on Alma (10, not 9). And yes, the kernel will be far more recent, same for Xen and so on.

dicode-nl

@olivierlambert I backported recent libceph/rbd/krb5 support to the xcp 8,3 kernel to support the new ceph authx aes256k method and include more rbd image features. I've added the rpms which install these newer versions as alt and extra modules and can be removed again as well. although I did run test on it I would suggest people to take caution and do not test in PROD

~~rpms removed - obsoleted~~

If anyone else is interested / has feedback please do let me know!

Kind regards
Tim

olivierlambert

Nice! Have you tested the snapshot, revert and so on?

dicode-nl

@olivierlambert busy writing/adapting de RBD SR and running tests. once I got the results and thins finalized I'll post and update with the RBDSR file (based on @benapetr)

dicode-nl

I've finished work on the RBD SR driver. Packaged as an RPM which makes sure the other kernel module drivers are required (otherwise the newer RBD features aren't available) and it doesn't require any rbd tooling on the dom0.

~~rpms removed - obsoleted~~

This needs a cephx key to have dom0 access the rbd images and a ceph mgr user/pw to manage the SR.

The driver creates a rbd namespace with the sr-uuid as it's name and the images are the vdi-uuid. so multiple sr's on the same ceph rbd pool are supported.

If leverages the xen-blkback dom0 to expose the vdi to the guest. No tapdisk, qcow2 or vhd involved. Uses native RBD snapshots, leveraging the new vdi_revert and uses GC to flatten and clean snapshots and images when required.

I'll be setting up a git repo with my sources for the kernel modules and sr driver for anyone to dive into the sources and perhaps help testing and improving this,

The next thing I'm diving into is SXM and SMAPIv3 version as I guess those are the missing parts to make this future proof.

dicode-nl

For these interested and for further development:

https://github.com/dicode-nl/xcp-ng-ceph-rbd

@olivierlambert any interest in fixing this for SMAPIv3 for inclusion into XCP 9?

benapetr

@dicode-nl hello, very nice, I will definitely check it out!

dicode-nl

@benapetr @olivierlambert I've made a new release which includes SMAPIv1 improvements and a proper SMAPIv3 volume + datapath plugin.

https://github.com/dicode-nl/xcp-ng-ceph-rbd/releases#release-v20260827

Let me know your thoughts and if there is anything you'll like to see added / changed / tested.

Next step for me is CBT and SXM.

dicode-nl

Did a lot of work this weekend and added more to the volume and datapath.
in native mode, everything is rbd native including CBT! still working on SXM
in tapdisk mode, everything is working with optimized rbd calls except SXM
in qemu mode everything works by using qemu-dp and also SXM in MSAPIv3

Huge improvements made, but qemu is the slowest mode due to extra nbt hop.
I only provided tapdisk and qemu modes to get more information on how CBT works and how SXM works. Right now I'm using that knowledge to provide SXM over native rbt. I have some good ideas but need to figure out how to make them work.

I've also made a backport to ceph v15.2 which runs on dom0 to also include the newer cephx authentication protocol. This is because the ceph dashboard cannot provide a diff between snaps (which is needed for CBT) but a pull request is pending to have this working in the future without the need for rbd userspace.

I'll update my git and packages later this week.