Well, so I found it, the culprit is not even the SM framework itself, but rather the XAPI implementation, the problem is here:
https://github.com/xapi-project/xen-api/blob/master/ocaml/xapi/xapi_vm_snapshot.ml#L250
This logic is hard-coded into XAPI - when you revert to snapshot it:
First deletes the VDI image(s) of the VM (that is "root" for entire snapshot hierarchy - this is actually illegal in CEPH to delete such image)
Then it creates new VDIs from the snapshot
Then it modifies the VM and rewrites all VDI references to newly created clones from the snapshot
This is fundamentally incompatible with the native CEPH snapshot logic because in CEPH:
You can create any amount of snapshots you want for an RBD image - but that makes it illegal to delete the RBD image as long as there is any snapshot. CEPH is using layered CoW for this, however snapshots are always read-only (which is actually fine in Xen world as it seems).
Creation of snapshot is very fast
Deletion of snapshot is also very fast
Rollback of snapshot is somehow also very fast (IDK how CEPH does this though)
You can create a clone (new RBD image) from a snapshot - that creates a parent reference to the snapshot - eg. snapshot can't be deleted until you make the new RBD independent of it via flatten operation (IO heavy and very slow).
In simple words when:
You create image A
You create snapshot S (A -> S)
You can very easily (cheap IO) drop S or revert A to S. However if you do what Xen does:
You create image A
You create snapshot S (A -> S)
You want to revert S - Xen clones S to B (A -> S -> B) and replaces VDI ref in VM from A to B
Now it's really hard for CEPH to clean both A and S as long as B depends on both of them in the CoW hierarchy. Making B independent is IO heavy.
What I can do as a nasty workaround is that I can hide VDI for A and when the user decides they want to delete S I would just hide S as well and schedule flatten of B as some background GC cleanup job (need to investigate what are my options here), which after finish would wipe S and subsequently A (if it was a last remaining snapshot for it).
That would work, but still would be awfully inefficient software emulation of CoW, completely ignoring that we can get a real CoW from CEPH that is actually efficient and IO cheap (because it happens on storage-array level).
Now I perfectly understand why nobody ever managed to deliver native RBD support to Xen - it's just that XAPI design makes it near-impossible. No wonder we ended up with weird (also inefficient) hacks like LVM pool on top of a single RBD, or CephFS.
