XCP-ng and LINBIT alliance #3

XOSTOR is one of the most anticipated products by our users. It's been too long since we've filled you in on the progress of XOSTOR. So we thought "now is the moment" to take a tour of the developments in our software-based hyperconvergence solution and let you know what's coming next.

📒 Previous episodes

This is the third blog post of our series about XOSTOR, feel free to read those previous episodes if you want to catch-up:

📅 What about a release?

We have never been so close to a release for XOSTOR - the last phase of the release candidate is coming in the next few weeks and the official release will follow relatively soon after that. The commercial offer will bundle our pro support and all the LINBIT expertise behind us when you need it.

This official release will grow in the future with active health checking and adding more GUI features linked to your subscription, while anyone could use it from the CLI and without any official support (hello home labbers!).

For all those who are interested in the subject, we have created a mailing list dedicated to XOSTOR, you will be the first to know about the next steps.

Finally, we are planning a dedicated webinar to demonstrate all the XOSTOR potential, so stay tuned!

🧬 Major evolutions

Virtually every aspect of XOSTOR has been improved since our last publication. Let's take a deeper dive.

🚀 SMAPI Driver command execution performance

SMAPI (XCP-ng's storage stack) is transparently handling every XOSTOR operation in the background: snapshot, disk creation, deletion, garbage collecting and so on…

We improved the driver command execution performance by:

• The implementation of a local cache to avoid retrieving metadata each time we do a request, using the LINSTOR API, while also reducing the general number of requests.  
• For VDI commands, listing all SR VDIs only when necessary (eg. to rollback failed transactions as snap, destroy a VDI...)

🔥 Better integration in XCP-ng

Originally, XOSTOR supported a maximum of 7 hosts per default. We have increased that limitation to 31 hosts so you can create much larger clusters in your infrastructure.

We also added a daemon to automatically update SR size info, useful when an SR is configured with THIN LVM backend.

Finally, we added the ability to configure hosts without using local storage, in this case it is always possible to use LINSTOR VMs, the data will be read and written entirely via the network. That means you can now create XOSTOR SR mixing hosts with physical disks and hosts without (and even choose the disks on each hosts you want to use to create the node).

Coming next

We are currently working on a backup system for the database. Currently, it's stored on a replicated DRBD volume itself. This will allow an even more "failure proof" solution.

🔒 Security, robustness and HA

Originally the LINSTOR database is stored locally on a single host and used by a LINSTOR controller. To avoid a SPOF situation, we integrated a daemon minidrbdcluster to share the LINSTOR database with all hosts of the pool. This daemon allows us to:

• Start a LINSTOR controller automatically on any host of the pool
• In situations where the host that runs the controller happened to be shutdown or restarted, another host will take over and start a new LINSTOR controller using the shared database
• Because a controller is always present, XOSTOR can still be used even with some offline hosts in your current cluster (as long as we still have access to the database). Therefore, it's always possible to make snapshots even when a host is offline
• This new daemon is not relying on XAPI and therefore LINSTOR controller can be started without waiting for the XAPI initialization

In addition, we made the xha daemon (the one used for HA in XCP-ng) compatible with XOSTOR, therefore you can have HA activated in a XOSTOR SR. We have a better atomicity now for the SMAPI snapshots and clone commands, in case of failure, it's now possible to rollback in a stable state automatically.

Finally, we fixed several major issues in collaboration with the LINBIT team, including a problem with DRBD being blocked in RO mode during a long period due to a network config in XCP-ng (related to the iptables firewall, it was a tricky one!).

🛠️ Maintenance improved

In order to deliver quality support to XOSTOR users, it was also crucial to improve our maintenance tools integrated in the solution. That's why we added a new script linstor-kv-tool. This helper is used to display VDI metadata and the relation between VDIs and DRBD volumes. Very handy when you want to diagnose an issue! It can also be used to destroy volume info in case of bad loading in the SMAPI driver.

Next, we increased the verbosity of the SMAPI logs in order to have explicit traces and an overview of the commands executed by the driver. This will be quite helpful when it comes to pinpointing issues occurring in a specific infrastructure. Still on the logs topic, we added useful info when a DRBD is open on another host and it's required to open it locally (eg. during the coalesce process or snapshot).

Finally, we added a new XAPI linstor-manager plugin to list DRBD volumes to force destroy a resource.

Also a health check command is present to see if there is an issue in the pool.

Conclusion

As you can see, we haven't slept since we started the project. We overcame many problems, most of them very subtle and only visible in some edge cases. That's why hyperconvergence is hard: creating a working PoC in a week is doable. Making it work when you pull a network cable randomly on various machines while doing hundred operations at the same time is another story.

That's why we took our time to deliver a product that simply works. Next steps are starting to integrate creation and management in the Xen Orchestra UI, but this won't block any product release first. Stay tuned for the dedicated newsletter for our next official announcements!