Bringing Rust to the Xen Project
This blog post is an intro series on our future contributions around the Xen Project, related to bring replacement programs in Rust, at various levels in the stack.
🧅 Context
The whole platform that makes up XCP-ng contains a lot of various programs: Xen (the hypervisor), the XAPI (the API/toolstack), SMAPI (storage stack) among many other things, like guest tools and so on.
This platform is a mix of various languages like C, Python, OCaml and even Go. Some components are somehow old and not easily maintainable. We wanted to use that opportunity to rethink some of them, and use that opportunity to bring Rust to the table.
🦀 Why Rust?
There's many reasons for that. You could argue there's already 4 languages used inside XCP-ng, so why add another? Luckily, Go is only used for one component we might entirely replace, there it won't raise the overall number of languages used. Good, but why then?
Technical aspects
Rust will be bring both performance and memory safety, which is -in my opinion- truly relevant in a virtualization platform like XCP-ng. The fact there's no garbage collector is also very attractive to us. Then, the borrow checker is also an extra feature that might help us to reduce potential bugs, which is another plus. But there's more than just technical reasons!
Other considerations
Since Rust is safe and fast, what else do we need? We also need a programming language that is able to work at various levels (lower and higher in the stack). I wouldn't trust Go or Python to deal with such low level things we can have in XCP-ng, and -in the same way- not C either to do higher level stuff. Using Rust brings the potential to be everywhere in the XCP-ng' stack.
Also, Rust is not a "niche" language anymore. For example, even if it's great, OCaml (used in XAPI) is not known enough, reducing our opportunities to easily hire experienced people in this language. This is also reducing the capacity for an open source community to get contributors. We believe that Rust won't be a blocker for that (both hiring and contributions), likely even the opposite: a driver to get more people on board, as it's a "wanted" technology.
And finally, it's praised by its users. You can find various sources on this, but the biggest one is the Stack Overflow yearly survey, and Rust is on top as "Loved" for many many years: it's not a temporary trend, it's really something growing and improving.
Rust is on its seventh year as the most loved language with 87% of developers saying they want to continue using it.
💗 Ramping up
Obviously, don't expect that we rewrite the Xen hypervisor and everything in Rust as our first attempt. In fact, our goal here is to start to replace some smaller components around it, allowing us to "ramp up" on the language itself, and think about how to replace things brick after brick, for the whole platform.
That's why we selected our first 2 components to be rewritten in Rust: the Linux guest tools (currently in Go) and the metrics daemon (in OCaml).
Linux Guest tools
We'll have a dedicated article soon on this piece of software. But basically, right now, it's a tool written in Go, not hosted within the Xen Project but on Github by XenServer team.
We are not really happy with the code quality, the responsiveness and the difficulties to package them on this "upstream". Also, it should be hosted within the Xen Project repo, not in a Citrix (now Cloud Software Group) XenServer organization. That's why we created a new repository at the place it should be.
rrdd metrics daemon
This part is a small daemon running in the Dom0, building the metrics exported and use by all the XAPI clients, like Xen Orchestra. This daemon generates RRDs encapsulated in XML, which is rather costly in terms of CPU usage to decode (and likely to generate in the first place). As a small and isolated piece of software, it's a also a good target to be converted in Rust!
This is also part of a wider project to bring Open Metrics to the whole stack, from our work started directly upstream within the hypervisor, that we'll move up to our XAPI client, Xen Orchestra. It will be great to have Open Metrics all along!
🎯 Other targets
We also have other "bigger projects" in Rust, probably starting next year (ideally earlier, but there's many things to achieve before getting there).
xenopsd-ng
As we said, using Rust is also the opportunity to improve the current architecture and remove some technical debt. xenopsd-ng is typically that kind of project: we'll try to unify the Xen toolstack while doing a single program for the low-level Xen API. The current stack is this:
With xenopsd-ng, the objective is to get here:
One piece to expose all the higher APIs, written in Rust.
Other components?
We will also continue to explore the whole ecosystem and prioritize on isolated components that could be rewritten in Rust.
🐼 Beyond the tools: Xen itself?
That's a very good question: should we start to bring Rust directly inside the Xen hypervisor itself?
Since it's a very "low level" piece of code, a micro-kernel, you can already have very specific and weird issues while it's mostly written in C (and tiny bits in Assembly). It's not rare to have something like a compiler optimization generating bugs, despite the maturity of the C ecosystem (even with gcc)!
That's why we prefer to start with simple things and a higher level first, while getting more experience in Rust. The other thing to do is to monitor Rust maturity on "very close to the metal" behaviors and potential unwanted side effects.
Anyway, there's enough to start with projects before modifying the core of Xen in Rust. But I'm sure this Rust journey will come to that point after some time. Stay tuned to this new series, to follow our progress in both the Xen and Rust worlds!