Xen on RISC-V: Dom0 Boot, Hypercalls and Docker Toolchain

Hi there! I'm Baptiste, a Hypervisor & Kernel intern engineer at Vates, working on adding guest domain creation support to the Xen hypervisor for RISC-V. I'd like to share my progress through a series of blog posts. This is the first one.

For the full background on where the RISC-V port stands, the 2023 and 2024 posts from the team are a good starting point. The short version: Xen can boot a dom0 on RISC-V, but hypercalls (the communication layer between the hypervisor and a VM) weren't working, and guest domain creation (booting an unprivileged VM) was even less so. My goal is to get there.

Xen's primary management interface is xl, the command-line tool built on top of libxl (libxenlight) that lets you create, list, and inspect virtual machines. At this point, xl info, xl list, and xl create are all working on RISC-V, along with a Docker environment that lets anyone build and run the full stack from any x86 laptop. This post covers the first two commands and the Docker toolchain. xl create and guest domain boot will be the subject of the next post.

Getting xl info to Work

xl info is the simplest useful command in the Xen toolstack, it queries the hypervisor for host information: memory, CPU count, Xen version, and so on. On RISC-V it was crashing immediately. Therefore, it was a good minimal start.

My starting point was a patch a Serbian team at rt-rk had submitted to the Linux mailing list. I picked it up, addressed the review comments that were made on it, and synced the Xen side.

On RISC-V, the dom0 communicates with the hypervisor via the ecall instruction, conforming to the SBI specification.

Xen intercepts these calls and dispatches them as hypercalls. The pieces needed for xl info had to be wired up one by one.

1. Event channels: connecting dom0 and Xen

The first step was getting the console to work between dom0 and Xen through event channel notifications. Event channels are Xen's interrupt mechanism between domains. Without them, dom0 can't receive asynchronous notifications from the hypervisor, which blocks all communication.

This required setting up the event channel infrastructure on the RISC-V side: allocating the shared info page and wiring up the interrupt path. My first attempt used IMSIC (Incoming Message Signaled Interrupt Controller), but it ran into an issue dispatching IRQs per-vCPU on the guest side. I switched to a local interrupt approach instead, which resolved it.

2. HYPERVISOR_vcpu_op

xl info triggers a vcpu_op hypercall to register the vcpu_info structure. This structure is a shared memory area between Xen and dom0 used to exchange per-vCPU state: pending events, runstate, and so on. Without this, dom0 can't register its vCPU with the hypervisor.

3. HYPERVISOR_sysctl

xl info uses sysctl to retrieve host-wide information: physical memory, number of CPUs, Xen version string. Adding the sysctl hypercall to the RISC-V dispatch table made this work.

4. arch_get_domain_info()

arch_get_domain_info() fills in the architecture-specific fields of the domain info structure returned by the hypervisor. On RISC-V this was a stub. Implementing it lets xl info retrieve correct domain metadata.

5. HYPERVISOR_xsm_op

XSM (Xen Security Module) operations are used by xl to check policy before performing privileged actions. The xsm_op hypercall wasn't in the RISC-V dispatch table; adding it removed another crash path.

6. HYPERVISOR_multicall

xl info batches several hypercalls together using multicall. Instead of making individual calls, each with its own context-switch overhead, the guest packages multiple requests into one. The RISC-V implementation of arch_do_multicall_call() was a stub so after implementing it and enabling multicall in the RISC-V hypercall table, xl info ran successfully!

Getting xl list to Work

xl list enumerates running domains. After xl info worked, this was the next milestone.

The problem

xl list printed the header line, then hung. No output. No response to Ctrl+C, Ctrl+Z, the process was stuck in an uninterruptible state.

~ # ./dist/install/usr/local/sbin/xl list
Name                                        ID   Mem VCPUs	State	Time(s)
^C^Z

Root cause

xl list calls libxl_domid_to_name() for each domain, which performs an xs_read() to xenstore at /local/domain/0/name. Xenstore (xenstored) is Xen's key-value database, the central registry where domains announce themselves and communicate. The read was blocking indefinitely because xenstored was not running.

Fix

The dom0 init script (rcS) inside the initrd was missing the xenstore startup sequence. Three steps were added:

mount -t xenfs xenfs /proc/xen
./dist/install/usr/local/sbin/xenstored
./lib/xen/bin/xen-init-dom0

The xenfs mount at /proc/xen is a prerequisite because xenstored uses /proc/xen/xsd_kva and /proc/xen/xsd_port to communicate with the hypervisor. After mounting it, xenstored starts, and xen-init-dom0 registers dom0's name and domid in the xenstore tree.

xen-init-dom0 was also failing on its own. It calls the hypercall XEN_DOMCTL_getdomaininfo to retrieve the domain type before writing to xenstore. That hypercall wasn't wired up on RISC-V. Adding it to the dispatch table in vsbi.c and enabling it in hypercall-defs.c fixed the issue.

After both fixes, xl list returns immediately with dom0 info.

Try It Yourself

When I started this internship, I was surprised there was no way to build Xen and its tools without significant manual toolchain setup. To remove that friction, I published baptleduc/xen-riscv64-trixie on Docker Hub. It automatically mounts your Xen source tree, builds and runs Xen with dom0 in a single command from any laptop, no cross-compiler wrestling required.

The image is based on Debian Trixie (amd64) with a full cross-compilation toolchain for riscv64. It uses pre-built Debian riscv64 packages combined with qemu-user-static. A first build stage cross-compiles BusyBox statically, which ends up in the initrd.
The image also includes: a Linux kernel, a QEMU build with AIA support, OpenSBI firmware, and the riscv64 runtime libraries needed by the Xen tools.

The source is open (of course) and lives in the xcp-ng/hypervisor-dev repository

To build everything and boot into dom0:

# Downstream Xen with RISC-V support
git clone --branch v1.1.0-riscv-domu-support --single-branch https://gitlab.com/xen-project/people/olkur/xen.git
olkur/xen.git

# Build environment
git clone https://github.com/xcp-ng/hypervisor-dev.git
cd hypervisor-dev/docker/riscv/trixie
make INITRD=initrd-tools run

What's Next for Xen on RISC-V?

With xl info and xl list working, the next milestone was xl create: booting an actual guest domain (domU) on RISC-V. It took work on the libxl toolstack for RISC-V, device tree generation for guests, vCPU context setup, and guest memory management on the hypervisor side. It's working now! But we’ll go through how in the next post.

In the meantime, the Docker image is the easiest way to get hands-on. You don't need to be a kernel developer to be useful: running the image, hitting a bug, and dropping a comment here or on the XCP-ng forum is already a contribution. And if you are comfortable with the code, the branch is open and patches are welcome!