Gentoo - Failed to boot with Xen

Fulgurance

Hi, I am Fulgurance, and I use actually daily Gentoo. Recently I choose to reinstall all of my system for a 64 bits only system with the Xen hypervisor.
I tried to install everything properly and follow as possible the gentoo wiki, but when I try to boot gentoo with xen, it doesn't work, I have the error:

VFS: Cannot open root device

I will give you all of my settings:

Emerge info:

zohran@alienware-m17-r3 ~ $ emerge --info
Portage 3.0.47 (python 3.11.3-final-0, default/linux/amd64/17.1/no-multilib/hardened/selinux, gcc-13, glibc-2.37-r2, 6.3.1-gentoo x86_64)
=================================================================
System uname: Linux-6.3.1-gentoo-x86_64-Intel-R-_Core-TM-_i9-10980HK_CPU_@_2.40GHz-with-glibc2.37
KiB Mem:    32469436 total,  28462716 free
KiB Swap:   50331644 total,  50331644 free
Timestamp of repository gentoo: Tue, 09 May 2023 08:30:01 +0000
Head commit of repository gentoo: ed1f36b65db2f31193609d8c6809bbe9428de5ab
sh bash 5.2_p15-r2
ld GNU ld (Gentoo 2.39 p6) 2.39.0
app-misc/pax-utils:        1.3.7::gentoo
app-shells/bash:           5.2_p15-r2::gentoo
dev-lang/perl:             5.36.1-r1::gentoo
dev-lang/python:           3.11.3::gentoo
dev-lang/rust:             1.69.0-r1::gentoo
dev-util/cmake:            3.26.3::gentoo
dev-util/meson:            1.1.0::gentoo
sec-policy/selinux-base:   2.20221101-r4::gentoo
sys-apps/baselayout:       2.13-r1::gentoo
sys-apps/openrc:           0.47.1::gentoo
sys-apps/sandbox:          2.30-r1::gentoo
sys-devel/autoconf:        2.13-r8::gentoo, 2.71-r6::gentoo
sys-devel/automake:        1.16.5-r1::gentoo
sys-devel/binutils:        2.39-r5::gentoo, 2.40-r5::gentoo
sys-devel/binutils-config: 5.5::gentoo
sys-devel/clang:           15.0.7-r1::gentoo, 16.0.3::gentoo
sys-devel/gcc:             12.2.1_p20230428-r1::gentoo, 13.1.0-r1::gentoo
sys-devel/gcc-config:      2.10::gentoo
sys-devel/libtool:         2.4.7-r1::gentoo
sys-devel/lld:             15.0.7::gentoo
sys-devel/llvm:            15.0.7::gentoo, 16.0.3::gentoo
sys-devel/make:            4.4.1::gentoo
sys-kernel/linux-headers:  6.3::gentoo (virtual/os-headers)
sys-libs/glibc:            2.37-r2::gentoo
sys-libs/libselinux:       3.5::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    volatile: False
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts: 

ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=skylake -O2 -pipe -mmovbe -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mpopcnt -mavx -mavx2 -maes -mpclmul -mfsgsbase -mrdrnd -mfma -mbmi -mbmi2 -mrdseed -madx -mprefetchwt1 -mclflushopt -mxsavec -mxsaves"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=skylake -O2 -pipe -mmovbe -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mpopcnt -mavx -mavx2 -maes -mpclmul -mfsgsbase -mrdrnd -mfma -mbmi -mbmi2 -mrdseed -madx -mprefetchwt1 -mclflushopt -mxsavec -mxsaves"
DISTDIR="/var/cache/distfiles"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GDK_PIXBUF_MODULE_FILE GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR XDG_STATE_HOME"
FCFLAGS="-march=skylake -O2 -pipe -mmovbe -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mpopcnt -mavx -mavx2 -maes -mpclmul -mfsgsbase -mrdrnd -mfma -mbmi -mbmi2 -mrdseed -madx -mprefetchwt1 -mclflushopt -mxsavec -mxsaves"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs binpkg-multi-instance buildpkg-live candy config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch preserve-libs protect-owned qa-unresolved-soname-deps sandbox selinux sesandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-march=skylake -O2 -pipe -mmovbe -mmmx -msse -msse2 -msse3 -mssse3 -msse4.1 -msse4.2 -mpopcnt -mavx -mavx2 -maes -mpclmul -mfsgsbase -mrdrnd -mfma -mbmi -mbmi2 -mrdseed -madx -mprefetchwt1 -mclflushopt -mxsavec -mxsaves"
GENTOO_MIRRORS="ftp://mirror.bytemark.co.uk/gentoo/ https://mirror.bytemark.co.uk/gentoo/ http://mirror.bytemark.co.uk/gentoo/ rsync://mirror.bytemark.co.uk/gentoo/ ftp://mirrors.gethosted.online/gentoo https://mirrors.gethosted.online/gentoo http://mirrors.gethosted.online/gentoo https://www.mirrorservice.org/sites/distfiles.gentoo.org/ http://www.mirrorservice.org/sites/distfiles.gentoo.org/ ftp://ftp.mirrorservice.org/sites/distfiles.gentoo.org/ rsync://rsync.mirrorservice.org/distfiles.gentoo.org/"
LANG="en_GB.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
LEX="flex"
MAKEOPTS="-j16"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
SHELL="/bin/bash"
USE="X aac acl acpi alsa amd64 audit bluetooth btrfs bzip2 caps cet cli compat crypt cryptsetup custom-cflags custom-optimization dbus device-mapper dhcpcd dri dri3 elogind experimental ffmpeg fortran gdbm git glamor gstreamer hardened iconv ipv6 jpeg jumbo-build kde libglvnd libtirpc lm-sensors lto lvm mp3 mp4 mtp ncurses networkmanager nftables nls nptl nvenc ogg open_perms opengl openmp pam pcre peer_perms phonon pie plasma png policykit pulseaudio readline seccomp selinux split-usr ssl ssp svg test-rust tiff ubac udev udisks unconfined unicode uvm v4l vorbis vulkan wayland wifi wireless x264 x265 xattr xtpax zlib" ABI_X86="64" ADA_TARGET="gnat_2021" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 ssse3 sse4_1 sse4_2" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" GRUB_PLATFORMS="efi-64 xen xen-pvh" INPUT_DEVICES="libinput synaptics" KERNEL="linux" L10N="en en-GB" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LLVM_TARGETS="NVPTX" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-4 php8-0" POSTGRES_TARGETS="postgres12 postgres13" PYTHON_SINGLE_TARGET="python3_11" PYTHON_TARGETS="python3_11" RUBY_TARGETS="ruby30" USERLAND="GNU" VIDEO_CARDS="intel i965 iris nvidia" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq proto steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  ADDR2LINE, AR, ARFLAGS, AS, ASFLAGS, CC, CCLD, CONFIG_SHELL, CPP, CPPFLAGS, CTARGET, CXX, CXXFILT, ELFEDIT, EMERGE_DEFAULT_OPTS, EXTRA_ECONF, F77FLAGS, FC, GCOV, GPROF, INSTALL_MASK, LC_ALL, LD, LFLAGS, LIBTOOL, LINGUAS, MAKE, MAKEFLAGS, NM, OBJCOPY, OBJDUMP, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS, RANLIB, READELF, RUSTFLAGS, SIZE, STRINGS, STRIP, YACC, YFLAGS

My /etc/fstab:

# <fs>                  <mountpoint>    <type>          <opts>                                  <dump/pass>

/dev/nvme0n1p2          /boot           btrfs           relatime                                0       2
/dev/mapper/System-Root /               btrfs           relatime                                0       1
/dev/mapper/System-Swap none            swap            sw                                      0       0
tmpfs                   /tmp            tmpfs           rw,nosuid,noatime,nodev,mode=1777       0       0
tmpfs                   /var/tmp        tmpfs           rw,nosuid,noatime,nodev,mode=1777       0       0
#/dev/sda1              /xen            btrfs           relative                                0       3

Just to explain, my laptop have a 500G SSD drive, partitioned in 3: nvme0n1p1 for /boot/efi, nvme0n1p2 for /boot and nvme0n1p3 encrypted with inside LVM, itself split in two logical partitions, 48G of Swap and the left space for root.

Without Xen, my system boot properly (just to let you know)

This is my grub config (I think maybe it's where the problem start, because I am not sure about my grub config:

GRUB_DISTRIBUTOR="Gentoo"
GRUB_ENABLE_CRYPTODISK=y

# Default menu entry
#GRUB_DEFAULT=0

# Boot the default entry this many seconds after the menu is displayed
#GRUB_TIMEOUT=5
#GRUB_TIMEOUT_STYLE=menu

# Append parameters to the linux kernel command line
#GRUB_CMDLINE_LINUX=""
#
# Examples:
#
# Boot with network interface renaming disabled
# GRUB_CMDLINE_LINUX="net.ifnames=0"
#
# Boot with systemd instead of sysvinit (openrc)
# GRUB_CMDLINE_LINUX="init=/usr/lib/systemd/systemd"

# Append parameters to the linux kernel command line for non-recovery entries
GRUB_CMDLINE_LINUX_DEFAULT="dolvm dobtrfs crypt_root=/dev/nvme0n1p3 root=/dev/mapper/System-Root root_trim=yes selinux=0 enforcing=0 quiet"
GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=1024M,max:1024M"
GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT="softlevel=xen dolvm dobtrfs crypt_root=/dev/nvme0n1p3 root=/dev/mapper/System-Root root_trim=yes selinux=0 enforcing=0"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal.
# Note that you can use only modes which your graphic card supports via VBE.
# You can see them in real GRUB with the command `vbeinfo'.
GRUB_GFXMODE=3840x2160

# Set to 'text' to force the Linux kernel to boot in normal text
# mode, 'keep' to preserve the graphics mode set using
# 'GRUB_GFXMODE', 'WIDTHxHEIGHT'['xDEPTH'] to set a particular
# graphics mode, or a sequence of these separated by commas or
# semicolons to try several modes in sequence.
GRUB_GFXPAYLOAD_LINUX=keep

# Path to theme spec txt file.
# The starfield is by default provided with use truetype.
# NOTE: when enabling custom theme, ensure you have required font/etc.
#GRUB_THEME="/boot/grub/themes/starfield/theme.txt"

GRUB_FONT="/boot/grub/fonts/terminus32b.pf2"

# Background image used on graphical terminal.
# Can be in various bitmap formats.
#GRUB_BACKGROUND="/boot/grub/mybackground.png"

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to kernel
#GRUB_DISABLE_LINUX_UUID=true

# Comment if you don't want GRUB to pass "root=PARTUUID=xxx" parameter to kernel
GRUB_DISABLE_LINUX_PARTUUID=false

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY=true

# Uncomment to disable generation of the submenu and put all choices on
# the top-level menu.
# Besides the visual affect of no sub menu, this makes navigation of the
# menu easier for a user who can't see the screen.
#GRUB_DISABLE_SUBMENU=y

# Uncomment to play a tone when the main menu is displayed.
# This is useful, for example, to allow users who can't see the screen
# to know when they can make a choice on the menu.
#GRUB_INIT_TUNE="60 800 1"

My /boot content:

zohran@alienware-m17-r3 ~ $ ls /boot/
config-6.3.1-gentoo  initramfs-6.3.1-gentoo.img  vmlinuz-6.3.1-gentoo   xen-4.17.gz  xenpolicy-4.17.1-pre
efi                  intel-uc.img                xen-4.17.1-pre.config  xen-4.gz
grub                 System.map-6.3.1-gentoo     xen-4.17.1-pre.gz      xen.gz

Last word, just to let you know, if I am able to boot with Xen properly, I wish to install all dom in my external SSD /dev/sda (I commented it actually under my /etc/fstab)

So what is my error in my configuration ?

Fulgurance

No one ?

olivierlambert

Well, there's mainly people around XCP-ng here, so less people able to assist on plain Xen (I'd love to have a bigger vanilla Xen community here, but it's not easy).

Have you tried on xen users mailing list?

Fulgurance

@olivierlambert So I sent now an email to the mailing list.

Just one question on my mind. If I am using a gentoo 64bits no multilib, can I run a Windows 11 with some 32 bits applications ?

olivierlambert

I think that should work, yes

AndreS

How are you building your kernel? This looks like either LVM is not available a boot time or encryption is gettting in the way. It's been too long ago that I build systems with root on LVM.
Have you tried it without encryption? This seems discussion seems to tough upon many possible issues; your error message alone is not enough to be more specific.
Gentoo Forum

Fulgurance

@AndreS Like how the gentoo wiki recommend:

https://wiki.gentoo.org/wiki/Xen

In few words, the wiki explained now, Xen can use the same kernel as the system for Xen. So normally Xen have the lvm and dmcrypt support.

But I agree with you, it look like Xen don't recognize the encrypted partition.

Even Xen use the gentoo-sources kernel, do you think I need to configure something more else ?

AndreS

The wiki only mentions that IF you use LVM how to address it in the grub config. Encryption is not even mentioned. Note that this really makes the setup much more complicated. As far as I know a plain vanilla kernel (or gentoo-sources kernel) does not have LVM enabled by default. You are trying to troubleshoot a setup that is not exactly mentioned in the wiki (only referenced in the grub setup). The Gentoo liveCD kernel has everything you can think of enabled and uses (I think) initramfs.

I would either start with a more simple scheme like building it on top of a ext4/xfs/whatever root filesystem and maybe experiment with the ecrypted/LVM version inside a VM unless you can find a guide/wiki that describes all three options: Xen dom0 on an encrypted LVM volume.

Maybe you can start here or here. Advise is to get Gentoo up and running before you build a Xen enabled kernel; that way you have first of all a fallback scenario (your system will boot and you dont't have to go through the chroot thing everytime) and you separate things to make troubleshooting easier. Good luck!

Fulgurance

@AndreS yeah I tried to see what configuration I need for Xen, but to be honest the documentation is very poor about that.

But I need encryption. I will pursue my investigations.

I emailed the xen mail list, but actually I don't have any answer

The documentation you linked me , I know already all about that , because WITHOUT xen, my system boot properly my encrypted system with LVM

Definitely the problem come from Xen unable to boot the partition, it's only xen

AndreS

@Fulgurance What did you base the Xen kernel config on? Did you start with the working kernel config and added the Xen requirements? In that case I believe you indeed need to reach out out the XEN team to understand what the issue is.

You could try to start first with LVM and add XEN (leave the encryption out) or with encryption and XEN (leave LVM out) to see if it is one of the two that is causing the issue.

Fulgurance

@AndreS Exactly what I did, I started with the working kernel config and added the Xen requirements

Fulgurance

To be honest it's a bit frustrating, I wrote to the gentoo mail list, normal and dedicated xen mail list, to the xen forum mail list and support. No one replied to me. It's like the Xen providers don't really now how to use it

I think I will try to ask the Qubes OS support

AndreS

@Fulgurance Good luck!

From you response I could not see if you tried with Xen config only first (or LVM first) instead of trying both at the same time.
This could help you narrow done the problem area. Remember you are trying to add Xen, LVM AND encrypted root at the same time.

Fulgurance

@AndreS I mentionned it already but I can repeat again. My system is already installed in 3 partitions: /boot, /boot/efi and the last one is encrypted, inside 2 LVM volumes, one for swap and and the other one as root.

I can't try just xen without LVM and dmcrypt, because my system is already installed.

And no, I am not trying to add 3 in one, without Xen, my system work already with LVM and dmcrypt