/dev/xvda not appearing in guest

keeely

The /dev/xvda device node is not appearing in my guest. How do I trouble-shoot this please?
/sys/bus/xen/drivers/vbd exists on dom0. There is nothing in /sys/bus/xen/drivers/ or /sys/bus/xen/devices/ on my domU.

thanks!

olivierlambert

Please provide more details. I assume it's a Linux guest, but which distro? Probably a lack of Xen drivers embed in the kernel.

keeely

@olivierlambert I compiled my own kernel v5.4.21, and included the PV stuff. I definitely included Xen virtual block device support (compiled in, not a module), CONFIG_XEN_BLKDEV_FRONTEND. I thought that was all I should need. The distro is Slackware. Thanks.

olivierlambert

Maybe @r1 can help on that

keeely

I should probably be a bit more specific about my kernel config:

# cat .config | grep XEN
CONFIG_XEN=y
# CONFIG_XEN_PV is not set
# CONFIG_XEN_PVHVM is not set
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_PCI_XEN=y
# CONFIG_NET_9P_XEN is not set
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_SCSI_FRONTEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_TCG_XEN is not set
# CONFIG_XEN_WDT is not set
# CONFIG_DRM_XEN is not set
CONFIG_XEN_FBDEV_FRONTEND=y
# CONFIG_SND_XEN_FRONTEND is not set
CONFIG_MMC_SDHCI_XENON=m
CONFIG_XEN_BALLOON=y
# CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set
CONFIG_XEN_SCRUB_PAGES_DEFAULT=y
CONFIG_XEN_DEV_EVTCHN=y
# CONFIG_XEN_BACKEND is not set
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
# CONFIG_XEN_GRANT_DMA_ALLOC is not set
CONFIG_SWIOTLB_XEN=y
# CONFIG_XEN_PVCALLS_FRONTEND is not set
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_EFI=y
CONFIG_XEN_ACPI=y

r1

@keeely Does your default distro kernel work? Is your VM PV or HVM? Can you try HVM if not?

Regarding driver, its generally compiled with CONFIG_XEN_BLKDEV_FRONTEND=m and added to initrd.

keeely

@r1 The default distro does work, however I had to compile in support for rtl8139 C+ to get networking to work because the usual rtl 8139 driver did not work.

<*>     RealTek RTL-8139 C+ PCI Fast Ethernet Adapter support.    # <==== needed this one
<*>     RealTek RTL-8129/8130/8139 PCI Fast Ethernet Adapter support

My VM is I believe HVM, however this is not all that easy to determine. I'm using XCP-ng Center and the only page that comes close to giving me an option is this one (General -> Properties):

I don't believe the virtual machine is presenting to me anything other than straight HVM hardware, because I'm sure I would have seen something in dmesg about the detection of the Xen bus at least.

The only guide to kernel configs that I've seen is here:
https://wiki.xenproject.org/wiki/Mainline_Linux_Kernel_Configs

I'm broadly following that, however it seems to be a little out of date.

I would like to find a page like that, but with one page per virtualisation mode, e.g. kernel options for each of the following cases:

HVM
HVM + PV Drivers
PVHVM
PVH
PV

After searching for some time I've been unable to find that information.

olivierlambert

This information is available in Xen Orchestra (current VM virt mode)

r1

@keeely said in /dev/xvda not appearing in guest:

I had to compile in support for rtl8139 C+ to get networking to work because the usual rtl 8139 driver did not work.

Is this for Host or VM?

keeely

That was for VM.

keeely

@olivierlambert I found only this under advanced:

Also under general section for the VM:

So what does that mean? HVM? If so where is the PVHVM option?

Interestingly under disks, I see this:

So the device is listed as xvda, however dmesg in the guest shows sda has been detected.

keeely

@keeely Looking at the XOA appliance, it does seem to have the right mode:

Shame there is no /proc/config.gz on that VM so I can lift the kernel config.

olivierlambert

So it means there's no emulated PV drivers installed (or they aren't loaded)

r1

Can you try and compile it with "M" and push them in initrd?

Without PV drivers (Pure emulation mode) it generally detects as /dev/hda, seems it can also do /dev/sda.

keeely

@r1 said in /dev/xvda not appearing in guest:

Can you try and compile it with "M" and push them in initrd?

I have been avoiding this due to the insane amount of time it takes to do the full module compile using nested virtualisation (Slackware under XCP-ng under VMWare Fusion). But assuming that I do do it, why do I need initrd? Can't I just boot using /dev/sda1, and load the vbd frontend driver after boot? Won't both those devices still be present?

r1

@keeely Once the disk is opened by a program (emulated driver), and OS runs from it, I don't think it will allow Xen blkdev to take over. Thus the new module would render useless.

By putting in initrd, the kernel will load it at boot time and then disk is opened to continue boot.

CONFIG_XEN_BLKDEV_FRONTEND=y should have worked too but I guess that's another troubleshooting.

keeely

@r1 That's very useful info. I will try to make initrd.

keeely

There is no need to use initrd, and I finally have this booting with kernel 4.4.14 and optimised IO.

Here are the kernel options I used:

# cat .config | grep XEN
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_512GB=y
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_PVH is not set
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
# CONFIG_XEN_BLKDEV_BACKEND is not set
CONFIG_XEN_SCSI_FRONTEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
# CONFIG_XEN_NETDEV_BACKEND is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_TCG_XEN is not set
# CONFIG_XEN_WDT is not set
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_XEN_BALLOON=y
# CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PCIDEV_BACKEND=m
# CONFIG_XEN_SCSI_BACKEND is not set
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_ACPI_PROCESSOR=m
# CONFIG_XEN_MCE_LOG is not set
CONFIG_XEN_HAVE_PVMMU=y
CONFIG_XEN_EFI=y
CONFIG_XEN_AUTO_XLATE=y
CONFIG_XEN_ACPI=y
CONFIG_XEN_SYMS=y
CONFIG_XEN_HAVE_VPMU=y

XCP-ng Center reports IO Optimised.
XOA still reports the VM as HVM.

I can finally see the /dev/xvda device inside the guest however.

r1

So some issue in between 4.4.14 (why not 4.4.229?) vs 5.4.21 (or 5.4.50?).

keeely

@r1 I haven't done a 'make oldconfig' using the 4.4.14 kernel config, which would obviously be the next step. There's a long time-period between those two kernels unfortunately, a lot could have changed. The config options in the kernel are somewhat different.