You start your post with the word 'untrue' Although I'm not sure what that's in response to. You then go on to state that my last statement is untrue. I presume there you are talking about the line:

"Unfortunately this means there is no free way for me to run XCP-ng except bare metal"

I stand by that statement. If you're telling me I can, in fact run XCP-ng (to experiment with it) if only I'll just install XCP-ng first then I don't actually need to run anything nested in the first place, I just wipe my desktop OS and put XCP-ng on. That is simply not an option (for me). Note that I said 'for me' in the above line. I'm surprised you'd say that's untrue.

I tested my VBox setup with Proxmox and it worked fine. So it should be capable of running nested hypervisors. I wonder what Xen does that's different.

Unfortunately this means there is no free way for me to run XCP-ng except bare metal

It completed with VMWare Fusion now, just took way longer than it did before for some reason. I need to be more patient.

I tried on VMWare Fusion, which worked before. It doesn't work there either. I used the same Fusion settings, the same XCP-ng ISO xcp-ng-8.1.0-2.iso.

I connected to the http web interface on port 80 to deploy XO this time.

BTW I would post a screenshot of what I'm seeing, however for the last few days this forum has given an error when I try to do so:

ERROR

Something went wrong installing the "sharp" module /lib64/libz.so.1 version ZLIB_1.2.9' not found (required by /usr/share/nginx/html/forum/node_modules/sharp
- Remove the "node_modules/sharp" directory then run npm install --ignore-scripts=false --verbose" and look for errors - Consult the installation documentation at https://sharp.pixelplumbing/com/install - Search for this error at https://github.com/lovell/sharp/issues

The above error appears in red in the bottom right corner of the screen in a toaster message, it fades after a few seconds.
(I may have made a mistake with the above because I'm typing it from a screenshot).

As I said, this only happened since last week.

@olivierlambert Good point. I definitely did check the box for nested virt. I will try again from scratch with VMWare Fusion where it worked before.

The first time I tried this it worked (once I'd worked around certificate problems). Now I've tried it again yesterday on a completely different setup, with latest XCP-ng running under VirtualBox instead of VMWare but I just get the spinner forever. When I checked the XCP-ng console with xe vm-list it shows that the VM has been created. What are the steps to trouble-shoot this process please?

thanks!

It seems I got a kernel panic for some other reason with 5.4.21. It's all working now, thanks.

@r1 I haven't done a 'make oldconfig' using the 4.4.14 kernel config, which would obviously be the next step. There's a long time-period between those two kernels unfortunately, a lot could have changed. The config options in the kernel are somewhat different.

There is no need to use initrd, and I finally have this booting with kernel 4.4.14 and optimised IO.

Here are the kernel options I used:

# cat .config | grep XEN
CONFIG_XEN=y
CONFIG_XEN_DOM0=y
CONFIG_XEN_PVHVM=y
CONFIG_XEN_512GB=y
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
# CONFIG_XEN_PVH is not set
CONFIG_PCI_XEN=y
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
# CONFIG_XEN_BLKDEV_BACKEND is not set
CONFIG_XEN_SCSI_FRONTEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
# CONFIG_XEN_NETDEV_BACKEND is not set
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_TCG_XEN is not set
# CONFIG_XEN_WDT is not set
CONFIG_XEN_FBDEV_FRONTEND=y
CONFIG_XEN_BALLOON=y
# CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set
CONFIG_XEN_SCRUB_PAGES=y
CONFIG_XEN_DEV_EVTCHN=y
CONFIG_XEN_BACKEND=y
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
CONFIG_SWIOTLB_XEN=y
CONFIG_XEN_PCIDEV_BACKEND=m
# CONFIG_XEN_SCSI_BACKEND is not set
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_ACPI_PROCESSOR=m
# CONFIG_XEN_MCE_LOG is not set
CONFIG_XEN_HAVE_PVMMU=y
CONFIG_XEN_EFI=y
CONFIG_XEN_AUTO_XLATE=y
CONFIG_XEN_ACPI=y
CONFIG_XEN_SYMS=y
CONFIG_XEN_HAVE_VPMU=y

XCP-ng Center reports IO Optimised.
XOA still reports the VM as HVM.

I can finally see the /dev/xvda device inside the guest however.

@r1 That's very useful info. I will try to make initrd.

@r1 said in /dev/xvda not appearing in guest:

Can you try and compile it with "M" and push them in initrd?

I have been avoiding this due to the insane amount of time it takes to do the full module compile using nested virtualisation (Slackware under XCP-ng under VMWare Fusion). But assuming that I do do it, why do I need initrd? Can't I just boot using /dev/sda1, and load the vbd frontend driver after boot? Won't both those devices still be present?

@keeely Looking at the XOA appliance, it does seem to have the right mode:

Shame there is no /proc/config.gz on that VM so I can lift the kernel config.

@olivierlambert I found only this under advanced:

Also under general section for the VM:

So what does that mean? HVM? If so where is the PVHVM option?

Interestingly under disks, I see this:

So the device is listed as xvda, however dmesg in the guest shows sda has been detected.

That was for VM.

@r1 The default distro does work, however I had to compile in support for rtl8139 C+ to get networking to work because the usual rtl 8139 driver did not work.

<*>     RealTek RTL-8139 C+ PCI Fast Ethernet Adapter support.    # <==== needed this one
<*>     RealTek RTL-8129/8130/8139 PCI Fast Ethernet Adapter support  

My VM is I believe HVM, however this is not all that easy to determine. I'm using XCP-ng Center and the only page that comes close to giving me an option is this one (General -> Properties):

I don't believe the virtual machine is presenting to me anything other than straight HVM hardware, because I'm sure I would have seen something in dmesg about the detection of the Xen bus at least.

The only guide to kernel configs that I've seen is here:
https://wiki.xenproject.org/wiki/Mainline_Linux_Kernel_Configs

I'm broadly following that, however it seems to be a little out of date.

I would like to find a page like that, but with one page per virtualisation mode, e.g. kernel options for each of the following cases:

HVM
HVM + PV Drivers
PVHVM
PVH
PV

After searching for some time I've been unable to find that information.

I should probably be a bit more specific about my kernel config:

# cat .config | grep XEN
CONFIG_XEN=y
# CONFIG_XEN_PV is not set
# CONFIG_XEN_PVHVM is not set
CONFIG_XEN_SAVE_RESTORE=y
# CONFIG_XEN_DEBUG_FS is not set
CONFIG_PCI_XEN=y
# CONFIG_NET_9P_XEN is not set
CONFIG_XEN_PCIDEV_FRONTEND=y
CONFIG_XEN_BLKDEV_FRONTEND=y
CONFIG_XEN_SCSI_FRONTEND=y
CONFIG_NETXEN_NIC=m
CONFIG_XEN_NETDEV_FRONTEND=y
CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y
CONFIG_HVC_XEN=y
CONFIG_HVC_XEN_FRONTEND=y
# CONFIG_TCG_XEN is not set
# CONFIG_XEN_WDT is not set
# CONFIG_DRM_XEN is not set
CONFIG_XEN_FBDEV_FRONTEND=y
# CONFIG_SND_XEN_FRONTEND is not set
CONFIG_MMC_SDHCI_XENON=m
CONFIG_XEN_BALLOON=y
# CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set
CONFIG_XEN_SCRUB_PAGES_DEFAULT=y
CONFIG_XEN_DEV_EVTCHN=y
# CONFIG_XEN_BACKEND is not set
CONFIG_XENFS=y
CONFIG_XEN_COMPAT_XENFS=y
CONFIG_XEN_SYS_HYPERVISOR=y
CONFIG_XEN_XENBUS_FRONTEND=y
CONFIG_XEN_GNTDEV=m
CONFIG_XEN_GRANT_DEV_ALLOC=m
# CONFIG_XEN_GRANT_DMA_ALLOC is not set
CONFIG_SWIOTLB_XEN=y
# CONFIG_XEN_PVCALLS_FRONTEND is not set
CONFIG_XEN_PRIVCMD=y
CONFIG_XEN_EFI=y
CONFIG_XEN_ACPI=y

@olivierlambert I compiled my own kernel v5.4.21, and included the PV stuff. I definitely included Xen virtual block device support (compiled in, not a module), CONFIG_XEN_BLKDEV_FRONTEND. I thought that was all I should need. The distro is Slackware. Thanks.

The /dev/xvda device node is not appearing in my guest. How do I trouble-shoot this please?
/sys/bus/xen/drivers/vbd exists on dom0. There is nothing in /sys/bus/xen/drivers/ or /sys/bus/xen/devices/ on my domU.

thanks!