OmniOS / Illumos / Solaris as a guest - not working

BobTB12

OmniOS is used by many people as storage solution with ZFS and passthrough devices (SSD&HDD) for NFS and SMB sharing. We have been using it for years on ESXI. Now because of ESXI Exodus, we have tried to set it up on XCP-ng.

OmniOS guest fails to boot in XCP-ng on version 8.2, 8.3.

Same goes for Illumos and Solaris 11.4.

Installation of OmniOS is fine: create a new vm, set viridian=false and installation progress smoothly.

When booting from freshly installed disk it panics with:
WARNING: Couldn't initialize xpv framework

panic[cpu0]/thread=ffffffffffbca07e0: unable to configure /xpvd nexus

I've tried several version of OmniOS (and latest OpenIndiana), I also tried Solaris 11.4, tried disabling apix with apix_enable=0 in /etc/system and all without success.
The same setup in xcp-ng 8.0 works like a charm with working PV-drivers (but no tools).

The interesting thing is that there is an OmniOS image for AMAZON AWS - https://omnios.org/setup/aws which runs on XEN (or did they go to KVM?) so it should actually run? How can I get it to run... it seems very close.

Anyone with an idea of a workaround or even better, a resolution to this.

DustinB

@BobTB12 I've never heard of this Distro, but I'll see if I encounter the same issues in my lab.

DustinB

@DustinB said in OmniOS / Illumos / Solaris as a guest - not working:

Reply Quote 0

No issues installing the community edition.

yet anyways

DustinB

You'll need to explain further where the issue is, I was able to install without any issue at all.

Are you attempting to create a ZFS Pool on XCP-ng?

Danp

@DustinB Did you try booting the VM?

https://xcp-ng.org/forum/topic/5141/omnios-won-t-boot-on-xcp-ng-8-1-and-8-2?_=1707855432340

https://www.illumos.org/issues/15884

DustinB

@Danp going again.. give me a minute.

DustinB

Well dang it, I could've swore it booted once.. but now I'm getting

BobTB12

@DustinB
Yep, this is the problem we are having.

Interestingly it works in ESXI and Proxmox. Some simple thing maybe in the XCP-ng setting to make it work would be a dream

xpvd is Xen Paravirtualized Driver?

It seems its not the disk controller driver problem, as the setup finishes ok. Some device is being started thereafter (svga? network?) which causes the crash.

Andrew

@BobTB12 Very odd.... A while ago I installed OI Hipster and booted it as a test. I just booted the installed VM and it works. I tried installing it again and I can't get it to install/boot now but the old install still works.

It looks like some Xen change in XCP 8.2.1, but I don't know what or when...

The good news, I just installed it on XCP 8.3 and it works..... but it's slow because it does not have any Xen drivers.

BobTB12

@Andrew
Wow, interesting... I am installing this on 8.3 beta 4 (the latest). I thought it is broken in the new 8.3, but is broken for 8.2.1 too...

I tried Solaris 11.4 again (the free one) and I could perhaps make it work if I could enter the console at boot to set grub. The console just ignores my keystrokes. I am now completely out.

BobTB12

@Danp Do you think this is hypervisor problem or guest problem?

Danp

@BobTB12 My guess would be the guest, but it would be interesting to know if the behavior changes on a newer version of Xen.

BobTB12

@Danp Just now tried on Xen 4.17. Still the same. Seems as @Andrew said, on older XCP it did still work.

Andrew

@BobTB12 On XCP 8.3/Xen 4.17 the OI installer boots and runs but errors out (not kernel crash) before it finishes. The installed OS is bootable but has the errors and does not finish booting. This is with BIOS boot as UEFI boot fails.

BobTB12

@Andrew
You gave me the idea to boot it in UEFI mode. This loads much further down, almost finishes compared to BIOS. Might be that the guest does not support 1.3 NVMe.

Is there a way in XCP-ng to change the controller/disk type to SCSI or SATA or something like that. In ESXI it is easy to do, I can not find this option here. This might then work!

nvme0: NVMe spec version 1.3
panic[cpu1]/thread=fffffe00233cdc20: programming error: invalid NS/format in cmd fffffe16dbc1dd40

Warning - stack not written to the dump buffer
fffffe00233cd710 genunix:dev_err+81 ()
fffffe00233cd750 nvme:nvme_check_generic_cmd_status+236 ()
fffffe00233cd7d0 nvme:nvme_set_features+e5 ()
fffffe00233cd800 nvme:nvme_write_cache_set+47 ()
fffffe00233cd8a0 nvme:nvme_init+9f0 ()
fffffe00233cd970 nvme:nvme_attach+51b ()
fffffe00233cd9e0 genunix:devi_attach+b1 ()
fffffe00233cda20 genunix:attach_node+8b ()
fffffe00233cda70 genunix:i_ndi_config_node+118 ()
fffffe00233cdaa0 genunix:i_ddi_attachchild+3e ()
fffffe00233cdae0 genunix:devi_attach_node+64 ()
fffffe00233cdb50 genunix:config_immediate_children+b0 ()
fffffe00233cdba0 genunix:devi_config_common+6a ()
fffffe00233cdc00 genunix:mt_config_thread+10b ()
fffffe00233cdc10 unix:thread_start+b ()

mchadley002

@BobTB12,

For what it's worth, I came back to a project I was working on trying to use OpenIndiana in a VM on XCP-ng 8.2 today and serendipitously, that project just released a new version 2024.04 on the same day. I gave the new ISOs a try and was able to boot into the installer and install it and then reboot into OI no problem. I tried both the GUI and text installers without an issue. I don't actually know what the difference is, but the release note does say "New Hardware Support" so I guess whatever that is worked! Not sure about other Illumos based distro; whether this is something later versions of them will pickup as well from their common base or if it's something that OI did specifically .

Andrew

@mchadley002 @BobTB12 Thanks for the info! I concur, it boots now! (XCP 8.2 and 8.3)

Two important hints: First ENABLE Viridian for the VM. It does not want to boot without it. Second BOOT with UEFI, BIOS does work but disk IO sucks.

I used the ISO boot image: OI-hipster-minimal-20240426.iso

Buckle8711

Another potential new user here looking to migrate from VMWare and evaluating Proxmox and xcp-ng.
I was leaning towards xcp-ng until I hit a blocker with laetst OmniOS 151052 and xcp-ng 8.3 that fails to boot in UEFI or BIOS modes. I tried 8.2 as well that also fails.

any suggestions greatly appreciated.

Buckle8711

Enabling 'Viridian' enables the client to boot but with some errors.

Andrew

@Buckle8711 It boots just fine for me with XCP 8.3... with the same PCI Warning... And the notices are not errors, so it works (I don't have the notices on my VM).