OmniOS / Illumos / Solaris as a guest - not working

DustinB

Well dang it, I could've swore it booted once.. but now I'm getting

BobTB12

@DustinB
Yep, this is the problem we are having.

Interestingly it works in ESXI and Proxmox. Some simple thing maybe in the XCP-ng setting to make it work would be a dream

xpvd is Xen Paravirtualized Driver?

It seems its not the disk controller driver problem, as the setup finishes ok. Some device is being started thereafter (svga? network?) which causes the crash.

Andrew

@BobTB12 Very odd.... A while ago I installed OI Hipster and booted it as a test. I just booted the installed VM and it works. I tried installing it again and I can't get it to install/boot now but the old install still works.

It looks like some Xen change in XCP 8.2.1, but I don't know what or when...

The good news, I just installed it on XCP 8.3 and it works..... but it's slow because it does not have any Xen drivers.

BobTB12

@Andrew
Wow, interesting... I am installing this on 8.3 beta 4 (the latest). I thought it is broken in the new 8.3, but is broken for 8.2.1 too...

I tried Solaris 11.4 again (the free one) and I could perhaps make it work if I could enter the console at boot to set grub. The console just ignores my keystrokes. I am now completely out.

BobTB12

@Danp Do you think this is hypervisor problem or guest problem?

Danp

@BobTB12 My guess would be the guest, but it would be interesting to know if the behavior changes on a newer version of Xen.

BobTB12

@Danp Just now tried on Xen 4.17. Still the same. Seems as @Andrew said, on older XCP it did still work.

Andrew

@BobTB12 On XCP 8.3/Xen 4.17 the OI installer boots and runs but errors out (not kernel crash) before it finishes. The installed OS is bootable but has the errors and does not finish booting. This is with BIOS boot as UEFI boot fails.

BobTB12

@Andrew
You gave me the idea to boot it in UEFI mode. This loads much further down, almost finishes compared to BIOS. Might be that the guest does not support 1.3 NVMe.

Is there a way in XCP-ng to change the controller/disk type to SCSI or SATA or something like that. In ESXI it is easy to do, I can not find this option here. This might then work!

nvme0: NVMe spec version 1.3
panic[cpu1]/thread=fffffe00233cdc20: programming error: invalid NS/format in cmd fffffe16dbc1dd40

Warning - stack not written to the dump buffer
fffffe00233cd710 genunix:dev_err+81 ()
fffffe00233cd750 nvme:nvme_check_generic_cmd_status+236 ()
fffffe00233cd7d0 nvme:nvme_set_features+e5 ()
fffffe00233cd800 nvme:nvme_write_cache_set+47 ()
fffffe00233cd8a0 nvme:nvme_init+9f0 ()
fffffe00233cd970 nvme:nvme_attach+51b ()
fffffe00233cd9e0 genunix:devi_attach+b1 ()
fffffe00233cda20 genunix:attach_node+8b ()
fffffe00233cda70 genunix:i_ndi_config_node+118 ()
fffffe00233cdaa0 genunix:i_ddi_attachchild+3e ()
fffffe00233cdae0 genunix:devi_attach_node+64 ()
fffffe00233cdb50 genunix:config_immediate_children+b0 ()
fffffe00233cdba0 genunix:devi_config_common+6a ()
fffffe00233cdc00 genunix:mt_config_thread+10b ()
fffffe00233cdc10 unix:thread_start+b ()

mchadley002

@BobTB12,

For what it's worth, I came back to a project I was working on trying to use OpenIndiana in a VM on XCP-ng 8.2 today and serendipitously, that project just released a new version 2024.04 on the same day. I gave the new ISOs a try and was able to boot into the installer and install it and then reboot into OI no problem. I tried both the GUI and text installers without an issue. I don't actually know what the difference is, but the release note does say "New Hardware Support" so I guess whatever that is worked! Not sure about other Illumos based distro; whether this is something later versions of them will pickup as well from their common base or if it's something that OI did specifically .

Andrew

@mchadley002 @BobTB12 Thanks for the info! I concur, it boots now! (XCP 8.2 and 8.3)

Two important hints: First ENABLE Viridian for the VM. It does not want to boot without it. Second BOOT with UEFI, BIOS does work but disk IO sucks.

I used the ISO boot image: OI-hipster-minimal-20240426.iso

Buckle8711

Another potential new user here looking to migrate from VMWare and evaluating Proxmox and xcp-ng.
I was leaning towards xcp-ng until I hit a blocker with laetst OmniOS 151052 and xcp-ng 8.3 that fails to boot in UEFI or BIOS modes. I tried 8.2 as well that also fails.

any suggestions greatly appreciated.

Buckle8711

Enabling 'Viridian' enables the client to boot but with some errors.

Andrew

@Buckle8711 It boots just fine for me with XCP 8.3... with the same PCI Warning... And the notices are not errors, so it works (I don't have the notices on my VM).

Buckle8711

@Andrew Thanks for confirming, it seems to be working fine so at least I can progress.
Can I ask what vm guest tools you are running? I dont see Solaris/Illumos/OmniOS listed in the wiki so not sure how to proceed.

Andrew

@Buckle8711 No guest tools.... OpenSolaris does include Xen drivers but I don't know if they have been updated. I don't seem them loading.

Buckle8711

I've hit a blocker in my migration to xcp-ng. I'm using OmniOS as a lightweight file server with Napp-it management interface.

I reached a point where I needed to add two additional VIF's to support some additional VLANs bring the OmniOS VM total to four. Unfortunately only the first two are mapped to devices in OmniOS. Open to any ideas here.

Andrew

@Buckle8711 I see that too. QEMU is only started with two interfaces passed to the guest. I don't know if that's because of the lack of tools or native drivers or some other config issue.

@olivierlambert XCP/Xen/Qemu expert question: Why does qemu start with only two interfaces when more are configured in XO?
qemu-dm-7 -machine pc-i440fx-2.10,accel=xen ... -device e1000,netdev=tapnet1,mac=5a:7f:c3:ba:04:d6,addr=5,rombar=0 -netdev tap,id=tapnet1,fd=7 -device e1000,netdev=tapnet0,mac=8e:68:02:55:da:c7,addr=4,rombar=0 -netdev tap,id=tapnet0,fd=8 ...
Where's the third and fourth? I see other VMs with more. Is it a template or vm-param issue?

olivierlambert

Ping @Teddy-Astie if you know the best person internally to take a look at this

TeddyAstie

@Andrew said in OmniOS / Illumos / Solaris as a guest - not working:

@olivierlambert XCP/Xen/Qemu expert question: Why does qemu start with only two interfaces when more are configured in XO?
qemu-dm-7 -machine pc-i440fx-2.10,accel=xen ... -device e1000,netdev=tapnet1,mac=5a:7f:c3:ba:04:d6,addr=5,rombar=0 -netdev tap,id=tapnet1,fd=7 -device e1000,netdev=tapnet0,mac=8e:68:02:55:da:c7,addr=4,rombar=0 -netdev tap,id=tapnet0,fd=8 ...
Where's the third and fourth? I see other VMs with more. Is it a template or vm-param issue?

~~These are the same "interfaces" just plugged differently to the guest (e.g tapnet0 exposed as e1000 and "tap" and tapnet1 exposed as e1000 and "tap" too).~~

You have 4 interfaces configured ?