OmniOS / Illumos / Solaris as a guest - not working

Danp

@BobTB12 My guess would be the guest, but it would be interesting to know if the behavior changes on a newer version of Xen.

BobTB12

@Danp Just now tried on Xen 4.17. Still the same. Seems as @Andrew said, on older XCP it did still work.

Andrew

@BobTB12 On XCP 8.3/Xen 4.17 the OI installer boots and runs but errors out (not kernel crash) before it finishes. The installed OS is bootable but has the errors and does not finish booting. This is with BIOS boot as UEFI boot fails.

BobTB12

@Andrew
You gave me the idea to boot it in UEFI mode. This loads much further down, almost finishes compared to BIOS. Might be that the guest does not support 1.3 NVMe.

Is there a way in XCP-ng to change the controller/disk type to SCSI or SATA or something like that. In ESXI it is easy to do, I can not find this option here. This might then work!

nvme0: NVMe spec version 1.3
panic[cpu1]/thread=fffffe00233cdc20: programming error: invalid NS/format in cmd fffffe16dbc1dd40

Warning - stack not written to the dump buffer
fffffe00233cd710 genunix:dev_err+81 ()
fffffe00233cd750 nvme:nvme_check_generic_cmd_status+236 ()
fffffe00233cd7d0 nvme:nvme_set_features+e5 ()
fffffe00233cd800 nvme:nvme_write_cache_set+47 ()
fffffe00233cd8a0 nvme:nvme_init+9f0 ()
fffffe00233cd970 nvme:nvme_attach+51b ()
fffffe00233cd9e0 genunix:devi_attach+b1 ()
fffffe00233cda20 genunix:attach_node+8b ()
fffffe00233cda70 genunix:i_ndi_config_node+118 ()
fffffe00233cdaa0 genunix:i_ddi_attachchild+3e ()
fffffe00233cdae0 genunix:devi_attach_node+64 ()
fffffe00233cdb50 genunix:config_immediate_children+b0 ()
fffffe00233cdba0 genunix:devi_config_common+6a ()
fffffe00233cdc00 genunix:mt_config_thread+10b ()
fffffe00233cdc10 unix:thread_start+b ()

mchadley002

@BobTB12,

For what it's worth, I came back to a project I was working on trying to use OpenIndiana in a VM on XCP-ng 8.2 today and serendipitously, that project just released a new version 2024.04 on the same day. I gave the new ISOs a try and was able to boot into the installer and install it and then reboot into OI no problem. I tried both the GUI and text installers without an issue. I don't actually know what the difference is, but the release note does say "New Hardware Support" so I guess whatever that is worked! Not sure about other Illumos based distro; whether this is something later versions of them will pickup as well from their common base or if it's something that OI did specifically .

Andrew

@mchadley002 @BobTB12 Thanks for the info! I concur, it boots now! (XCP 8.2 and 8.3)

Two important hints: First ENABLE Viridian for the VM. It does not want to boot without it. Second BOOT with UEFI, BIOS does work but disk IO sucks.

I used the ISO boot image: OI-hipster-minimal-20240426.iso

Buckle8711

Another potential new user here looking to migrate from VMWare and evaluating Proxmox and xcp-ng.
I was leaning towards xcp-ng until I hit a blocker with laetst OmniOS 151052 and xcp-ng 8.3 that fails to boot in UEFI or BIOS modes. I tried 8.2 as well that also fails.

any suggestions greatly appreciated.

Buckle8711

Enabling 'Viridian' enables the client to boot but with some errors.

Andrew

@Buckle8711 It boots just fine for me with XCP 8.3... with the same PCI Warning... And the notices are not errors, so it works (I don't have the notices on my VM).

Buckle8711

@Andrew Thanks for confirming, it seems to be working fine so at least I can progress.
Can I ask what vm guest tools you are running? I dont see Solaris/Illumos/OmniOS listed in the wiki so not sure how to proceed.

Andrew

@Buckle8711 No guest tools.... OpenSolaris does include Xen drivers but I don't know if they have been updated. I don't seem them loading.

Buckle8711

I've hit a blocker in my migration to xcp-ng. I'm using OmniOS as a lightweight file server with Napp-it management interface.

I reached a point where I needed to add two additional VIF's to support some additional VLANs bring the OmniOS VM total to four. Unfortunately only the first two are mapped to devices in OmniOS. Open to any ideas here.

Andrew

@Buckle8711 I see that too. QEMU is only started with two interfaces passed to the guest. I don't know if that's because of the lack of tools or native drivers or some other config issue.

@olivierlambert XCP/Xen/Qemu expert question: Why does qemu start with only two interfaces when more are configured in XO?
qemu-dm-7 -machine pc-i440fx-2.10,accel=xen ... -device e1000,netdev=tapnet1,mac=5a:7f:c3:ba:04:d6,addr=5,rombar=0 -netdev tap,id=tapnet1,fd=7 -device e1000,netdev=tapnet0,mac=8e:68:02:55:da:c7,addr=4,rombar=0 -netdev tap,id=tapnet0,fd=8 ...
Where's the third and fourth? I see other VMs with more. Is it a template or vm-param issue?

olivierlambert

Ping @Teddy-Astie if you know the best person internally to take a look at this

TeddyAstie

@Andrew said in OmniOS / Illumos / Solaris as a guest - not working:

@olivierlambert XCP/Xen/Qemu expert question: Why does qemu start with only two interfaces when more are configured in XO?
qemu-dm-7 -machine pc-i440fx-2.10,accel=xen ... -device e1000,netdev=tapnet1,mac=5a:7f:c3:ba:04:d6,addr=5,rombar=0 -netdev tap,id=tapnet1,fd=7 -device e1000,netdev=tapnet0,mac=8e:68:02:55:da:c7,addr=4,rombar=0 -netdev tap,id=tapnet0,fd=8 ...
Where's the third and fourth? I see other VMs with more. Is it a template or vm-param issue?

~~These are the same "interfaces" just plugged differently to the guest (e.g tapnet0 exposed as e1000 and "tap" and tapnet1 exposed as e1000 and "tap" too).~~

You have 4 interfaces configured ?

Andrew

@Teddy-Astie Yes...

Buckle8711

@TeddyAstie thanks for looking into this. Do you have any updates here? Any workarounds or bug reports I can follow for updates?

TeddyAstie

@Buckle8711 Hello,

I took a look but unfortunately, I haven't found a workaround for this.
I suspect the Illumos/OmniOS Xen PV drivers to be too old to work in current Xen, but we currently don't have a way to prevent those drivers from loading (e.g HVM-only (non-PVHVM) domains).

Buckle8711

@TeddyAstie thank you for investigating.