<![CDATA[Wyse 5070 VM won't booting after update bios 1.27]]>Wanted to share this with group because this had me thrown for a loop for the better part of three day & I know there are quite a few other homelab aficionados that repurposed these little thinclients because they are just so small & versatile.

TLDR: microcode update was to culprit. Hope dell either push an update or it otherwise get resolved or...I'll be unsecure, I guess.

Fixed it by rolling back to version 2:2.1-26.xs28.1.xcpng8.2

yum downgrade microcode_ctl 2:2.1-26.xs28.1.xcpng8.2

yum downgrade microcode_ctl

might work as well, but that really depends on your upgrade path.

Detail:

After a recent update I discovered that none on my virtual machines would boot save for one that would boot sporadically. It was accidentally set to pv drivers...not entirely sure how that happened, but it also could have just loaded that way when I imported the cloud image from Harvester.

The symptoms:

Not keyboard or mouse input for windows or linux.

Windows vms would boot & do one of two thing the screen would "Guest has not initialised the display (yet)" or get to the point where the drive was initialized & hang...they would basically max out what CPU they were allowed.

Linux vms would boot & always hang at the same place right after grub loaded initial ramdisk.

Try every bios setting I could no joy.

Hope you other poor stuck soles stumble of this & save your self's some time.

]]>https://xcp-ng.org/forum/topic/8736/wyse-5070-vm-won-t-booting-after-update-bios-1-27RSS for NodeMon, 13 Apr 2026 23:23:06 GMTMon, 01 Apr 2024 00:21:33 GMT60<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 28 May 2024 15:02:39 GMT]]>@andyhhp giving this a shot now.

]]>https://xcp-ng.org/forum/post/78047https://xcp-ng.org/forum/post/78047Tue, 28 May 2024 15:02:39 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 28 May 2024 14:31:49 GMT]]>@t-chamberlain I've got a fix from Intel, and @stormi has packaged it.

yum update microcode_ctl --enablerepo=xcp-ng-testing should get you microcode_ctl-2.1-26.xs29.2.xcpng8.2 which has the fixed microcode for this issue in it.

]]>https://xcp-ng.org/forum/post/78044https://xcp-ng.org/forum/post/78044Tue, 28 May 2024 14:31:49 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Fri, 05 Apr 2024 07:20:20 GMT]]>Doc about XTF testing: https://docs.xcp-ng.org/project/development-process/tests/#test-the-xen-hypervisor-itself

]]>https://xcp-ng.org/forum/post/74929https://xcp-ng.org/forum/post/74929Fri, 05 Apr 2024 07:20:20 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Thu, 04 Apr 2024 22:56:33 GMT]]>@t-chamberlain In addition to the XTF testing, could you also please (with the bad microcode) try booting Xen with spec-ctrl=no-verw on the command line, and seeing whether that changes the behaviour of your regular VMs? Please capture xl dmesg from this run too.

]]>https://xcp-ng.org/forum/post/74926https://xcp-ng.org/forum/post/74926Thu, 04 Apr 2024 22:56:33 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Thu, 04 Apr 2024 13:40:46 GMT]]>However, logs from a debug Xen might give more clues.

If you can, please follow the instructions given by @andyhhp - a Xen developer - at https://xcp-ng.org/forum/post/74855.

]]>https://xcp-ng.org/forum/post/74882https://xcp-ng.org/forum/post/74882Thu, 04 Apr 2024 13:40:46 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Thu, 04 Apr 2024 13:31:26 GMT]]>@t-chamberlain No need to, unless you have some doubts. For now the conclusion will be xl dmesg doesn't output anything particular when the VM hangs.

]]>https://xcp-ng.org/forum/post/74879https://xcp-ng.org/forum/post/74879Thu, 04 Apr 2024 13:31:26 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Thu, 04 Apr 2024 13:28:33 GMT]]>@stormi At that point that vm dsk05002 was hung for about 20 minutes. I don't know if it where are like this that an alert event is ever generated.

If need be I can boot a vm & just let it go.

]]>https://xcp-ng.org/forum/post/74878https://xcp-ng.org/forum/post/74878Thu, 04 Apr 2024 13:28:33 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Thu, 04 Apr 2024 10:08:02 GMT]]>@t-chamberlain Was the xl dmesg output produced before, or after the VM hanged?

]]>https://xcp-ng.org/forum/post/74856https://xcp-ng.org/forum/post/74856Thu, 04 Apr 2024 10:08:02 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Wed, 03 Apr 2024 21:41:24 GMT]]>@stormi

You can see the vms just kind of hang here.

374c345a-8aef-4d42-8aaa-d997590415a6-image.png

]]>https://xcp-ng.org/forum/post/74829https://xcp-ng.org/forum/post/74829Wed, 03 Apr 2024 21:41:24 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Wed, 03 Apr 2024 21:39:57 GMT]]>@stormi

Here is the xl_dmesg output after the microcode update.

xenserver02.trc.blox_microcode_upd_xl_dmesg.txt

]]>https://xcp-ng.org/forum/post/74828https://xcp-ng.org/forum/post/74828Wed, 03 Apr 2024 21:39:57 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 17:07:44 GMT]]>@t-chamberlain Yes, please. One host, with one failed VM start, would be enough

]]>https://xcp-ng.org/forum/post/74723https://xcp-ng.org/forum/post/74723Tue, 02 Apr 2024 17:07:44 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 16:53:06 GMT]]>@stormi Sorry about that only answered part of the question. I did have a vm fail to boot, but it was because of an underlying issues with drbd/xostor & don't believe related to the microcode.

]]>https://xcp-ng.org/forum/post/74721https://xcp-ng.org/forum/post/74721Tue, 02 Apr 2024 16:53:06 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 16:43:57 GMT]]>@stormi

Those are the microcode downgraded hosts. I can upgrade the packages & get those outputs as well if need be.

]]>https://xcp-ng.org/forum/post/74719https://xcp-ng.org/forum/post/74719Tue, 02 Apr 2024 16:43:57 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 16:08:02 GMT]]>@t-chamberlain Is the output of xl dmesg with, or without the bad microcode? Did a VM fail to boot since the hosts were booted?

]]>https://xcp-ng.org/forum/post/74718https://xcp-ng.org/forum/post/74718Tue, 02 Apr 2024 16:08:02 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 15:28:42 GMT]]>@stormi said in Wyse 5070 VM won't booting after update bios 1.27:

head /proc/cpuinfo

Here info for my 3 nodes.

xenserver01.trc.blox_cpuinfo.txt
xenserver01.trc.blox_xl_dmesg.txt
xenserver02.trc.blox_cpuinfo.txt
xenserver02.trc.blox_xl_dmesg.txt
xenserver03.trc.blox_cpuinfo.txt
xenserver03.trc.blox_xl_dmesg.txt

]]>https://xcp-ng.org/forum/post/74712https://xcp-ng.org/forum/post/74712Tue, 02 Apr 2024 15:28:42 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 10:38:32 GMT]]>So, could you get some logs from the boot failures?

  • head /proc/cpuinfo
  • full output of xl dmesg including booting the problem VM with the good and bad ucode in place
]]>https://xcp-ng.org/forum/post/74672https://xcp-ng.org/forum/post/74672Tue, 02 Apr 2024 10:38:32 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 10:30:59 GMT]]>My search engine just made me discover this one: https://xcp-ng.org/forum/topic/8584/dell-wyse-fw-update-breaks-vm-booting-console-frozen-tianocore-edk2-related

]]>https://xcp-ng.org/forum/post/74670https://xcp-ng.org/forum/post/74670Tue, 02 Apr 2024 10:30:59 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Tue, 02 Apr 2024 10:29:48 GMT]]>Making sure (how?) this topic is indexed by search engines would be a good starting point.

]]>https://xcp-ng.org/forum/post/74668https://xcp-ng.org/forum/post/74668Tue, 02 Apr 2024 10:29:48 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Mon, 01 Apr 2024 14:53:41 GMT]]>@olivierlambert Completely agree, years old consumer/non-HCL compliant hardware shouldn't rate in overall development progress & a pin is about the best we should hope for.

It might just be time for some home labbers to start shopping & retire come gear!

The XCP-NG community & platform are amazing...keep up the great work!

]]>https://xcp-ng.org/forum/post/74613https://xcp-ng.org/forum/post/74613Mon, 01 Apr 2024 14:53:41 GMT<![CDATA[Reply to Wyse 5070 VM won't booting after update bios 1.27 on Mon, 01 Apr 2024 08:36:35 GMT]]>@stormi I'm not sure there's many things we can do about it, maybe a proper/documented way to pin the latest working microcode?

]]>https://xcp-ng.org/forum/post/74587https://xcp-ng.org/forum/post/74587Mon, 01 Apr 2024 08:36:35 GMT