Wyse 5070 VM won't booting after update bios 1.27

olivierlambert

@stormi I'm not sure there's many things we can do about it, maybe a proper/documented way to pin the latest working microcode?

t.chamberlain

@olivierlambert Completely agree, years old consumer/non-HCL compliant hardware shouldn't rate in overall development progress & a pin is about the best we should hope for.

It might just be time for some home labbers to start shopping & retire come gear!

The XCP-NG community & platform are amazing...keep up the great work!

stormi

Making sure (how?) this topic is indexed by search engines would be a good starting point.

stormi

My search engine just made me discover this one: https://xcp-ng.org/forum/topic/8584/dell-wyse-fw-update-breaks-vm-booting-console-frozen-tianocore-edk2-related

stormi

So, could you get some logs from the boot failures?

head /proc/cpuinfo
full output of xl dmesg including booting the problem VM with the good and bad ucode in place

t.chamberlain

@stormi said in Wyse 5070 VM won't booting after update bios 1.27:

head /proc/cpuinfo

Here info for my 3 nodes.

xenserver01.trc.blox_cpuinfo.txt
xenserver01.trc.blox_xl_dmesg.txt
xenserver02.trc.blox_cpuinfo.txt
xenserver02.trc.blox_xl_dmesg.txt
xenserver03.trc.blox_cpuinfo.txt
xenserver03.trc.blox_xl_dmesg.txt

stormi

@t-chamberlain Is the output of xl dmesg with, or without the bad microcode? Did a VM fail to boot since the hosts were booted?

t.chamberlain

@stormi

Those are the microcode downgraded hosts. I can upgrade the packages & get those outputs as well if need be.

t.chamberlain

@stormi Sorry about that only answered part of the question. I did have a vm fail to boot, but it was because of an underlying issues with drbd/xostor & don't believe related to the microcode.

stormi

@t-chamberlain Yes, please. One host, with one failed VM start, would be enough

t.chamberlain

@stormi

Here is the xl_dmesg output after the microcode update.

xenserver02.trc.blox_microcode_upd_xl_dmesg.txt

t.chamberlain

@stormi

You can see the vms just kind of hang here.

stormi

@t-chamberlain Was the xl dmesg output produced before, or after the VM hanged?

t.chamberlain

@stormi At that point that vm dsk05002 was hung for about 20 minutes. I don't know if it where are like this that an alert event is ever generated.

If need be I can boot a vm & just let it go.

stormi

@t-chamberlain No need to, unless you have some doubts. For now the conclusion will be xl dmesg doesn't output anything particular when the VM hangs.

stormi

However, logs from a debug Xen might give more clues.

If you can, please follow the instructions given by @andyhhp - a Xen developer - at https://xcp-ng.org/forum/post/74855.

andyhhp

@t-chamberlain In addition to the XTF testing, could you also please (with the bad microcode) try booting Xen with spec-ctrl=no-verw on the command line, and seeing whether that changes the behaviour of your regular VMs? Please capture xl dmesg from this run too.

stormi

Doc about XTF testing: https://docs.xcp-ng.org/project/development-process/tests/#test-the-xen-hypervisor-itself

andyhhp

@t-chamberlain I've got a fix from Intel, and @stormi has packaged it.

yum update microcode_ctl --enablerepo=xcp-ng-testing should get you microcode_ctl-2.1-26.xs29.2.xcpng8.2 which has the fixed microcode for this issue in it.

t.chamberlain

@andyhhp giving this a shot now.