Issue after latest host update
-
Any specific MTU settings?
-
A way to check if it's not network related would be using a local SR to boot a VM and see if it works.
-
@john-c I already took a look at dmesg and /var/log/xensource.log (I crawled through >1k log lines) and couldn't find anything revealing. The NFS server is unrelated, because, as stated before, I currently only use host's local storage to eliminate possible external issues.
-
@olivierlambert That's what I'm doing, to make sure, it's not a network related issue.
-
@olivierlambert I didn't change anything, at least. Just
yum updateand it went down the flush. -
I'm not sure the
yum updateis really related. It could be a coincidence, otherwise we would have been swamped in similar reports. Or it's a very specific combo that's unseen elsewhere.What kind of hardware are we talking about?
-
@olivierlambert I finally made some progress. And it really seems to be update related.
I took one of the hosts and plugged a display and keyboard into it. When booting up, I can choose to use an older version of Xen from the boot menu. Doing so makes VMs work again.
Culprit: Xen 4.13.5-9.39 (current default)
Working: Xen 4.13.4-9.19.1 (which I can choose from boot menu)All three hosts are Fujitsu Futro 740 thin clients.
-
@RealTehreal said in Issue after latest host update:
@olivierlambert I finally made some progress. And it really seems to be update related.
I took one of the hosts and plugged a display and keyboard into it. When booting up, I can choose to use an older version of Xen from the boot menu. Doing so makes VMs work again.
Culprit: Xen 4.13.5-9.39 (current default)
Working: Xen 4.13.4-9.19.1 (which I can choose from boot menu)All three hosts are Fujitsu Futro 740 thin clients.
What's the BIOS version of the Fujitsu Futro 740 and also the more exact model please? There's lots of Fujitsu Futro 740 thin clients, so you could be using any one of them.
-
@john-c
Model: FUJITSU FUTRO S740/D3544-A1
BIOS: V5.0.0.13 R1.13.0 for D3544-A1x (09/23/2022) -
@RealTehreal said in Issue after latest host update:
@john-c
Model: FUJITSU FUTRO S740/D3544-A1
BIOS: V5.0.0.13 R1.13.0 for D3544-A1x (09/23/2022)Thanks that will help. As it enables identification if there's any issues, specific to that device. As well as its specific included CPU and its functions and features, especially its instruction set capabilities.
-
@john-c All such information should be available in the dmesg file in post: https://xcp-ng.org/forum/post/74791
Any ideas on how to revert the update? I would really like to have the setup running again. It may be "just" a home lab, but I was still using it (at least semi-) productively...
-
I'd be even fine to only use two machines and keep one of them offline for further testing.
-
For reference: I now decided to use a less intrusive approach and changed the default boot entry in grub config to the working failover entry. I will now try to get the pool up again.
-
What's the CPU on this? I would suspect a micro code update issue then.
-
-
@olivierlambert Following info from
/proc/cpuinfo:
Intel(R) Celeron(R) J4105 CPU @ 1.50GHzTrue enough, regarding the Wyse topic. I'll try reverting only the microcode update and see, what happens.
-
@RealTehreal said in Issue after latest host update:
Intel(R) Celeron(R) J4105 CPU @ 1.50GHz
Another Gemini Lake… So it's clearly related.
-
@olivierlambert Yep, I can confirm that in this case the microcode update is the culprit, too.
I just downgraded
microcode_ctl-2.1-26.xs28.1.xcpng8.2.x86_64
to
microcode_ctl-2.1-26.xs26.2.xcpng8.2.x86_64and it's working again. Man, what a mess.
-
@RealTehreal
Step-by-step instructions, in case, someone else has the same issue:1.:
yum history listto get the transaction id of the last update.2.:
yum history info #with # being the id from step 1, to list the updates done in this transaction. The interesting part for me wasUpdated microcode_ctl-2:2.1-26.xs26.2.xcpng8.2.x86_64 Update 2:2.1-26.xs28.1.xcpng8.2.x86_643.:
yum downgrade microcode_ctl-2:2.1-26.xs26.2.xcpng8.2.x86_64to downgrade to the previous version. You will have to enter the older version for this command.4.: Wait until it's done, reboot, test, pray it'll work again.
This is just a workaround! Microcode updates are important security and/or functional updates. Downgrading can lead to security issues.
-
@olivierlambert Thank you very much for pointing out the real issue.
