@mgigirey said in Issue after latest host update:
@andyhhp Any plans to update the intel-microcode for XCP-ng 8.3? latest know version working in my setup is intel-microcode-20231009-1.xcpng8.3.noarch.rpm
I am not an XCP-ng developer. You'll have to ask @stormi for that.
@eb-xcp said in XCP-ng 8.3 betas and RCs feedback 
:
Edit: Confirmed; after enabling execution disable option within bios, installer booted without issues and the install is currently ongoing.
That is a bug. Xen is supposed to be able to detect this case and re-activate NX on it's own.
For the EFI path in your screenshot, that one doesn't have logic to re-activate. IIRC, we weren't sure whether it was needed, because surely an EFI system wasn't still using Pentium4 compatibility. Clearly some wrong reasoning, and it's fairly easy to adjust.
However, fixing that path wont fix the normal MB2 path, which does have logic to reactivate and should have been able to cope fine.
What system do you have?
@flakpyro If Singlewire have already fixed the bug, then just do what is is necessary to update the VM and be done with it.
That screenshot of grub poses far more questions than it answered, and I doubt we want to get into any of them.
This is ultimately a bug in Linux. There was a range of Linux kernels which did something unsafe on kexec which worked most of the time but only by luck. (Specifically - holding a 64bit value in a register while passing through 32bit mode, and expecting it to still be intact later; both Intel and AMD identify this as having model specific behaviour and not to rely on it).
A consequence of a security fix in Xen (https://xenbits.xen.org/xsa/advisory-454.html) makes it reliably fail when depended upon in a VM.
Linux fixed the bug years ago, but one distro managed to pick it up.
Ideally, get SingleWire to fix their kernel. Failing that, adjust the VM's kernel command line to take any ,low or ,high off the crashkernel= line, because that was the underlying way to tickle the bug IIRC.
The property you need to end up with is that /proc/iomem shows the Crash kernel range being below the 4G boundary, because the handover logic from one kernel to the other simply didn't work correctly if the new kernel was above 4G.
Intel Xeon E5-2683 v4 CPUs vs E5-2697 v4 CPUs
You are correct. These are adjacent rows in the SKU table; they've got the same core count, and only differ by 500MHz frequency. They're basically identical as far as software is concerned.
I suggest using this as a learning opportunity. Look at the RPM log and see what depends on busybox, and therefore what (else) got uninstalled in order to keep the dependencies satisfied.
(Hint: you uninstalled all of Xapi, hence why nothing works)
@rubberhose I've got a fix from Intel, and @stormi has packaged it.
yum update microcode_ctl --enablerepo=xcp-ng-testing should get you microcode_ctl-2.1-26.xs29.2.xcpng8.2 which has the fixed microcode for this issue in it.
When you've got that installed, it should be safe to update back to the latest firmware.
@t-chamberlain I've got a fix from Intel, and @stormi has packaged it.
yum update microcode_ctl --enablerepo=xcp-ng-testing should get you microcode_ctl-2.1-26.xs29.2.xcpng8.2 which has the fixed microcode for this issue in it.
@RealTehreal I've got a fix from Intel, and @stormi has packaged it.
yum update microcode_ctl --enablerepo=xcp-ng-testing should get you microcode_ctl-2.1-26.xs29.2.xcpng8.2 which has the fixed microcode for this issue in it.
@RealTehreal Thank-you very much for that information. I'll follow up with Intel.
In the short term, I'd recommend just using the old microcode.
@RealTehreal Sorry to keep adding to the list of diagnostics, but everything here will help. After you've tried the other options, could you try this:
If the XTF testing shows any XTF test looping, use that single test, otherwise use your regular VM. Get one VM into the looping state. Check xl list to confirm that you've only got Domain-0 and the one other VM, and note it's domid (the "ID" column).
In dom0, run xentrace to capture a system trace. It's looping so the dump file is going to be large, but it also means that you can CTRL-C as quickly as you can on the shell and it will be fine (a few hundred milliseconds of samples will almost certainly be enough).
Anyway, run xentrace -D -e 0x0008f000 xentrace.dmp and then give me created xentrace.dmp file. If you're interested in what's in it, you can decode it using xenalyze -a xentrace.dmp |& less.
Then, run xen-hvmctx $domid two or three times, and share the output of all.
@t-chamberlain In addition to the XTF testing, could you also please (with the bad microcode) try booting Xen with spec-ctrl=no-verw on the command line, and seeing whether that changes the behaviour of your regular VMs? Please capture xl dmesg from this run too.
@RealTehreal In addition to the XTF testing, could you also please try (with the bad microcode) booting Xen with spec-ctrl=no-verw on the command line, and seeing whether that changes the behaviour of your regular VMs? Please capture xl dmesg from this run too.
@RealTehreal It's an Intel issue, but while this is enough to show that there is an issue, it's not enough to figure out what is wrong.
Sadly, a VM falling into a busy loop can be one of many things. It's clearly on the (v)BSP prior to starting (v)APs, hence why it's only ever a single CPU spinning.
Can you switch to using the debug hypervisor (change the /boot/xen.gz symlink to point at the -d suffixed hypervisor), and then capture xl dmesg after trying to boot one VM. Depending on how broken things are, we might see some diagnostics.
Could you also try running xtf as described here: https://xcp-ng.org/forum/post/57804 It's a long-shot, but if it does happen to stumble on the issue, then it will be orders of magnitude easier to debug than something misc broken in the middle of OVMF.
Windows isn't going to be tricked into being happy about the CPU just by changing the reported model. It cross-checks real features, and you simply can't fake those up.
There is no ability in XenServer/XCP-ng to configure this, and I have no intention to offer people the ability to shoot themselves in the foot like this.
@Andrew said in Xen 4.17 on XCP-ng 8.3!:
xtf HARD system freeze at test-hvm64-xsa-304. (only XCP hard lockup I have seen)
xtf With ept=no-exec-sp, all tests SKIP/SUCCESS.
It's guest exploitable, and locks up the CPU so hard it doesn't even reset properly. It's also very expensive to work around, hence why it's not mitigated by default.
So, we've had reports on xen-devel which look a little like this.
@BlueBadger are you able to switch back to your 7950x and try booting Xen with x2apic_phys=true ? It appears that the -X processors are missing a feature in their IOMMU and Xen was getting confused when setting up interrupt handling.
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=0d2686f6b66b4b1b3c72c3525083b0ce02830054 is at least part of the fix, but so far feedback on the mailing lists suggests it's not a complete fix.
This is way way outside of a normal-ish looking server usecase. I'm honestly surprised you've got anything to function...
To start with, you're probably booting Xen with console=vga (because that's the default). It will be handed over to dom0 too, so start by going through the bootloader configuration and making sure that neither Xen nor dom0 are trying to use the display at all.
I suspect this is the root cause of the display going periodically back to black.
You cannot mix Intel and AMD CPUs in the same pool. They're simply not compatible enough for a VM to survive a migration between the two, so we explicitly disallow it. (You can't even --force a pool join to this effect.)
@Andrew Intel E5450, that's very retro.
It's also first-gen VT-x and doesn't have HAP, which is why the test that is looking explicitly for HAP doesn't work.
As a stopgap, remove hap from the VARY-CFG := hap shadow line in tests/invlpg/Makefile and rebuild. In the meantime I'll try to figure out a nice way to cope with this.