"CROSSTalk" CPU vulnerabilty (cross-core data leak)

stormi

Yeah the performance impact is very dependent on the kind of workload.

stormi

As part of this update candidate, there is now a kernel update to lower the performance impact of the microcode update.

We only have the testing results from one user for now. Can other users spare some hardware and time today?

stormi

Update published: https://xcp-ng.org/blog/2020/06/12/intel-microcode-security-update-crosstalk/

demanzke

Hi stormi,

I know I am a bit late to report but I just updated my XCP-ng install from the main repo. I'm using an i3-7350K that shouldn't be vulnerable according to Intels list you posted.
After the update default boot settings don't work, the loading screen stalls for a long time then prints a bunch of messages containing "dracut-initqueue timeout - starting timing scripts" followed by "could not boot" and stops there (I don't have the exact wording as I don't have access to the system right now).
If I select safe boot in GRUB there is a kernel panic during boot "couldn't enable IOMMU and iommu=required/forced".
Selecting the 4.19.0 kernel during boot works as usual.

Is there anything else I could try?

stormi

@demanzke thanks for the report.

To be sure I understand, is it like described on the following picture?

The XCP-ng (Xen 4.13.0 / Linux 4.19.0+1) uses the Xen and linux Kernel version from the last ISO installation or upgrade.

A screenshot from the failed boot could be useful. I think the "dracut-initque timeout" is usually followed by some information about what failed. Could you also run xen-bugtool -y from the booted host, upload the resulting tar.gz somewhere and send me the link in a private message?

It looks like it's a kernel issue, but since the boot option that works for you also reverts Xen to a previous version, the way to be sure would be to downgrade the kernel and then boot again:

# yum downgrade won't work for the kernel because it's a protected package, so let's use rpm
yumdownloader kernel-4.19.19-6.0.10.1.xcpng8.1
rpm -Uv --oldpackage kernel-4.19.19-6.0.10.1.xcpng8.1.x86_64.rpm

demanzke

It is the "Safe Mode" options that results in a kernel panic, not "Serial".
I will grab the screenshots and bugtool logs and test the different kernel later today.

stormi

@demanzke fixed the picture

stormi

Do you get an emergency shell after dracut-initqueue timeout? If yes, there are probably logs that you can read from the current filesystem (which is in RAM at this stage of the boot process so probably disappears afterwards).

demanzke

There is no emergency shell after the failed boot, sadly.

This is what happens after the loading bar on default settings:

Right after selecting "Safe Boot":

Installing the suggested kernel 6.0.10 changed nothing. Should I try downgrading other packages or an even older kernel version?

stormi

@demanzke said in "CROSSTalk" CPU vulnerabilty (cross-core data leak):

Installing the suggested kernel 6.0.10 changed nothing. Should I try downgrading other packages or an even older kernel version?

Yes, please try:

yum downgrade xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools

Then if it still changes nothing:

yum downgrade microcode_ctl

After all this, you'll be theoretically back to the state from before the update... Though there may be an issue with the initrd generation, which would still not allow you to boot.

demanzke

@stormi said in "CROSSTalk" CPU vulnerabilty (cross-core data leak):

Yes, please try:
yum downgrade xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools
Then if it still changes nothing:
yum downgrade microcode_ctl
After all this, you'll be theoretically back to the state from before the update... Though there may be an issue with the initrd generation, which would still not allow you to boot.

Sadly nothing changed after downgrading the packages. The only thing I have changed after the base install was installing your ZFS port.
At this point I would try a fresh install on the weekend and see if the problem reappears unless you have another suggestion.

stormi

If we want to understand fully what happens, we could compare the contents of the initial ramdisks:

initrd-4.19.0+1.img => doesn't work anymore
initrd-fallback.img => still works

One can extract them with:

mkdir initrd-current
cd initrd-current/
/usr/lib/dracut/skipcpio /boot/initrd-4.19.0+1.img | zcat | cpio -ivd
cd ..
mkdir initrd-fallback
cd initrd-fallback/
/usr/lib/dracut/skipcpio /boot/initrd-fallback.img | zcat | cpio -ivd

I don't know what differences to look for, to be honest. Maybe you could save those files and upload them somewhere for anyone interested to look at?

Reinstalling the host then trying the update again, without ZFS first, then with it (which probably means reinstalling again and redoing the steps), could also be interesting to help precisely understand what happens.

For now, it mainly looks like it's related to the initrd, which is generated by dracut when the kernel or other kernel modules (such as the kernel module for ZFS) are installed. As you may know, the initrd is the initial ramdisk which contains a minimal system booted before the actual system and which must be able to mount your root filesystem to be able to continue. Unfortunately we don't know from the output you get what the error is so it's all conjectures.

Biggen

When this Crosstalk microcode update hit last week there was an issue with certain Intel CPUs where we coudn't boot after the patch was applied. I run Linux Mint on my laptop and I couldn't boot it after taking the microcode update. I had to boot into recovery and then apt remove intel-microcode to get it back to a working state. Later that day, Ubuntu (or whoever) released a new intel-microcode update that corrected the problem.

Not sure if this is even remotely close to the same issue but wanted to put this out there.

Danp

Has anyone else encountered this issue? Wondering if these patches should be pulled until this gets resolved.

stormi

As far as I know, those patches work well on Citrix' test hosts. They also work well on our hosts at Vates. The microcodes underwent Intel's QA so I don't expect them to break on the vast majority of hardware, though there are reports of issues with some specific models. In @demanzke's case, reverting to the previous microcode did not fix the issue so at first it doesn't look like it's related to the microcode.

stormi

Intel just released updated microcode (actually it's a revert) for some models: https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/releases

I'll update the microcode_ctl package. ~~The "older" microcode that is used instead is still recent enough to contain the fixes against CROSSTalk / SRBDS.~~ Or so I had understood, but I can't find evidence about it.

demanzke

Thanks @Biggen and @stormi
I'll try updating then removing the microcode_ctl package tomorrow and share the results.

markxc

Hi do i need to patch my xenserver using AMD EPYC ? Those patches get offered to my AMD nodes by XO.
On intel Xeon nodes it makes sense to me ....

olivierlambert

I would say: always apply patches, but you are free to reboot when you want. Obviously, for you, it won't change anything (no microcode update) but keeping your hosts up to date is a good practice

lefty

@stormi said in "CROSSTalk" CPU vulnerabilty (cross-core data leak):

Intel just released updated microcode (actually it's a revert) for some models: https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/releases

I'll update the microcode_ctl package. ~~The "older" microcode that is used instead is still recent enough to contain the fixes against CROSSTalk / SRBDS.~~ Or so I had understood, but I can't find evidence about it.

So should I wait applying these updates? You seem to be unsure of which microcode version to distribute.