XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    "CROSSTalk" CPU vulnerabilty (cross-core data leak)

    Scheduled Pinned Locked Moved News
    29 Posts 8 Posters 6.4k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      demanzke
      last edited by

      It is the "Safe Mode" options that results in a kernel panic, not "Serial".
      I will grab the screenshots and bugtool logs and test the different kernel later today.

      stormiS 1 Reply Last reply Reply Quote 0
      • stormiS Offline
        stormi Vates 🪐 XCP-ng Team @demanzke
        last edited by

        @demanzke fixed the picture 🙂

        1 Reply Last reply Reply Quote 0
        • stormiS Offline
          stormi Vates 🪐 XCP-ng Team
          last edited by

          Do you get an emergency shell after dracut-initqueue timeout? If yes, there are probably logs that you can read from the current filesystem (which is in RAM at this stage of the boot process so probably disappears afterwards).

          1 Reply Last reply Reply Quote 0
          • D Offline
            demanzke
            last edited by

            There is no emergency shell after the failed boot, sadly.

            This is what happens after the loading bar on default settings:
            IMG_20200615_192408.jpg

            Right after selecting "Safe Boot":
            IMG_20200615_192531.jpg

            Installing the suggested kernel 6.0.10 changed nothing. Should I try downgrading other packages or an even older kernel version?

            stormiS 1 Reply Last reply Reply Quote 0
            • stormiS Offline
              stormi Vates 🪐 XCP-ng Team @demanzke
              last edited by

              @demanzke said in "CROSSTalk" CPU vulnerabilty (cross-core data leak):

              Installing the suggested kernel 6.0.10 changed nothing. Should I try downgrading other packages or an even older kernel version?

              Yes, please try:

              yum downgrade xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools
              

              Then if it still changes nothing:

              yum downgrade microcode_ctl
              

              After all this, you'll be theoretically back to the state from before the update... Though there may be an issue with the initrd generation, which would still not allow you to boot.

              D 1 Reply Last reply Reply Quote 0
              • D Offline
                demanzke @stormi
                last edited by

                @stormi said in "CROSSTalk" CPU vulnerabilty (cross-core data leak):

                Yes, please try:

                yum downgrade xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools
                

                Then if it still changes nothing:

                yum downgrade microcode_ctl
                

                After all this, you'll be theoretically back to the state from before the update... Though there may be an issue with the initrd generation, which would still not allow you to boot.

                Sadly nothing changed after downgrading the packages. The only thing I have changed after the base install was installing your ZFS port.
                At this point I would try a fresh install on the weekend and see if the problem reappears unless you have another suggestion.

                1 Reply Last reply Reply Quote 0
                • stormiS Offline
                  stormi Vates 🪐 XCP-ng Team
                  last edited by

                  If we want to understand fully what happens, we could compare the contents of the initial ramdisks:

                  • initrd-4.19.0+1.img => doesn't work anymore
                  • initrd-fallback.img => still works

                  One can extract them with:

                  mkdir initrd-current
                  cd initrd-current/
                  /usr/lib/dracut/skipcpio /boot/initrd-4.19.0+1.img | zcat | cpio -ivd
                  cd ..
                  mkdir initrd-fallback
                  cd initrd-fallback/
                  /usr/lib/dracut/skipcpio /boot/initrd-fallback.img | zcat | cpio -ivd
                  

                  I don't know what differences to look for, to be honest. Maybe you could save those files and upload them somewhere for anyone interested to look at?

                  Reinstalling the host then trying the update again, without ZFS first, then with it (which probably means reinstalling again and redoing the steps), could also be interesting to help precisely understand what happens.

                  For now, it mainly looks like it's related to the initrd, which is generated by dracut when the kernel or other kernel modules (such as the kernel module for ZFS) are installed. As you may know, the initrd is the initial ramdisk which contains a minimal system booted before the actual system and which must be able to mount your root filesystem to be able to continue. Unfortunately we don't know from the output you get what the error is so it's all conjectures.

                  1 Reply Last reply Reply Quote 0
                  • B Offline
                    Biggen
                    last edited by

                    When this Crosstalk microcode update hit last week there was an issue with certain Intel CPUs where we coudn't boot after the patch was applied. I run Linux Mint on my laptop and I couldn't boot it after taking the microcode update. I had to boot into recovery and then apt remove intel-microcode to get it back to a working state. Later that day, Ubuntu (or whoever) released a new intel-microcode update that corrected the problem.

                    Not sure if this is even remotely close to the same issue but wanted to put this out there.

                    1 Reply Last reply Reply Quote 1
                    • DanpD Offline
                      Danp Pro Support Team
                      last edited by

                      Has anyone else encountered this issue? Wondering if these patches should be pulled until this gets resolved.

                      1 Reply Last reply Reply Quote 0
                      • stormiS Offline
                        stormi Vates 🪐 XCP-ng Team
                        last edited by stormi

                        As far as I know, those patches work well on Citrix' test hosts. They also work well on our hosts at Vates. The microcodes underwent Intel's QA so I don't expect them to break on the vast majority of hardware, though there are reports of issues with some specific models. In @demanzke's case, reverting to the previous microcode did not fix the issue so at first it doesn't look like it's related to the microcode.

                        1 Reply Last reply Reply Quote 0
                        • stormiS Offline
                          stormi Vates 🪐 XCP-ng Team
                          last edited by stormi

                          Intel just released updated microcode (actually it's a revert) for some models: https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/releases

                          I'll update the microcode_ctl package. The "older" microcode that is used instead is still recent enough to contain the fixes against CROSSTalk / SRBDS. Or so I had understood, but I can't find evidence about it.

                          L 1 Reply Last reply Reply Quote 1
                          • D Offline
                            demanzke
                            last edited by

                            Thanks @Biggen and @stormi
                            I'll try updating then removing the microcode_ctl package tomorrow and share the results.

                            1 Reply Last reply Reply Quote 0
                            • M Offline
                              markxc
                              last edited by

                              Hi do i need to patch my xenserver using AMD EPYC ? Those patches get offered to my AMD nodes by XO.
                              On intel Xeon nodes it makes sense to me ....

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                I would say: always apply patches, but you are free to reboot when you want. Obviously, for you, it won't change anything (no microcode update) but keeping your hosts up to date is a good practice 🙂

                                1 Reply Last reply Reply Quote 1
                                • L Offline
                                  lefty @stormi
                                  last edited by

                                  @stormi said in "CROSSTalk" CPU vulnerabilty (cross-core data leak):

                                  Intel just released updated microcode (actually it's a revert) for some models: https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/releases

                                  I'll update the microcode_ctl package. The "older" microcode that is used instead is still recent enough to contain the fixes against CROSSTalk / SRBDS. Or so I had understood, but I can't find evidence about it.

                                  So should I wait applying these updates? You seem to be unsure of which microcode version to distribute.

                                  1 Reply Last reply Reply Quote 0
                                  • stormiS Offline
                                    stormi Vates 🪐 XCP-ng Team
                                    last edited by

                                    I'm unsure for Skylake. Not for other CPUs.

                                    1 Reply Last reply Reply Quote 0
                                    • L Offline
                                      lefty
                                      last edited by

                                      Thanks for the clarification. No Skylake present, so I will proceed.

                                      1 Reply Last reply Reply Quote 0
                                      • D Offline
                                        demanzke
                                        last edited by demanzke

                                        Finally got some time to test your suggestions.
                                        Removing the microcode_ctl package without dependencies did not help.
                                        Here are both initial ramdisks for anyone interested to look at.

                                        Reinstalling XCP, then ZFS, then updating all packages worked fine.

                                        stormiS 1 Reply Last reply Reply Quote 0
                                        • stormiS Offline
                                          stormi Vates 🪐 XCP-ng Team @demanzke
                                          last edited by

                                          @demanzke So this time no boot issue after installing the update?

                                          D 1 Reply Last reply Reply Quote 0
                                          • D Offline
                                            demanzke @stormi
                                            last edited by

                                            @stormi Exactly. Must've been related to something other than just the latest packages.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post