XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Boot Windows VM to check for system corruption

    Scheduled Pinned Locked Moved Xen Orchestra
    xoabackupwindows serverblack-screen
    18 Posts 4 Posters 996 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      DustinB
      last edited by

      I had to restore a VM to a backup made this morning ~ 6 hours ago, I'm trying to sort out what caused the system to essentially boot to a black screen (it will show the windows login screen, but upon entry, blackness).

      What would be the recommended way of determining if the guest OS is dealing with corruption and how could I fix it if restoring to ~6 hours ago wasn't an option.

      My assumption at the moment is a windows patch broke something, I'm more interested to know what would be the best way of looking at the underlying OS, before POST so I can check for any windows OS corruption issues using XOA.

      Is there a key combo to boot into the Windows Recovery mode within XO this easy? On a physical system I would normally hold down Shift+Power to get to this menu.

      PS also opening a ticket.

      1 Reply Last reply Reply Quote 0
      • D Offline
        DustinB
        last edited by

        These patches were installed at some point today

        KB5034770
        Servicing Stack 10.0.20348.2305
        KB5034613

        Nothing else on this system has changed.

        1 Reply Last reply Reply Quote 0
        • D Offline
          DustinB
          last edited by

          @DustinB said in Boot Windows VM to check for system corruption:

          KB5034770

          Seems to have been the root cause, I uninstalled this after managing to get the system to boot into safe mode and with a normal boot process, the system is back in working condition.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Hi!

            Do you know what the patch is about and how it could trigger the problem?

            J D 2 Replies Last reply Reply Quote 0
            • J Offline
              john.c @olivierlambert
              last edited by

              @olivierlambert said in Boot Windows VM to check for system corruption:

              Hi!

              Do you know what the patch is about and how it could trigger the problem?

              It's a security patch for Azure Stack HCI. It does a variety of things including an alteration (attempt at a fix) to an issue where networking data isn't returned the right way on VMS, involving remote direct memory access RDMA.

              It also attempts to fix memory leaks in ctfmon.exe and TextInputHost.exe

              https://support.microsoft.com/en-us/topic/february-13-2024-security-update-kb5034770-80671395-9f9c-4efa-8a8a-ccc0bd06e41e

              1 Reply Last reply Reply Quote 0
              • D Offline
                DustinB @olivierlambert
                last edited by

                @olivierlambert https://support.microsoft.com/en-us/topic/february-13-2024-kb5034770-os-build-20348-2322-6ac88bee-2b14-44c3-b8b4-b185259a27f5

                The complete break down from MS themselves, this was the only patch I removed from the system (going top down from most recent installed) and the first one happened to fix the issue.

                DanpD 1 Reply Last reply Reply Quote 0
                • DanpD Offline
                  Danp Pro Support Team @DustinB
                  last edited by

                  @DustinB Are you able to reproduce the issue by repeating the upgrade process?

                  D 3 Replies Last reply Reply Quote 0
                  • D Offline
                    DustinB @Danp
                    last edited by

                    @Danp said in Boot Windows VM to check for system corruption:

                    @DustinB Are you able to reproduce the issue by repeating the upgrade process?

                    I'll see if I can, I have to confirm the production system isn't reinstalled with it at the moment first though.

                    1 Reply Last reply Reply Quote 0
                    • D Offline
                      DustinB @Danp
                      last edited by

                      @Danp The patch is in C:\Windows\SoftwareDistribution\Download so I should be able to validate.

                      1 Reply Last reply Reply Quote 0
                      • D Offline
                        DustinB @Danp
                        last edited by

                        @Danp said in Boot Windows VM to check for system corruption:

                        @DustinB Are you able to reproduce the issue by repeating the upgrade process?

                        Just finalized the reinstallation, and the system is performing in similar fashion (black screen).

                        1 Reply Last reply Reply Quote 0
                        • DanpD Offline
                          Danp Pro Support Team
                          last edited by

                          FWIW, it could be localized to that VM (ie: some other conflict / corruption). I imagine there would be more news about this if it was happening to a bunch of other Windows 2022 servers. 😉

                          J 1 Reply Last reply Reply Quote 0
                          • J Offline
                            john.c @Danp
                            last edited by john.c

                            @Danp said in Boot Windows VM to check for system corruption:

                            FWIW, it could be localized to that VM (ie: some other conflict / corruption). I imagine there would be more news about this if it was happening to a bunch of other Windows 2022 servers. 😉

                            I have in the past found updates which by their very installation would corrupt the very state of the OS on which Windows is based. In this case it was during the Windows 8 and Windows 8.1 years and it was involving a Microsoft .Net Framework 3.5 and 4.x update for Windows 8 which was bad. It was bad as it corrupted the image (illuminated by trying to use sfc and dism to correct having found corruption) though then not being able to correct. It also prevented the installation of Microsoft .Net Framework 3.5 which a significant portion of software during that period used.

                            The only way to correct this was to remove the offending update, and black list it from re-installing. Followed by repairing the Windows installation, using sfc and dism.

                            Worst case scenario required a complete re-install or restoration of a backup image in order, to fix the corruption.

                            Eventually Microsoft pulled that bad update, but not before it became major headlines. I can't find the articles any more as they seem to have been removed from the Internet, but it was pretty major news as it was on all or almost all of the major news outlets (BBC's technology news section, etc).

                            D 1 Reply Last reply Reply Quote 1
                            • D Offline
                              DustinB @john.c
                              last edited by

                              @john-c That is what I'm thinking as well, there are some other factors with this server, and its really only used as for minimal workloads.

                              Of course, confirming where the issue stems from is paramount so that other workloads aren't impacted similarly.

                              1 Reply Last reply Reply Quote 0
                              • D Offline
                                DustinB
                                last edited by

                                Checkdisk found no issues (granted this was run online) also running a dism check for posterity.

                                J 1 Reply Last reply Reply Quote 0
                                • J Offline
                                  john.c @DustinB
                                  last edited by john.c

                                  @DustinB said in Boot Windows VM to check for system corruption:

                                  Checkdisk found no issues (granted this was run online) also running a dism check for posterity.

                                  Check disk (chkdsk) didn't at the time of the bad patch for me either, as the disk itself had no issues. Before the bad update was installed or after it was removed. Even during its presence installed.

                                  The only programs (commands) which revealed issues were sfc /scannow and dism /online /cleanup-image /restorehealth.

                                  D 1 Reply Last reply Reply Quote 1
                                  • D Offline
                                    DustinB @john.c
                                    last edited by

                                    @john-c will run an sfc next

                                    1 Reply Last reply Reply Quote 0
                                    • D Offline
                                      DustinB
                                      last edited by

                                      SFC did find and fix some issues, so fingers crossed going forward.

                                      D 1 Reply Last reply Reply Quote 0
                                      • D Offline
                                        DustinB @DustinB
                                        last edited by

                                        Even with a sfc and cleanup, manually reinstalling the package caused the same behavior. I've got backups of this system and will build a template from the working backup for if it goes sideways.

                                        I've got other stuff to handle.

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post