Boot Windows VM to check for system corruption
-
I had to restore a VM to a backup made this morning ~ 6 hours ago, I'm trying to sort out what caused the system to essentially boot to a black screen (it will show the windows login screen, but upon entry, blackness).
What would be the recommended way of determining if the guest OS is dealing with corruption and how could I fix it if restoring to ~6 hours ago wasn't an option.
My assumption at the moment is a windows patch broke something, I'm more interested to know what would be the best way of looking at the underlying OS, before POST so I can check for any windows OS corruption issues using XOA.
Is there a key combo to boot into the Windows Recovery mode within XO this easy? On a physical system I would normally hold down Shift+Power to get to this menu.
PS also opening a ticket.
-
These patches were installed at some point today
KB5034770
Servicing Stack 10.0.20348.2305
KB5034613Nothing else on this system has changed.
-
DustinB said in Boot Windows VM to check for system corruption:
KB5034770
Seems to have been the root cause, I uninstalled this after managing to get the system to boot into safe mode and with a normal boot process, the system is back in working condition.
-
Hi!
Do you know what the patch is about and how it could trigger the problem?
-
olivierlambert said in Boot Windows VM to check for system corruption:
Hi!
Do you know what the patch is about and how it could trigger the problem?
It's a security patch for Azure Stack HCI. It does a variety of things including an alteration (attempt at a fix) to an issue where networking data isn't returned the right way on VMS, involving remote direct memory access RDMA.
It also attempts to fix memory leaks in ctfmon.exe and TextInputHost.exe
-
olivierlambert https://support.microsoft.com/en-us/topic/february-13-2024-kb5034770-os-build-20348-2322-6ac88bee-2b14-44c3-b8b4-b185259a27f5
The complete break down from MS themselves, this was the only patch I removed from the system (going top down from most recent installed) and the first one happened to fix the issue.
-
DustinB Are you able to reproduce the issue by repeating the upgrade process?
-
Danp said in Boot Windows VM to check for system corruption:
DustinB Are you able to reproduce the issue by repeating the upgrade process?
I'll see if I can, I have to confirm the production system isn't reinstalled with it at the moment first though.
-
Danp The patch is in
C:\Windows\SoftwareDistribution\Downloadso I should be able to validate. -
Danp said in Boot Windows VM to check for system corruption:
DustinB Are you able to reproduce the issue by repeating the upgrade process?
Just finalized the reinstallation, and the system is performing in similar fashion (black screen).
-
FWIW, it could be localized to that VM (ie: some other conflict / corruption). I imagine there would be more news about this if it was happening to a bunch of other Windows 2022 servers.
-
Danp said in Boot Windows VM to check for system corruption:
FWIW, it could be localized to that VM (ie: some other conflict / corruption). I imagine there would be more news about this if it was happening to a bunch of other Windows 2022 servers.
I have in the past found updates which by their very installation would corrupt the very state of the OS on which Windows is based. In this case it was during the Windows 8 and Windows 8.1 years and it was involving a Microsoft .Net Framework 3.5 and 4.x update for Windows 8 which was bad. It was bad as it corrupted the image (illuminated by trying to use sfc and dism to correct having found corruption) though then not being able to correct. It also prevented the installation of Microsoft .Net Framework 3.5 which a significant portion of software during that period used.
The only way to correct this was to remove the offending update, and black list it from re-installing. Followed by repairing the Windows installation, using sfc and dism.
Worst case scenario required a complete re-install or restoration of a backup image in order, to fix the corruption.
Eventually Microsoft pulled that bad update, but not before it became major headlines. I can't find the articles any more as they seem to have been removed from the Internet, but it was pretty major news as it was on all or almost all of the major news outlets (BBC's technology news section, etc).
-
@john-c That is what I'm thinking as well, there are some other factors with this server, and its really only used as for minimal workloads.
Of course, confirming where the issue stems from is paramount so that other workloads aren't impacted similarly.
-
Checkdisk found no issues (granted this was run online) also running a dism check for posterity.
-
DustinB said in Boot Windows VM to check for system corruption:
Checkdisk found no issues (granted this was run online) also running a dism check for posterity.
Check disk (chkdsk) didn't at the time of the bad patch for me either, as the disk itself had no issues. Before the bad update was installed or after it was removed. Even during its presence installed.
The only programs (commands) which revealed issues were sfc /scannow and dism /online /cleanup-image /restorehealth.
-
@john-c will run an sfc next
-
SFC did find and fix some issues, so fingers crossed going forward.
-
Even with a sfc and cleanup, manually reinstalling the package caused the same behavior. I've got backups of this system and will build a template from the working backup for if it goes sideways.
I've got other stuff to handle.
