VM Failing to Reboot

dinhngtu

@kagbasi-ngc Try this method: https://xcp-ng.org/forum/post/85097

kagbasi-ngc

@dinhngtu Thanks, that helped. Download has started, once it completes, I'll get it installed and retry.

I'd also found this - https://gist.github.com/alexrintt/9502728c6d75d27e2885c9cdaab0e550

kagbasi-ngc

@dinhngtu Thanks for your guidance, I was able to get the new version of Windbg installed and successfully made a connection to the VM. Below is the output that was captured.

I did notice that while the debugger was connected and running, the VM would not BSOD or proceed any further than the splash screen (picture below). I left it running and will go check on it after I get out of my meeting, and report back.

Windbg Output:

************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   NonInteractiveNuget : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true
   EnableRedirectToChakraJsProvider : false

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.031 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 42

Microsoft (R) Windows Debugger Version 10.0.27725.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Opened 10.0.10.12
Waiting to reconnect...
Connected to Windows 10 20348 x64 target at (Wed Jan 22 09:28:50.709 2025 (UTC - 5:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*
Executable search path is: 
Windows 10 Kernel Version 20348 MP (1 procs) Free x64
Edition build lab: 20348.859.amd64fre.fe_release_svc_prod2.220707-1832
Kernel base = 0xfffff806`0b600000 PsLoadedModuleList = 0xfffff806`0c233d20
System Uptime: 0 days 0:00:00.000
Unable to load image ntkrnlmp.exe, Win32 error 0n2
KDTARGET: Refreshing KD connection

*** Fatal System Error: 0x0000007b
                       (0xFFFF8005512071F8,0xFFFFFFFFC0000034,0x0000000000000000,0x0000000000000001)

Break instruction exception - code 80000003 (first chance)

A fatal system error has occurred.
Debugger entered on first try; Bugcheck callbacks have not been invoked.

A fatal system error has occurred.

For analysis of this file, run !analyze -v
nt!DbgBreakPointWithStatus:
fffff806`0ba29560 cc              int     3

dinhngtu

@kagbasi-ngc That's pretty odd, you're still getting INACCESSIBLE_BOOT_DEVICE. Did XenBootFix finish successfully? If !analyze -v runs for too long, interrupt it with Ctrl+Break then run kb; !devnode 0 1

kagbasi-ngc

@dinhngtu Wait, was I supposed to run !analyze -v? I simply kicked of the debugger and just let things be.

Yes, XenBootFix.exe ran successfully and displayed a list of all the device drivers it disabled, however, since I haven't been able to get into the OS I haven't been able to run the Cleanup script.

kagbasi-ngc

@dinhngtu I went back and clicked on !analyze -v within the debugger and it spit out the following (log attached below). Please pardon me, as I'm not a developer and have never used this tool before, so didn't realize the boot process had actually been paused, awaiting my input from the debugger.

Also, based on the output looks like my WinDbg setup isn't complete. Something about missing symbols. Googling how to get that setup now....

PS. The VM is still sitting at the splash screen (showing the Windows logo).

MAIL02-Windbg_Log.txt

The contents of the debugger output exceeds the character limitation of this post, so I attached the file instead.

dinhngtu

@kagbasi-ngc The symbols require an Internet connection on your WinDbg machine.

@kagbasi-ngc said in VM Failing to Reboot:

PS. The VM is still sitting at the splash screen (showing the Windows logo).

It's normal when kernel debugging with WinDbg. You can resume the VM by clicking Go.

Nevertheless, XenBootFix should have gotten rid of INACCESSIBLE_BOOT_DEVICE. If it's not sensitive, can you send me the output of !devnode 0 1 and the contents of your Registry at C:\Windows\System32\config\SYSTEM?

kagbasi-ngc

@dinhngtu Of course, I can do that. I will be heading to back to the lab in about an hour.

I'm currently at home trying to figure out how to cache the symbols on another computer with Internet access and then transfer them to the offline PC where WinDbg is running at.

dinhngtu

@kagbasi-ngc Normally you can't cache them, since the symbols depend on your exact Windows and updates version. The analysis output shouldn't be too important for INACCESSIBLE_BOOT_DEVICE.

kagbasi-ngc

@dinhngtu Okay, I'll stop pursuing that then and head to the lab. Should take me about 30 minutes to get there. Thanks again.

kagbasi-ngc

@dinhngtu said in VM Failing to Reboot:

can you send me the output of !devnode 0 1 ..?

OUTPUT of !devnode 0 1

1: kd> !devnode 0 1
Error retrieving address of IopRootDeviceNode

CONTENTS of C:\Windows\System32\config\SYSTEM
I'm still trying to figure out how I'm going to do this for the VM (since it won't boot into the OS). I tried attaching the disk to another VM, but XOA won't allow it to connect. Keeps throwing this error vbd.connect {"code": "DISK_VBD_MUST_BE_READWRITE_FOR_HVM"}

dinhngtu

@kagbasi-ngc I forgot that the devnode command doesn't work since there's no symbols.

For the config/SYSTEM file, can you copy it outside with Hiren's boot?

kagbasi-ngc

@dinhngtu Yeah, I'm trying Hiren's now. Standby.

kagbasi-ngc

@dinhngtu I managed to export the actual SYSTEM hive file. I tried to open it in the Registry Editor so I can export it to a .reg file, but it was proving challenging from within Hiren's Boot Disc.

Since I am unable to attach it to this post, can you send me a Nextcloud upload link that I can drop the files in? You can direct message me the link.

I hope this will be helpful? I can hang around for a little while longer, if you want me to.

dinhngtu

@kagbasi-ngc Okay, you have the Intel storage drivers installed on the VM so XenBootFix did not work correctly. Here's a new version of XenBootFix that should fix your VM: https://nextcloud.vates.tech/index.php/s/C784bGgbqWZDrki

kagbasi-ngc

@dinhngtu Thank you, I got the file. Should be at work in a couple of hours and will try it.

kagbasi-ngc

@dinhngtu Thanks for the patch to XenBootFix.exe. I was able to run it on the VM, and I think it worked (sort of ), because now the VM isn't crashing anymore but still failing to boot completely into Windows (all I get is the spinning wheel). We're very close, I can feel it...lol.

AFTER RUNNING XenBootFix-9.0.9024:

VM BOOT BEHAVIOR (No BSOD but Not Loading Windows either):

dinhngtu

@kagbasi-ngc I think it's due to the Intel drivers acting up after being replaced by Xen drivers. I'll try to find a way to reproduce things on my end. Which Intel RST driver package did you install on the VM?

kagbasi-ngc

@dinhngtu Roger that, standing by. Thank you for all the assistance thus far. I know the easier approach would've been to simply revert the snapshot, however, I believe this process will prove beneficial for the project as a whole.

dinhngtu

@kagbasi-ngc I'd like to have some information regarding your VM template; how did you prepare your template, and how were the Intel drivers included? What was installed before the VM started failing to boot? I tried installing several versions of Intel RST drivers but all of them were rejected at install time.

Edit: Does this work for you? https://nextcloud.vates.tech/index.php/s/yCTHF536JHTgoJL