VM Failing to Reboot
-
@dinhngtu Thanks for the patch to XenBootFix.exe. I was able to run it on the VM, and I think it worked (sort of
), because now the VM isn't crashing anymore but still failing to boot completely into Windows (all I get is the spinning wheel). We're very close, I can feel it...lol.AFTER RUNNING XenBootFix-9.0.9024:
VM BOOT BEHAVIOR (No BSOD but Not Loading Windows either):
-
@kagbasi-ngc I think it's due to the Intel drivers acting up after being replaced by Xen drivers. I'll try to find a way to reproduce things on my end. Which Intel RST driver package did you install on the VM?
-
@dinhngtu Roger that, standing by. Thank you for all the assistance thus far. I know the easier approach would've been to simply revert the snapshot, however, I believe this process will prove beneficial for the project as a whole.
-
@kagbasi-ngc I'd like to have some information regarding your VM template; how did you prepare your template, and how were the Intel drivers included? What was installed before the VM started failing to boot? I tried installing several versions of Intel RST drivers but all of them were rejected at install time.
Edit: Does this work for you? https://nextcloud.vates.tech/index.php/s/yCTHF536JHTgoJL
-
@dinhngtu The VM Template was built using a hardened version of the Windows Server 2022 OS. It was hardened by our security team using the NSA Cyber Secure Host Baseline. For security reasons, I cannot share the ISO file with you, as that would constitute an export action.
I can't speak to the build process but I can confirm that much of what we do for hardening is simply applying the DISA STIGs (generally via Group Policies). For post installation, I generally grab missing drivers from OEM sites, but since this is a VM all I did was install the Citrix Guest Tools. Once that completed and I verified that the VM rebooted successfully, I ran Sysprep and shut it down, then converted the VM to a template.
Now what I'd installed just prior to the BSOD happening, was MailEnable. However, the BSOD only happened after I initiated a reboot by using the OS restart button. Prior to that, I'd been rebooting using the controls in XOA. I'd been rebooting the same VM without any issues. I had to activate the OS license, then I rebooted. Joined it to the domain; rebooted. Installed MailEnable; rebooted. All went well.
The BSOD happened after I attempted to enable the Active Directory integration in MailEnable. I checked the box to enable the integration, and it wasn't working. So I read in their documentation that the user account needed to have some local User Rights. So I modified this in Group Policy, ran
gpupdate, and initiated a reboot just to make sure the Group Policy took effect cleanly. That's when I ran into the BSOD.Honestly, I set those local User Rights all the time and have never run into a BSOD. I can't tell you how those Intel drivers got there; I certainly didn't install them. Hope this is a helpful response?
-
@kagbasi-ngc I can't tell why the VM failed to boot originally. However, having the Intel RST and Xen drivers installed at the same time made me think that as the Xen drivers were installed before Sysprep, once the Xen drivers stopped functioning, the IRST drivers were no longer able to find your Windows device path. You could try the following procedure:
- Boot into Windows PE
- Use
dism /image:C:\ /Get-Driversto find the published name ofiaStorAC.inf(oemxx.inf) - Use
dism /image:C:\ /Remove-Driver /Driver:oemxx.infto removeiaStorAC - You should be able to boot into Safe Mode. Rebooting will make things normal again.
As for why the drivers are there, they are likely present in the installation ISO you used.
-
@dinhngtu Cool, I'll try your suggestion and report back.
However, if you hunch holds, then I should be seeing this behavior on all my other VMs but I'm not. If you'll recall from a couple of days ago, you actually asked me to test this by building a new VM, installing MailEnable and seeing if the problem resurfaces - and it didn't.
Here's the video: https://photos.app.goo.gl/Uw7WgFRY1BEem8gA8
No worries, I'll report back my findings shortly.
-
@dinhngtu I see two instances of iaStorAC.inf. Should I remove both of them?
-
@kagbasi-ngc Yes, both.
-
@dinhngtu So I removed both drivers,
oem3.infandoem5.infsuccessfully. Unfortunately, the VM is still crashing with a BSOD ofINACCESSIBLE_BOOT_DEVICEwith Secure Boot on or off. -
@kagbasi-ngc That's expected since the boot storage driver is gone. You'll need to get into Safe Mode with the Recovery or F8 menu.
-
@dinhngtu I got into Safe Mode, then did nothing and just rebooted, result was still the same BSOD.
-
@kagbasi-ngc It worked on my VM when I tried that procedure on my local lab system. Unfortunately without a closer inspection I can't tell what's going on. If you still want to recover the VM, I'd start at removing all the non-inbox drivers.
-
@dinhngtu I’m not sure I understand what you mean by
non-inbox drivers.I will talk with our ISSO to see if sending an export of the VM is allowable. If not, would you be interested in a video call where we can troubleshoot in realtime?
-
@kagbasi-ngc I think for that it'd be more appropriate to contact our support team, which will be able to help you directly on your infrastructure. I'd also like to keep any troubleshooting information on the forum in case someone runs into a similar problem.
Non-inbox drivers mean drivers with "Inbox : No" as seen in your screenshot.
-
@dinhngtu Roger that, agreed.
I just got back to the lab, so I'm gonna try and remove those
non-inbox driversand see what happens. -
@dinhngtu Unfortunately, I don't have good news. I removed all the
non-inbox drivers, one by one (rebooting after removing each one), yet still the VM is crashing with the same BSOD message. -
@kagbasi-ngc Could the VM get into Safe Mode?
-
@dinhngtu Yes, it does. There is no
Last Known Good Stateoption however. -
@kagbasi-ngc The fact that your VM still boots in Safe Mode means that there's still some drivers blocking Windows from booting in normal mode. Please enable boot logging by running
bcdedit /store bcd /set {default} bootlog yesthen post the boot logs of normal mode versus safe mode. This log is found atC:\Windows\ntbtlog.txt.Could you send another copy of your SYSTEM hive?