XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    VM Failing to Reboot

    Scheduled Pinned Locked Moved Unsolved Management
    92 Posts 4 Posters 3.2k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K Offline
      kagbasi-ngc @dinhngtu
      last edited by

      dinhngtu Of course, I can do that. I will be heading to back to the lab in about an hour.

      I'm currently at home trying to figure out how to cache the symbols on another computer with Internet access and then transfer them to the offline PC where WinDbg is running at.

      D 1 Reply Last reply Reply Quote 0
      • D Offline
        dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
        last edited by

        kagbasi-ngc Normally you can't cache them, since the symbols depend on your exact Windows and updates version. The analysis output shouldn't be too important for INACCESSIBLE_BOOT_DEVICE.

        K 1 Reply Last reply Reply Quote 0
        • K Offline
          kagbasi-ngc @dinhngtu
          last edited by

          dinhngtu Okay, I'll stop pursuing that then and head to the lab. Should take me about 30 minutes to get there. Thanks again.

          1 Reply Last reply Reply Quote 0
          • K Offline
            kagbasi-ngc @dinhngtu
            last edited by

            dinhngtu said in VM Failing to Reboot:

            can you send me the output of !devnode 0 1 ..?

            OUTPUT of !devnode 0 1

            1: kd> !devnode 0 1
            Error retrieving address of IopRootDeviceNode
            

            CONTENTS of C:\Windows\System32\config\SYSTEM
            I'm still trying to figure out how I'm going to do this for the VM (since it won't boot into the OS). I tried attaching the disk to another VM, but XOA won't allow it to connect. Keeps throwing this error vbd.connect {"code": "DISK_VBD_MUST_BE_READWRITE_FOR_HVM"}

            D 1 Reply Last reply Reply Quote 0
            • D Offline
              dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
              last edited by

              kagbasi-ngc I forgot that the devnode command doesn't work since there's no symbols.

              For the config/SYSTEM file, can you copy it outside with Hiren's boot?

              K 1 Reply Last reply Reply Quote 0
              • K Offline
                kagbasi-ngc @dinhngtu
                last edited by

                dinhngtu Yeah, I'm trying Hiren's now. Standby.

                K 1 Reply Last reply Reply Quote 0
                • K Offline
                  kagbasi-ngc @kagbasi-ngc
                  last edited by

                  dinhngtu I managed to export the actual SYSTEM hive file. I tried to open it in the Registry Editor so I can export it to a .reg file, but it was proving challenging from within Hiren's Boot Disc.

                  Since I am unable to attach it to this post, can you send me a Nextcloud upload link that I can drop the files in? You can direct message me the link.

                  I hope this will be helpful? I can hang around for a little while longer, if you want me to.

                  D 1 Reply Last reply Reply Quote 0
                  • D Offline
                    dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
                    last edited by

                    kagbasi-ngc Okay, you have the Intel storage drivers installed on the VM so XenBootFix did not work correctly. Here's a new version of XenBootFix that should fix your VM: https://nextcloud.vates.tech/index.php/s/C784bGgbqWZDrki

                    K 1 Reply Last reply Reply Quote 0
                    • K Offline
                      kagbasi-ngc @dinhngtu
                      last edited by

                      dinhngtu Thank you, I got the file. Should be at work in a couple of hours and will try it.

                      K 1 Reply Last reply Reply Quote 0
                      • K Offline
                        kagbasi-ngc @kagbasi-ngc
                        last edited by

                        dinhngtu Thanks for the patch to XenBootFix.exe. I was able to run it on the VM, and I think it worked (sort of 😁 ), because now the VM isn't crashing anymore but still failing to boot completely into Windows (all I get is the spinning wheel). We're very close, I can feel it...lol.

                        AFTER RUNNING XenBootFix-9.0.9024:

                        XenBootFix_Patch_Run_1.PNG

                        VM BOOT BEHAVIOR (No BSOD but Not Loading Windows either):

                        XenBootFix_Patch_Run_2_VM-Still-Not-Booting.PNG

                        D 1 Reply Last reply Reply Quote 0
                        • D Offline
                          dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
                          last edited by dinhngtu

                          kagbasi-ngc I think it's due to the Intel drivers acting up after being replaced by Xen drivers. I'll try to find a way to reproduce things on my end. Which Intel RST driver package did you install on the VM?

                          K 1 Reply Last reply Reply Quote 0
                          • K Offline
                            kagbasi-ngc @dinhngtu
                            last edited by

                            dinhngtu Roger that, standing by. Thank you for all the assistance thus far. I know the easier approach would've been to simply revert the snapshot, however, I believe this process will prove beneficial for the project as a whole.

                            D 1 Reply Last reply Reply Quote 0
                            • D Offline
                              dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
                              last edited by dinhngtu

                              kagbasi-ngc I'd like to have some information regarding your VM template; how did you prepare your template, and how were the Intel drivers included? What was installed before the VM started failing to boot? I tried installing several versions of Intel RST drivers but all of them were rejected at install time.

                              Edit: Does this work for you? https://nextcloud.vates.tech/index.php/s/yCTHF536JHTgoJL

                              K 1 Reply Last reply Reply Quote 0
                              • K Offline
                                kagbasi-ngc @dinhngtu
                                last edited by

                                dinhngtu The VM Template was built using a hardened version of the Windows Server 2022 OS. It was hardened by our security team using the NSA Cyber Secure Host Baseline. For security reasons, I cannot share the ISO file with you, as that would constitute an export action.

                                I can't speak to the build process but I can confirm that much of what we do for hardening is simply applying the DISA STIGs (generally via Group Policies). For post installation, I generally grab missing drivers from OEM sites, but since this is a VM all I did was install the Citrix Guest Tools. Once that completed and I verified that the VM rebooted successfully, I ran Sysprep and shut it down, then converted the VM to a template.

                                Now what I'd installed just prior to the BSOD happening, was MailEnable. However, the BSOD only happened after I initiated a reboot by using the OS restart button. Prior to that, I'd been rebooting using the controls in XOA. I'd been rebooting the same VM without any issues. I had to activate the OS license, then I rebooted. Joined it to the domain; rebooted. Installed MailEnable; rebooted. All went well.

                                The BSOD happened after I attempted to enable the Active Directory integration in MailEnable. I checked the box to enable the integration, and it wasn't working. So I read in their documentation that the user account needed to have some local User Rights. So I modified this in Group Policy, ran gpupdate, and initiated a reboot just to make sure the Group Policy took effect cleanly. That's when I ran into the BSOD.

                                Honestly, I set those local User Rights all the time and have never run into a BSOD. I can't tell you how those Intel drivers got there; I certainly didn't install them. Hope this is a helpful response?

                                D 1 Reply Last reply Reply Quote 0
                                • D Offline
                                  dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
                                  last edited by dinhngtu

                                  kagbasi-ngc I can't tell why the VM failed to boot originally. However, having the Intel RST and Xen drivers installed at the same time made me think that as the Xen drivers were installed before Sysprep, once the Xen drivers stopped functioning, the IRST drivers were no longer able to find your Windows device path. You could try the following procedure:

                                  • Boot into Windows PE
                                  • Use dism /image:C:\ /Get-Drivers to find the published name of iaStorAC.inf (oemxx.inf)
                                  • Use dism /image:C:\ /Remove-Driver /Driver:oemxx.inf to remove iaStorAC
                                  • You should be able to boot into Safe Mode. Rebooting will make things normal again.

                                  As for why the drivers are there, they are likely present in the installation ISO you used.

                                  K 1 Reply Last reply Reply Quote 0
                                  • K Offline
                                    kagbasi-ngc @dinhngtu
                                    last edited by

                                    dinhngtu Cool, I'll try your suggestion and report back.

                                    However, if you hunch holds, then I should be seeing this behavior on all my other VMs but I'm not. If you'll recall from a couple of days ago, you actually asked me to test this by building a new VM, installing MailEnable and seeing if the problem resurfaces - and it didn't.

                                    Here's the video: https://photos.app.goo.gl/Uw7WgFRY1BEem8gA8

                                    No worries, I'll report back my findings shortly.

                                    K 1 Reply Last reply Reply Quote 0
                                    • K Offline
                                      kagbasi-ngc @kagbasi-ngc
                                      last edited by

                                      dinhngtu I see two instances of iaStorAC.inf. Should I remove both of them?

                                      iaStoreAC-Removal_1.PNG

                                      D 1 Reply Last reply Reply Quote 0
                                      • D Offline
                                        dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
                                        last edited by

                                        kagbasi-ngc Yes, both.

                                        K 1 Reply Last reply Reply Quote 0
                                        • K Offline
                                          kagbasi-ngc @dinhngtu
                                          last edited by

                                          dinhngtu So I removed both drivers, oem3.inf and oem5.inf successfully. Unfortunately, the VM is still crashing with a BSOD of INACCESSIBLE_BOOT_DEVICE with Secure Boot on or off.

                                          D 1 Reply Last reply Reply Quote 0
                                          • D Offline
                                            dinhngtu Vates 🪐 XCP-ng Team @kagbasi-ngc
                                            last edited by

                                            kagbasi-ngc That's expected since the boot storage driver is gone. You'll need to get into Safe Mode with the Recovery or F8 menu.

                                            K 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post