@kagbasi-ngc Could you apply the hardening controls now and see if the VM still works?

@mrchip Could you post the contents of /var/log/{xensource,daemon}.log on the host (as well as their .1 version) as well? They're pretty big, so you'll have to upload them somewhere.

@mrchip Could you post a kernel log?

@kagbasi-ngc It seems like the debugger was attached a little late, after the Xen driver was loaded. Could you try attaching Windbg at the F8 menu instead?

Another thing to also point out, I cloned the snapshot of this VM to a new VM and observed the following:

• The cloned VM from snapshot booted up all the way into Windows, and I was able to login.

• I noticed that the Start Menu would not open (after several clicks), even though the Windows Explorer opens. So, I initiated a reboot - this time using the XO restart button.

• The VM went down for a reboot and got stuck in the same boot state (i.e., spinning wheel after the Windows splash logo). The VM's NIC comes up, however, because it starts responding to pings.

This tells me perhaps something is wrong with your template VM itself. Could you build a new template VM without any prebaked drivers to see if it's a driver or group policy issue?

@kagbasi-ngc {dbgsettings} are there but {default}/debug is not set to Yes.

@kagbasi-ngc Could you check bcdedit /store bcd /enum all (you can do this from Safe Mode, just do bcdedit /enum all) to see if the debugger settings are still there? For reference, it should look like:

Windows Boot Loader
-------------------
identifier              {current}
...
debug                   Yes

...

Debugger Settings
-----------------
identifier              {dbgsettings}
debugtype               Serial
debugport               1
baudrate                115200

@kagbasi-ngc Does it get stuck at "Waiting to reconnect..."? Do you get anything when telnet-ing to the VM's serial port?

@exime Could you gather the outputs of /run/initramfs/rdsosreport.txt, ls /sys/module/nvme/drivers/pci\:nvme and other relevant commands from the recovery shell?

If you can't type or use a USB drive during the recovery shell, do dracut --add-drivers "usbhid uas" -f /boot/initrd-temp.img and boot from that initrd.

@exime What does your grub.cfg and the output of ls -lR /dev/disk look like? At the Grub menu, try replacing quiet vga=785 splash plymouth.ignore-serial-consoles with vga=785 loglevel=8 and see if you can gather some logs.

@exime Is the error coming from Grub or the Dom0 kernel?

@kagbasi-ngc It shouldn't, the debugger drivers are built into Windows. If you're having issues with attaching the debugger, you can get into the F8 menu, start attaching WinDbg then choose Debugging mode in the F8 menu to start the guest. This should give WinDbg the correct timing to attach to the guest.

@kagbasi-ngc You might need the debugger in normal mode. Attach the debugger right at boot time and run the following command (beware the exact spelling):

sxe -c "lm1mna (poi(rdx));g" ld

Click Go until Windows loads completely or until you get a VM crash in Windbg, then paste the entire Windbg output.

@kagbasi-ngc Just to confirm, if you use the F8 menu it boots into Safe Mode without getting a BSOD, right? Please upload the SYSTEM hive to the same location.

@kagbasi-ngc The fact that your VM still boots in Safe Mode means that there's still some drivers blocking Windows from booting in normal mode. Please enable boot logging by running bcdedit /store bcd /set {default} bootlog yes then post the boot logs of normal mode versus safe mode. This log is found at C:\Windows\ntbtlog.txt.

Could you send another copy of your SYSTEM hive?

@kagbasi-ngc Could the VM get into Safe Mode?

@kagbasi-ngc I think for that it'd be more appropriate to contact our support team, which will be able to help you directly on your infrastructure. I'd also like to keep any troubleshooting information on the forum in case someone runs into a similar problem.

Non-inbox drivers mean drivers with "Inbox : No" as seen in your screenshot.

@kagbasi-ngc It worked on my VM when I tried that procedure on my local lab system. Unfortunately without a closer inspection I can't tell what's going on. If you still want to recover the VM, I'd start at removing all the non-inbox drivers.

@kagbasi-ngc That's expected since the boot storage driver is gone. You'll need to get into Safe Mode with the Recovery or F8 menu.

@kagbasi-ngc Yes, both.

@kagbasi-ngc I can't tell why the VM failed to boot originally. However, having the Intel RST and Xen drivers installed at the same time made me think that as the Xen drivers were installed before Sysprep, once the Xen drivers stopped functioning, the IRST drivers were no longer able to find your Windows device path. You could try the following procedure:

• Boot into Windows PE
• Use dism /image:C:\ /Get-Drivers to find the published name of iaStorAC.inf (oemxx.inf)
• Use dism /image:C:\ /Remove-Driver /Driver:oemxx.inf to remove iaStorAC
• You should be able to boot into Safe Mode. Rebooting will make things normal again.

As for why the drivers are there, they are likely present in the installation ISO you used.