VM Failing to Reboot
-
@kagbasi-ngc Could you check
bcdedit /store bcd /enum all(you can do this from Safe Mode, just dobcdedit /enum all) to see if the debugger settings are still there? For reference, it should look like:Windows Boot Loader ------------------- identifier {current} ... debug Yes ... Debugger Settings ----------------- identifier {dbgsettings} debugtype Serial debugport 1 baudrate 115200 -
@dinhngtu Checking now.
As soon as I selected
Safe Mode with Command Prompt, the following came up. So I'm waiting for it to finish.
-
@dinhngtu said in VM Failing to Reboot:
@kagbasi-ngc Could you check
bcdedit /store bcd /enum all(you can do this from Safe Mode, just dobcdedit /enum all) to see if the debugger settings are still there? For reference, it should look like:Here you go. As far as I can tell, the settings are there.
Firmware Boot Manager --------------------- identifier {fwbootmgr} displayorder {869b8842-cfa5-11ef-88bd-6690debbc340} {bootmgr} {4953fce7-d2e8-11ef-9050-806e6f6e6963} {68f8db89-dac7-11ef-b603-806e6f6e6963} {869b8841-cfa5-11ef-88bd-6690debbc340} {869b8846-cfa5-11ef-88bd-6690debbc340} {cc338294-d2fd-11ef-904b-806e6f6e6963} {869b8844-cfa5-11ef-88bd-6690debbc340} timeout 0 Windows Boot Manager -------------------- identifier {bootmgr} device partition=S: path \EFI\Microsoft\Boot\bootmgfw.efi description Windows Boot Manager locale en-US inherit {globalsettings} bootshutdowndisabled Yes default {default} resumeobject {869b8847-cfa5-11ef-88bd-6690debbc340} displayorder {default} toolsdisplayorder {memdiag} timeout 30 Firmware Application (101fffff) ------------------------------- identifier {4953fce7-d2e8-11ef-9050-806e6f6e6963} description UEFI Misc Device 2 Firmware Application (101fffff) ------------------------------- identifier {68f8db89-dac7-11ef-b603-806e6f6e6963} description UEFI Misc Device 3 Firmware Application (101fffff) ------------------------------- identifier {869b8841-cfa5-11ef-88bd-6690debbc340} description UiApp Firmware Application (101fffff) ------------------------------- identifier {869b8842-cfa5-11ef-88bd-6690debbc340} description UEFI Misc Device Firmware Application (101fffff) ------------------------------- identifier {869b8844-cfa5-11ef-88bd-6690debbc340} description UEFI PXEv6 (MAC:7E30B1BB2307) Firmware Application (101fffff) ------------------------------- identifier {869b8846-cfa5-11ef-88bd-6690debbc340} description EFI Internal Shell Firmware Application (101fffff) ------------------------------- identifier {cc338294-d2fd-11ef-904b-806e6f6e6963} description UEFI PXEv4 (MAC:7E30B1BB2307) Windows Boot Loader ------------------- identifier {default} device partition=C: path \windows\system32\winload.efi description Windows Server locale en-US inherit {bootloadersettings} recoverysequence {ad69be4c-cf8c-11ef-9047-6690debbc340} displaymessageoverride Recovery recoveryenabled Yes isolatedcontext Yes allowedinmemorysettings 0x15000075 osdevice partition=C: systemroot \windows resumeobject {869b8847-cfa5-11ef-88bd-6690debbc340} nx OptOut bootlog Yes Windows Boot Loader ------------------- identifier {ad69be4c-cf8c-11ef-9047-6690debbc340} device ramdisk=[\Device\HarddiskVolume4]\Recovery\WindowsRE\Winre.wim,{ad69be4d-cf8c-11ef-9047-6690debbc340} path \windows\system32\winload.efi description Windows Recovery Environment locale en-US inherit {bootloadersettings} displaymessage Recovery osdevice ramdisk=[\Device\HarddiskVolume4]\Recovery\WindowsRE\Winre.wim,{ad69be4d-cf8c-11ef-9047-6690debbc340} systemroot \windows nx OptIn bootmenupolicy Standard winpe Yes Resume from Hibernate --------------------- identifier {869b8847-cfa5-11ef-88bd-6690debbc340} device partition=C: path \windows\system32\winresume.efi description Windows Resume Application locale en-US inherit {resumeloadersettings} recoverysequence {ad69be4c-cf8c-11ef-9047-6690debbc340} recoveryenabled Yes isolatedcontext Yes allowedinmemorysettings 0x15000075 filedevice partition=C: custom:21000026 partition=C: filepath \hiberfil.sys debugoptionenabled No Windows Memory Tester --------------------- identifier {memdiag} device partition=S: path \EFI\Microsoft\Boot\memtest.efi description Windows Memory Diagnostic locale en-US inherit {globalsettings} badmemoryaccess Yes EMS Settings ------------ identifier {emssettings} bootems Yes Debugger Settings ----------------- identifier {dbgsettings} debugtype Serial debugport 1 baudrate 115200 RAM Defects ----------- identifier {badmemory} Global Settings --------------- identifier {globalsettings} inherit {dbgsettings} {emssettings} {badmemory} Boot Loader Settings -------------------- identifier {bootloadersettings} inherit {globalsettings} {hypervisorsettings} Hypervisor Settings ------------------- identifier {hypervisorsettings} hypervisordebugtype Serial hypervisordebugport 1 hypervisorbaudrate 115200 Resume Loader Settings ---------------------- identifier {resumeloadersettings} inherit {globalsettings} Device options -------------- identifier {ad69be4d-cf8c-11ef-9047-6690debbc340} description Windows Recovery ramdisksdidevice partition=\Device\HarddiskVolume4 ramdisksdipath \Recovery\WindowsRE\boot.sdi -
@kagbasi-ngc
{dbgsettings}are there but{default}/debugis not set to Yes. -
@dinhngtu My apologies for the delayed response. Got busy with several meetings.
So, I finally got the debugger to attach to the VM (after many tries). I was able to capture some logs, but was not able to get any response to the command you asked me to run:
sxe -c "lm1mna (poi(rdx));g" ldOne thing I did was to re-install the xen drivers, so you'll see them get loaded in the debug logs.
Another thing to also point out, I cloned the snapshot of this VM to a new VM and observed the following:
-
The cloned VM from snapshot booted up all the way into Windows, and I was able to login.
-
I noticed that the Start Menu would not open (after several clicks), even though the Windows Explorer opens. So, I initiated a reboot - this time using the XO restart button.
-
The VM went down for a reboot and got stuck in the same boot state (i.e., spinning wheel after the Windows splash logo). The VM's NIC comes up, however, because it starts responding to pings.
Anyway, here're the debug logs from the original VM:
************* Preparing the environment for Debugger Extensions Gallery repositories ************** ExtensionRepository : Implicit UseExperimentalFeatureForNugetShare : true AllowNugetExeUpdate : true NonInteractiveNuget : true AllowNugetMSCredentialProviderInstall : true AllowParallelInitializationOfLocalRepositories : true EnableRedirectToChakraJsProvider : false -- Configuring repositories ----> Repository : LocalInstalled, Enabled: true ----> Repository : UserExtensions, Enabled: true >>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds ************* Waiting for Debugger Extensions Gallery to Initialize ************** >>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.015 seconds ----> Repository : UserExtensions, Enabled: true, Packages count: 0 ----> Repository : LocalInstalled, Enabled: true, Packages count: 42 Microsoft (R) Windows Debugger Version 10.0.27725.1000 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. Opened 10.0.10.12 Waiting to reconnect... Connected to Windows 10 20348 x64 target at (Tue Jan 28 20:04:11.011 2025 (UTC - 5:00)), ptr64 TRUE Kernel Debugger connection established. Symbol search path is: srv* Executable search path is: Windows 10 Kernel Version 20348 MP (1 procs) Free x64 Edition build lab: 20348.859.amd64fre.fe_release_svc_prod2.220707-1832 Kernel base = 0xfffff803`1be00000 PsLoadedModuleList = 0xfffff803`1ca33d20 System Uptime: 0 days 0:00:00.000 Unable to load image ntkrnlmp.exe, Win32 error 0n2 xen|LogReadLogLevel: fail1 (c0000034) xen|LogReadLogLevel: fail1 (c0000034) xen|DllInitialize: 9.1.9 (105) (10.09.2024) xen|AcpiFindRsdp: 0x00000000000EA020 xen|SystemInitialize: CPUs 2 / 4 xen|SystemGetStartOptions: NOEXECUTE=OPTOUT BOOTLOG DEBUGPORT=COM1 BAUDRATE=115200 FVEBOOT=2670592 NOVGA DEBUG xen|SystemGetVersionInformation: KERNEL: 10.0 (BUILD 20348) PLATFORM WIN32_NT (x64) xen|SystemGetVersionInformation: SUITES: xen|SystemGetVersionInformation: - TERMINAL xen|SystemGetVersionInformation: - SINGLEUSERTS xen|SystemGetVersionInformation: TYPE: SERVER xen|SystemGetMemoryInformation: RANGE[0] 00000000.00001000 - 00000000.0009ffff xen|SystemGetMemoryInformation: RANGE[1] 00000000.00100000 - 00000000.ee4fbfff xen|SystemGetMemoryInformation: RANGE[2] 00000000.ee4fd000 - 00000000.ee510fff xen|SystemGetMemoryInformation: RANGE[3] 00000000.ee51b000 - 00000000.ef59afff xen|SystemGetMemoryInformation: RANGE[4] 00000000.ef5ff000 - 00000000.effdefff xen|SystemGetMemoryInformation: RANGE[5] 00000000.effff000 - 00000000.efffffff xen|SystemGetMemoryInformation: RANGE[6] 00000001.00000000 - 00000004.0f7fffff xen|SystemGetMemoryInformation: MaximumPhysicalAddress = 00000004.0f7fffff xen|AcpiGetXsdt: 0x00000000FC00A5E0 xen|SystemProcessorDpc: ====> (0:0) xen|SystemViridianInformation: ====> xen|SystemViridianInformation: Interface Identifier: Hv#1 xen|SystemViridianInformation: Hypervisor Features: xen|SystemViridianInformation: - Partition Reference Counter xen|SystemViridianInformation: - Basic SynIC MSRs xen|SystemViridianInformation: - Synthetic Timer MSRs xen|SystemViridianInformation: - APIC Access MSRs xen|SystemViridianInformation: - Hypercall MSRs xen|SystemViridianInformation: - Virtual Processor Index MSR xen|SystemViridianInformation: - Partition Reference TSC MSR xen|SystemViridianInformation: - Timer Frequency MSR xen|SystemViridianInformation: Recommendations: xen|SystemViridianInformation: - EOI, ICR and TPR access via MSR xen|SystemViridianInformation: - Use relaxed timing xen|SystemViridianInformation: - Retry spinlocks 2047 times xen|SystemViridianInformation: Hardware Features: xen|SystemViridianInformation: - APIC overlay assist xen|SystemViridianInformation: - MSR bitmaps xen|SystemViridianInformation: - Second Level Address Translation (SLAT) xen|SystemViridianInformation: <==== xen|SystemProcessorInitialize: Manufacturer: GenuineIntel xen|SystemProcessorInitialize: APIC ID: 00 xen|SystemProcessorInitialize: PROCESSOR ID: 00 xen|SystemProcessorDpc: <==== (0:0) xen|SystemProcessorDpc: ====> (0:1) xen|SystemProcessorInitialize: Manufacturer: GenuineIntel xen|SystemProcessorInitialize: APIC ID: 02 xen|SystemProcessorInitialize: PROCESSOR ID: 01 xen|SystemProcessorDpc: <==== (0:1) xen|SystemGetTimeInformation: FALSE xen|ModuleAdd: FFFFF8031BE00000 - FFFFF8031CE46FFF [ntoskrnl.exe] xen|ModuleAdd: FFFFF80319DA0000 - FFFFF80319DA5FFF [hal.dll] xen|ModuleAdd: FFFFF80319DB0000 - FFFFF80319DBDFFF [kdcom.dll] xen|ModuleAdd: FFFFF803199F0000 - FFFFF80319D73FFF [mcupdate_GenuineIntel.dll] xen|ModuleAdd: FFFFF8031F200000 - FFFFF8031F26CFFF [CLFS.SYS] xen|ModuleAdd: FFFFF80319DC0000 - FFFFF80319DE7FFF [tm.sys] xen|ModuleAdd: FFFFF8031F270000 - FFFFF8031F289FFF [PSHED.dll] xen|ModuleAdd: FFFFF80319DF0000 - FFFFF80319DFAFFF [BOOTVID.dll] xen|ModuleAdd: FFFFF8031F3B0000 - FFFFF8031F420FFF [FLTMGR.SYS] xen|ModuleAdd: FFFFF8031F460000 - FFFFF8031F4C1FFF [msrpc.sys] xen|ModuleAdd: FFFFF8031F430000 - FFFFF8031F458FFF [ksecdd.sys] xen|ModuleAdd: FFFFF8031F290000 - FFFFF8031F3A2FFF [clipsp.sys] xen|ModuleAdd: FFFFF8031F4D0000 - FFFFF8031F4DDFFF [cmimcext.sys] xen|ModuleAdd: FFFFF8031F4E0000 - FFFFF8031F4F0FFF [werkernel.sys] xen|ModuleAdd: FFFFF8031F500000 - FFFFF8031F50BFFF [ntosext.sys] xen|ModuleAdd: FFFFF8031F510000 - FFFFF8031F5F4FFF [CI.dll] xen|ModuleAdd: FFFFF8031F600000 - FFFFF8031F6BAFFF [cng.sys] xen|ModuleAdd: FFFFF8031F6C0000 - FFFFF8031F79CFFF [Wdf01000.sys] xen|ModuleAdd: FFFFF8031F7A0000 - FFFFF8031F7B3FFF [WDFLDR.SYS] xen|ModuleAdd: FFFFF8031F7C0000 - FFFFF8031F7CCFFF [PRM.sys] xen|ModuleAdd: FFFFF8031F7D0000 - FFFFF8031F7F5FFF [acpiex.sys] xen|ModuleAdd: FFFFF8031F800000 - FFFFF8031F810FFF [WppRecorder.sys] xen|ModuleAdd: FFFFF8031F820000 - FFFFF8031F82DFFF [msseccore.sys] xen|ModuleAdd: FFFFF8031F830000 - FFFFF8031F84AFFF [SgrmAgent.sys] xen|ModuleAdd: FFFFF8031F850000 - FFFFF8031F91AFFF [ACPI.sys] xen|ModuleAdd: FFFFF8031F920000 - FFFFF8031F92BFFF [WMILIB.SYS] xen|ModuleAdd: FFFFF8031F930000 - FFFFF8031F940FFF [WdBoot.sys] xen|ModuleAdd: FFFFF8031F950000 - FFFFF8031F966FFF [WindowsTrustedRT.sys] xen|ModuleAdd: FFFFF8031F970000 - FFFFF8031F9DAFFF [intelpep.sys] xen|ModuleAdd: FFFFF8031F9E0000 - FFFFF8031F9EAFFF [WindowsTrustedRTProxy.sys] xen|ModuleAdd: FFFFF8031F9F0000 - FFFFF8031F9FAFFF [IntelPMT.sys] xen|ModuleAdd: FFFFF8031FA00000 - FFFFF8031FA14FFF [pcw.sys] xen|ModuleAdd: FFFFF8031FA20000 - FFFFF8031FA3DFFF [sacdrv.sys] xen|ModuleAdd: FFFFF8031FAE0000 - FFFFF8031FC5BFFF [NDIS.SYS] xen|ModuleAdd: FFFFF8031FA40000 - FFFFF8031FADDFFF [NETIO.SYS] xen|ModuleAdd: FFFFF8031FC60000 - FFFFF8031FC6AFFF [msisadrv.sys] xen|ModuleAdd: FFFFF8031FC70000 - FFFFF8031FC85FFF [vdrvroot.sys] xen|ModuleAdd: FFFFF8031FC90000 - FFFFF8031FD13FFF [pci.sys] xen|ModuleAdd: FFFFF8031FD20000 - FFFFF8031FD50FFF [xenbus.sys] xen|ModuleAdd: FFFFF8031FD60000 - FFFFF8031FD80FFF [xen.sys] xen|ModuleAdd: FFFFF8031FD90000 - FFFFF8031FDA0FFF [xenfilt.sys] xen|ModuleAdd: FFFFF8031FDB0000 - FFFFF8031FDDEFFF [pdc.sys] xen|ModuleAdd: FFFFF8031FDE0000 - FFFFF8031FDF7FFF [CEA.sys] xen|ModuleAdd: FFFFF8031FE00000 - FFFFF8031FE30FFF [partmgr.sys] xen|ModuleAdd: FFFFF8031FE40000 - FFFFF8031FF0BFFF [spaceport.sys] xen|ModuleAdd: FFFFF8031FF10000 - FFFFF8031FF29FFF [volmgr.sys] xen|ModuleAdd: FFFFF8031FF30000 - FFFFF8031FF92FFF [volmgrx.sys] xen|ModuleAdd: FFFFF8031FFA0000 - FFFFF8031FFAAFFF [intelide.sys] xen|ModuleAdd: FFFFF8031FFB0000 - FFFFF8031FFC2FFF [PCIIDEX.SYS] xen|ModuleAdd: FFFFF8031FFD0000 - FFFFF8031FFEDFFF [mountmgr.sys] xen|ModuleAdd: FFFFF8031FFF0000 - FFFFF8031FFFCFFF [atapi.sys] xen|ModuleAdd: FFFFF80320000000 - FFFFF8032003BFFF [ataport.SYS] xen|ModuleAdd: FFFFF80320040000 - FFFFF80320075FFF [stornvme.sys] xen|ModuleAdd: FFFFF80320080000 - FFFFF8032014AFFF [storport.sys] xen|ModuleAdd: FFFFF80320150000 - FFFFF8032016BFFF [xenvbd.sys] xen|ModuleAdd: FFFFF80320170000 - FFFFF803201A9FFF [xencrsh.sys] xen|ModuleAdd: FFFFF803201B0000 - FFFFF803201BDFFF [xendisk.sys] xen|ModuleAdd: FFFFF803201C0000 - FFFFF803201E1FFF [EhStorClass.sys] xen|ModuleAdd: FFFFF803201F0000 - FFFFF80320231FFF [Wof.sys] xen|ModuleAdd: FFFFF80320240000 - FFFFF803202A9FFF [WdFilter.sys] xen|ModuleAdd: FFFFF803202B0000 - FFFFF803205C0FFF [Ntfs.sys] xen|ModuleAdd: FFFFF803205D0000 - FFFFF803205DCFFF [Fs_Rec.sys] xen|ModuleAdd: FFFFF803205E0000 - FFFFF80320612FFF [ksecpkg.sys] xen|ModuleAdd: FFFFF80320620000 - FFFFF8032093EFFF [tcpip.sys] xen|ModuleAdd: FFFFF80320940000 - FFFFF803209C0FFF [fwpkclnt.sys] xen|ModuleAdd: FFFFF803209D0000 - FFFFF803209FFFFF [wfplwfs.sys] xen|ModuleAdd: FFFFF80320A10000 - FFFFF80320A1AFFF [volume.sys] xen|ModuleAdd: FFFFF80320A20000 - FFFFF80320A93FFF [volsnap.sys] xen|ModuleAdd: FFFFF80320AA0000 - FFFFF80320AC5FFF [mup.sys] xen|ModuleAdd: FFFFF80320AD0000 - FFFFF80320ADFFFF [hwpolicy.sys] xen|ModuleAdd: FFFFF80320AF0000 - FFFFF80320B0EFFF [disk.sys] xen|ModuleAdd: FFFFF80320B10000 - FFFFF80320B84FFF [CLASSPNP.SYS] xen|UnplugSetRequest: DISKS (0) xen|UnplugSetRequest: NICS (0) xenbus|DriverEntry: 9.1.9 (105) (10.09.2024) xen|LogReadLogLevel: fail1 (c0000034) xen|ConfigSetActive: PCI\VEN_5853&DEV_0002&SUBSYS_00015853&REV_01\18: \Registry\Machine\System\CurrentControlSet\Services\pci,65536,0,3,0 xenbus|FdoCreate: FFFFB889490CBC00 (XS0002 XENBUS) [ACTIVE] xen|FiltersInstallClass: DEVCLASS_SYSTEM XENFILT xen|FiltersInstallClass: DEVCLASS_HDC XENFILT xenbus|FdoConnectInterrupt: FFFFB889490D5910: Shared LevelSensitive CPU 0:0 VECTOR a1 xenbus|FdoConnectInterrupt: FFFFB889490D5B30: DeviceExclusive Latched CPU 0:0 VECTOR b1 xenbus|FdoConnectInterrupt: FFFFB889490D5D50: DeviceExclusive Latched CPU 0:1 VECTOR b1 xenbus|FdoScan: ====> xenbus|FdoSuspend: ====> xenbus|FdoBalloon: ====> xenbus|FdoPciHoleCreate: 000f0000 - 000f0fff xenbus|EvtchnAbiAcquire: FIFO xenbus|EvtchnInterruptEnable: CPU 0:0 (Vector = 177) xenbus|EvtchnInterruptEnable: CPU 0:1 (Vector = 177) xenbus|EvtchnInterruptEnable: CALLBACK VIA (Vector = 28) xenbus|GnttabExpand: added references [00000020 - 000001ff] xenbus|EvtchnFifoExpand: added ports [00000000 - 000003ff] xenbus|__FdoVirqCreate: DEBUG: CPU 0:0 xenbus|PdoCreate: FFFFB88948C7B540 (VBD) xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2 xenbus|PdoCreate: FFFFB88948D7F540 (VIF) xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2 xenbus|PdoCreate: FFFFB88948D7F870 (IFACE) xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2 xenfilt|DriverEntry: 9.1.9 (105) (10.09.2024) xenfilt|DriverAddDevice: FFFFB8894910B360 UNKNOWN xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE xenfilt|DriverAddDevice: FFFFB8894916BAB0 IDE xenfilt|FdoCreate: FFFFB88949170DA0 (PCIIDE\IDEChannel\0) xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE xenfilt|DriverAddDevice: FFFFB8894916C060 IDE xenfilt|FdoCreate: FFFFB88949176DA0 (PCIIDE\IDEChannel\1) xenfilt|PdoCreate: FFFFB889490E2510 (IDE\CdRomQEMU_QEMU_DVD-ROM_______________________2.5+____\1.1.0) Break instruction exception - code 80000003 (first chance) ******************************************************************************* * * * You are seeing this message because you pressed either * * CTRL+C (if you run console kernel debugger) or, * * CTRL+BREAK (if you run GUI kernel debugger), * * on your debugger machine's keyboard. * * * * THIS IS NOT A BUG OR A SYSTEM CRASH * * * * If you did not intend to break into the debugger, press the "g" key, then * * press the "Enter" key now. This message might immediately reappear. If it * * does, press "g" and "Enter" again. * * * ******************************************************************************* nt!DbgBreakPointWithStatus: fffff803`1c229560 cc int 3 0: kd> sxe -c "lm1mna (poi(rdx));g" ld 0: kd> g -
-
@kagbasi-ngc It seems like the debugger was attached a little late, after the Xen driver was loaded. Could you try attaching Windbg at the F8 menu instead?
Another thing to also point out, I cloned the snapshot of this VM to a new VM and observed the following:
-
The cloned VM from snapshot booted up all the way into Windows, and I was able to login.
-
I noticed that the Start Menu would not open (after several clicks), even though the Windows Explorer opens. So, I initiated a reboot - this time using the XO restart button.
-
The VM went down for a reboot and got stuck in the same boot state (i.e., spinning wheel after the Windows splash logo). The VM's NIC comes up, however, because it starts responding to pings.
This tells me perhaps something is wrong with your template VM itself. Could you build a new template VM without any prebaked drivers to see if it's a driver or group policy issue?
-
-
@dinhngtu I can certainly do that. I will be back in the lab in about 2 hours and will retry.
Concerning the template, hmmm....I was having the same thoughts, so I've pulled down a Windows Server 2022 ISO directly from Microsoft and I'm gonna build a new template with it, to see if the problem resurfaces. It's possible there's something in the Secure Host Baseline that is causing problems with this platform (not sure if it's XCP-ng, or Xen). However, if that ends up being the case, it would be a bummer as that would weaken my case for adoption of the Vates VMS.
In any case, I'll report back in a couple of hours.
-
@dinhngtu said in VM Failing to Reboot:
@kagbasi-ngc It seems like the debugger was attached a little late, after the Xen driver was loaded. Could you try attaching Windbg at the F8 menu instead?
As requested, I managed to attach Windbg at the F8 menu. Here's the output:
************* Preparing the environment for Debugger Extensions Gallery repositories ************** ExtensionRepository : Implicit UseExperimentalFeatureForNugetShare : true AllowNugetExeUpdate : true NonInteractiveNuget : true AllowNugetMSCredentialProviderInstall : true AllowParallelInitializationOfLocalRepositories : true EnableRedirectToChakraJsProvider : false -- Configuring repositories ----> Repository : LocalInstalled, Enabled: true ----> Repository : UserExtensions, Enabled: true >>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds ************* Waiting for Debugger Extensions Gallery to Initialize ************** >>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.032 seconds ----> Repository : UserExtensions, Enabled: true, Packages count: 0 ----> Repository : LocalInstalled, Enabled: true, Packages count: 42 Microsoft (R) Windows Debugger Version 10.0.27725.1000 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. Opened 10.0.10.12 Waiting to reconnect... Connected to Windows 10 20348 x64 target at (Wed Jan 29 20:06:47.664 2025 (UTC - 5:00)), ptr64 TRUE Kernel Debugger connection established. Symbol search path is: srv* Executable search path is: Windows 10 Kernel Version 20348 MP (1 procs) Free x64 Edition build lab: 20348.859.amd64fre.fe_release_svc_prod2.220707-1832 Kernel base = 0xfffff802`6c600000 PsLoadedModuleList = 0xfffff802`6d233d20 System Uptime: 0 days 0:00:00.000 Unable to load image ntkrnlmp.exe, Win32 error 0n2 xen|LogReadLogLevel: fail1 (c0000034) xen|LogReadLogLevel: fail1 (c0000034) xen|DllInitialize: 9.1.9 (105) (10.09.2024) xen|AcpiFindRsdp: 0x00000000000EA020 xen|SystemInitialize: CPUs 2 / 4 xen|SystemGetStartOptions: NOEXECUTE=OPTOUT BOOTLOG DEBUGPORT=COM1 BAUDRATE=115200 FVEBOOT=2670592 NOVGA DEBUG xen|SystemGetVersionInformation: KERNEL: 10.0 (BUILD 20348) PLATFORM WIN32_NT (x64) xen|SystemGetVersionInformation: SUITES: xen|SystemGetVersionInformation: - TERMINAL xen|SystemGetVersionInformation: - SINGLEUSERTS xen|SystemGetVersionInformation: TYPE: SERVER xen|SystemGetMemoryInformation: RANGE[0] 00000000.00001000 - 00000000.0009ffff xen|SystemGetMemoryInformation: RANGE[1] 00000000.00100000 - 00000000.ee4fbfff xen|SystemGetMemoryInformation: RANGE[2] 00000000.ee4fd000 - 00000000.ee510fff xen|SystemGetMemoryInformation: RANGE[3] 00000000.ee51b000 - 00000000.ef59afff xen|SystemGetMemoryInformation: RANGE[4] 00000000.ef5ff000 - 00000000.effdefff xen|SystemGetMemoryInformation: RANGE[5] 00000000.effff000 - 00000000.efffffff xen|SystemGetMemoryInformation: RANGE[6] 00000001.00000000 - 00000004.0f7fffff xen|SystemGetMemoryInformation: MaximumPhysicalAddress = 00000004.0f7fffff xen|AcpiGetXsdt: 0x00000000FC00A5E0 xen|SystemProcessorDpc: ====> (0:0) xen|SystemViridianInformation: ====> xen|SystemViridianInformation: Interface Identifier: Hv#1 xen|SystemViridianInformation: Hypervisor Features: xen|SystemViridianInformation: - Partition Reference Counter xen|SystemViridianInformation: - Basic SynIC MSRs xen|SystemViridianInformation: - Synthetic Timer MSRs xen|SystemViridianInformation: - APIC Access MSRs xen|SystemViridianInformation: - Hypercall MSRs xen|SystemViridianInformation: - Virtual Processor Index MSR xen|SystemViridianInformation: - Partition Reference TSC MSR xen|SystemViridianInformation: - Timer Frequency MSR xen|SystemViridianInformation: Recommendations: xen|SystemViridianInformation: - EOI, ICR and TPR access via MSR xen|SystemViridianInformation: - Use relaxed timing xen|SystemViridianInformation: - Retry spinlocks 2047 times xen|SystemViridianInformation: Hardware Features: xen|SystemViridianInformation: - APIC overlay assist xen|SystemViridianInformation: - MSR bitmaps xen|SystemViridianInformation: - Second Level Address Translation (SLAT) xen|SystemViridianInformation: <==== xen|SystemProcessorInitialize: Manufacturer: GenuineIntel xen|SystemProcessorInitialize: APIC ID: 00 xen|SystemProcessorInitialize: PROCESSOR ID: 00 xen|SystemProcessorDpc: <==== (0:0) xen|SystemProcessorDpc: ====> (0:1) xen|SystemProcessorInitialize: Manufacturer: GenuineIntel xen|SystemProcessorInitialize: APIC ID: 02 xen|SystemProcessorInitialize: PROCESSOR ID: 01 xen|SystemProcessorDpc: <==== (0:1) xen|SystemGetTimeInformation: FALSE xen|ModuleAdd: FFFFF8026C600000 - FFFFF8026D646FFF [ntoskrnl.exe] xen|ModuleAdd: FFFFF8026B910000 - FFFFF8026B915FFF [hal.dll] xen|ModuleAdd: FFFFF8026B920000 - FFFFF8026B92DFFF [kdcom.dll] xen|ModuleAdd: FFFFF8026B560000 - FFFFF8026B8E3FFF [mcupdate_GenuineIntel.dll] xen|ModuleAdd: FFFFF8026B960000 - FFFFF8026B9CCFFF [CLFS.SYS] xen|ModuleAdd: FFFFF8026B930000 - FFFFF8026B957FFF [tm.sys] xen|ModuleAdd: FFFFF8026B9D0000 - FFFFF8026B9E9FFF [PSHED.dll] xen|ModuleAdd: FFFFF8026B9F0000 - FFFFF8026B9FAFFF [BOOTVID.dll] xen|ModuleAdd: FFFFF80270F20000 - FFFFF80270F90FFF [FLTMGR.SYS] xen|ModuleAdd: FFFFF80270FD0000 - FFFFF80271031FFF [msrpc.sys] xen|ModuleAdd: FFFFF80270FA0000 - FFFFF80270FC8FFF [ksecdd.sys] xen|ModuleAdd: FFFFF80270E00000 - FFFFF80270F12FFF [clipsp.sys] xen|ModuleAdd: FFFFF80271040000 - FFFFF8027104DFFF [cmimcext.sys] xen|ModuleAdd: FFFFF80271050000 - FFFFF80271060FFF [werkernel.sys] xen|ModuleAdd: FFFFF80271070000 - FFFFF8027107BFFF [ntosext.sys] xen|ModuleAdd: FFFFF80271080000 - FFFFF80271164FFF [CI.dll] xen|ModuleAdd: FFFFF80271170000 - FFFFF8027122AFFF [cng.sys] xen|ModuleAdd: FFFFF80271230000 - FFFFF8027130CFFF [Wdf01000.sys] xen|ModuleAdd: FFFFF80271310000 - FFFFF80271323FFF [WDFLDR.SYS] xen|ModuleAdd: FFFFF80271330000 - FFFFF8027133CFFF [PRM.sys] xen|ModuleAdd: FFFFF80271340000 - FFFFF80271365FFF [acpiex.sys] xen|ModuleAdd: FFFFF80271370000 - FFFFF80271380FFF [WppRecorder.sys] xen|ModuleAdd: FFFFF80271390000 - FFFFF8027139DFFF [msseccore.sys] xen|ModuleAdd: FFFFF802713A0000 - FFFFF802713BAFFF [SgrmAgent.sys] xen|ModuleAdd: FFFFF802713C0000 - FFFFF8027148AFFF [ACPI.sys] xen|ModuleAdd: FFFFF80271490000 - FFFFF8027149BFFF [WMILIB.SYS] xen|ModuleAdd: FFFFF802714A0000 - FFFFF802714B0FFF [WdBoot.sys] xen|ModuleAdd: FFFFF802714C0000 - FFFFF802714D6FFF [WindowsTrustedRT.sys] xen|ModuleAdd: FFFFF802714E0000 - FFFFF8027154AFFF [intelpep.sys] xen|ModuleAdd: FFFFF80271550000 - FFFFF8027155AFFF [WindowsTrustedRTProxy.sys] xen|ModuleAdd: FFFFF80271560000 - FFFFF8027156AFFF [IntelPMT.sys] xen|ModuleAdd: FFFFF80271570000 - FFFFF80271584FFF [pcw.sys] xen|ModuleAdd: FFFFF80271590000 - FFFFF802715ADFFF [sacdrv.sys] xen|ModuleAdd: FFFFF80271650000 - FFFFF802717CBFFF [NDIS.SYS] xen|ModuleAdd: FFFFF802715B0000 - FFFFF8027164DFFF [NETIO.SYS] xen|ModuleAdd: FFFFF802717D0000 - FFFFF802717DAFFF [msisadrv.sys] xen|ModuleAdd: FFFFF802717E0000 - FFFFF802717F5FFF [vdrvroot.sys] xen|ModuleAdd: FFFFF80271800000 - FFFFF80271883FFF [pci.sys] xen|ModuleAdd: FFFFF80271890000 - FFFFF802718C0FFF [xenbus.sys] xen|ModuleAdd: FFFFF802718D0000 - FFFFF802718F0FFF [xen.sys] xen|ModuleAdd: FFFFF80271900000 - FFFFF80271910FFF [xenfilt.sys] xen|ModuleAdd: FFFFF80271920000 - FFFFF8027194EFFF [pdc.sys] xen|ModuleAdd: FFFFF80271950000 - FFFFF80271967FFF [CEA.sys] xen|ModuleAdd: FFFFF80271970000 - FFFFF802719A0FFF [partmgr.sys] xen|ModuleAdd: FFFFF802719B0000 - FFFFF80271A7BFFF [spaceport.sys] xen|ModuleAdd: FFFFF80271A80000 - FFFFF80271A99FFF [volmgr.sys] xen|ModuleAdd: FFFFF80271AA0000 - FFFFF80271B02FFF [volmgrx.sys] xen|ModuleAdd: FFFFF80271B10000 - FFFFF80271B1AFFF [intelide.sys] xen|ModuleAdd: FFFFF80271B20000 - FFFFF80271B32FFF [PCIIDEX.SYS] xen|ModuleAdd: FFFFF80271B40000 - FFFFF80271B5DFFF [mountmgr.sys] xen|ModuleAdd: FFFFF80271B60000 - FFFFF80271B6CFFF [atapi.sys] xen|ModuleAdd: FFFFF80271B70000 - FFFFF80271BABFFF [ataport.SYS] xen|ModuleAdd: FFFFF80271BB0000 - FFFFF80271BE5FFF [stornvme.sys] xen|ModuleAdd: FFFFF80271BF0000 - FFFFF80271CBAFFF [storport.sys] xen|ModuleAdd: FFFFF80271CC0000 - FFFFF80271CDBFFF [xenvbd.sys] xen|ModuleAdd: FFFFF80271CE0000 - FFFFF80271D19FFF [xencrsh.sys] xen|ModuleAdd: FFFFF80271D20000 - FFFFF80271D2DFFF [xendisk.sys] xen|ModuleAdd: FFFFF80271D30000 - FFFFF80271D51FFF [EhStorClass.sys] xen|ModuleAdd: FFFFF80271D60000 - FFFFF80271DA1FFF [Wof.sys] xen|ModuleAdd: FFFFF80271DB0000 - FFFFF80271E19FFF [WdFilter.sys] xen|ModuleAdd: FFFFF80271E20000 - FFFFF80272130FFF [Ntfs.sys] xen|ModuleAdd: FFFFF80272140000 - FFFFF8027214CFFF [Fs_Rec.sys] xen|ModuleAdd: FFFFF80272150000 - FFFFF80272182FFF [ksecpkg.sys] xen|ModuleAdd: FFFFF80272190000 - FFFFF802724AEFFF [tcpip.sys] xen|ModuleAdd: FFFFF802724B0000 - FFFFF80272530FFF [fwpkclnt.sys] xen|ModuleAdd: FFFFF80272540000 - FFFFF8027256FFFF [wfplwfs.sys] xen|ModuleAdd: FFFFF80272580000 - FFFFF8027258AFFF [volume.sys] xen|ModuleAdd: FFFFF80272590000 - FFFFF80272603FFF [volsnap.sys] xen|ModuleAdd: FFFFF80272610000 - FFFFF80272635FFF [mup.sys] xen|ModuleAdd: FFFFF80272640000 - FFFFF8027264FFFF [hwpolicy.sys] xen|ModuleAdd: FFFFF80272660000 - FFFFF8027267EFFF [disk.sys] xen|ModuleAdd: FFFFF80272680000 - FFFFF802726F4FFF [CLASSPNP.SYS] xen|UnplugSetRequest: DISKS (0) xen|UnplugSetRequest: NICS (0) xenbus|DriverEntry: 9.1.9 (105) (10.09.2024) xen|LogReadLogLevel: fail1 (c0000034) xen|ConfigSetActive: PCI\VEN_5853&DEV_0002&SUBSYS_00015853&REV_01\18: \Registry\Machine\System\CurrentControlSet\Services\pci,65536,0,3,0 xenbus|FdoCreate: FFFF9A05CD4AB550 (XS0002 XENBUS) [ACTIVE] xen|FiltersInstallClass: DEVCLASS_SYSTEM XENFILT xen|FiltersInstallClass: DEVCLASS_HDC XENFILT xenbus|FdoConnectInterrupt: FFFF9A05CD90E6E0: Shared LevelSensitive CPU 0:0 VECTOR a1 xenbus|FdoConnectInterrupt: FFFF9A05CD4AE6E0: DeviceExclusive Latched CPU 0:0 VECTOR b1 xenbus|FdoConnectInterrupt: FFFF9A05CD4AD6E0: DeviceExclusive Latched CPU 0:1 VECTOR b1 xenbus|FdoScan: ====> xenbus|FdoSuspend: ====> xenbus|FdoBalloon: ====> xenbus|FdoPciHoleCreate: 000f0000 - 000f0fff xenbus|EvtchnAbiAcquire: FIFO xenbus|EvtchnInterruptEnable: CPU 0:0 (Vector = 177) xenbus|EvtchnInterruptEnable: CPU 0:1 (Vector = 177) xenbus|EvtchnInterruptEnable: CALLBACK VIA (Vector = 28) xenbus|GnttabExpand: added references [00000020 - 000001ff] xenbus|EvtchnFifoExpand: added ports [00000000 - 000003ff] xenbus|__FdoVirqCreate: DEBUG: CPU 0:0 xenbus|PdoCreate: FFFF9A05CD8DB530 (VBD) xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2 xenbus|PdoCreate: FFFF9A05CD8DA530 (VIF) xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2 xenbus|PdoCreate: FFFF9A05CD8D94B0 (IFACE) xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2 xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2 xenfilt|DriverEntry: 9.1.9 (105) (10.09.2024) xenfilt|DriverAddDevice: FFFF9A05CD8E2360 UNKNOWN xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE xenfilt|DriverAddDevice: FFFF9A05CD8DE270 IDE xenfilt|FdoCreate: FFFF9A05CD5DDDA0 (PCIIDE\IDEChannel\0) xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE xenfilt|DriverAddDevice: FFFF9A05CD8DE5A0 IDE xenfilt|FdoCreate: FFFF9A05CD939DA0 (PCIIDE\IDEChannel\1) xenfilt|PdoCreate: FFFF9A05CD97A750 (IDE\CdRomQEMU_QEMU_DVD-ROM_______________________2.5+____\1.1.0)Could you build a new template VM without any prebaked drivers to see if it's a driver or group policy issue?
I'm in the process of building a new template now using a vanilla ISO of Windows Server 2022. So far the VM is running fine; no BSOD after multiple reboots. I'll provide an update when patching is completed and I've turned the new VM to a template and built a VM from that.
-
@dinhngtu Some positive news to report.
The new template and VM I created, using a vanilla ISO directly from Microsoft, is working flawlessly. I am able to control it from within the OS and by using the XO controls. I've performed many reboots and shutdowns, and thus far no BSOD.
Unfortunately, this suggests that there may be something with the hardening of the Secured Host Baseline image I'm using; which means my case for getting Vates VMS adopted is now weaker, until we identify the root cause of the BSOD.
-
Quick check: is hardening is done via any MS virtualization technology? If yes, it could be a nested virt problem maybe (just throwing ideas)
-
@kagbasi-ngc Could you apply the hardening controls now and see if the VM still works?
-
@olivierlambert said in VM Failing to Reboot:
Quick check: is hardening is done via any MS virtualization technology? If yes, it could be a nested virt problem maybe (just throwing ideas)
Certainly could be.
@dinhngtu said in VM Failing to Reboot:
@kagbasi-ngc Could you apply the hardening controls now and see if the VM still works?
Yeah, I'll try doing that today to see what happens. We generally get an already hardened ISO from our upstream Security Team, so as to minimize the deployment time. However, I can manually harden myself - although, since it may not be the exact same process as followed by the upstream team, there's a chance I will not be testing apples-to-apples.
-
D dinhngtu referenced this topic on
