@kagbasi-ngc I figured out what the problem was, so wanted to post it for the benefit of the community.

The root cause for me ended up being a firewall issue on the Windows Server where I was running the NFS server role. While I hadn't made any changes to the firewall, the profile of the network interface connected to the management network had changed from Domain to Public/Private, thus, Windows Defender Firewall applied the most restrictive settings since it saw that the NIC wasn't connected to the Domain profile (even though it physically was).

This post on Serverfault (https://serverfault.com/a/647201/189248) clued me in and I looked at the Network Location Awareness service (NLA). It was set automatic and was running, so I restarted it and this reverted the profile of the NIC back to Domain and caused Windows Defender Firewall to apply the previously configured rules.

It is important to note that this situation won't be common, as most of you won't be running your NAS on a dual-homed domain controller that is also running the DHCP server role. Hope this helps someone.

@Prilly Looks like @lawrencesystems has a good explainer video on YouTube at https://youtu.be/weVoKm8kDb4?si=QH93rOKaXLglrIbS

Check it out when you get a chance, thanks.

@nathanael-h Yeah, I've already done that several times and no luck.

What I'm gonna try now is see if I can boot it with the Windows installation disk, go into recovery and access the WinPE command line, and see if I can run xenbootfix.exe to clean out the guest tools/drivers.

If that doesn't work, I'll just blow away the VM and start over. I just hate doing this, cos it doesn't give us the chance to figure out why this happened.

@olivierlambert Cool, I'll run the other two when I get back. I'll kick them off simultaneously.

@olivierlambert No sir, I have not. I'll try that and report back shortly.

@stormi Sorry for the delayed reply. I did as you requested, and below are the resulting log files. I started the VM at timestamp Dec 17 14:43:34 and stopped it about 2 minutes later.

• daemon.log
• xensource.log

If I can be of assistance, don't hesitate to ask please. Thank you.

@olivierlambert No, it's the other way around. Once the subject user is a memberof 3 or more groups, the problem occurs.

@manilx Yeah, not doing it in production, just in a test environment. Plus I only did it because I actually found a write-up by the Vates team on how to get it done, which confirmed to me that while they don't recommend it for production, they understand that it has some merit in a testing environment and want to see it working.

That said, I definitely hear your admonition and would never do it in a production setting.

@olivierlambert Certainly strange, then, that the same settings and same plugin works in XOA but not in XOCE. We could argue perhaps some configuration variances within the Domain Controllers, but given that I'm the same person that configured both environments - I'm highly confident that I stood up the DCs in the same way. Only things that would obviously be different would be the Domain Name and OU Structure.

I'll confirm the Forest and Domain functional levels later this evening and perhaps run an XOA trial and see if it works. I'll report back.

@olivierlambert So XOA is running on local storage of host #1 (VMH01) and all the VMs run off an NFS datastore which I've checked and is not in read-only mode or full (has about 60% free space).

What I haven't checked is whether the partition being used as the Local SR on VMH01 is perhaps the culprit. I ran xe check yesterday and the only three things it complained about were NTP, Updates, and not having Internet connectivity.

I should be in the office in about an hour, so I'll inspect and report back.

@Danp Aaah, thanks for pointing me to these resources. After reading, I thought it best to simply increase the quotas in my resource set definition to accommodate.

I felt having to modify the config file was too clunky. This seems like something that should be exposed in the UI when defining the resource set.

Good-day Folks,

I need some help understanding the Resource Set quotas in Xen Orchestra. I've read through the official documentation but couldn't find anything that addresses what I'm seeing.

So, it's my understanding that the quotas that can be set in a Resource Set are global and not per VM created - is this correct? If so, then I'm not understanding why I am unable to create a snapshot when it seems I have the resources available to do so. Essentially, as you may gleam from the screenshot below, I created a Resource Set to have a limit of 8 GiB RAM, 150 GiB Storage, and 1 vCPU. With these limits, I thought I should be able to create a snapshot of the VM I created using this Resource Set. Unfortunately, an attempt to snapshot this VM, results in the following error message:

not enough resources in resource set

It appears my understanding of the quotas may be incorrect and would appreciate some input from the community, thank you.

@DustinB Hmm, hadn't thought of that one. I'll try it and see.

Good-day Folks (and a Happy Friday),

Does anyone know of a way to set the startup order of VMs within a pool? In my use case, I have a DHCP server that runs on one of my VMs, and it would be very beneficial to have that VM up first (at least immediately after the XO VM comes up) before any other VM within the pool.

@CodeMercenary Just adding my two cents - for what it's worth.

I have a single node pool that I run at my church (about to add a second host this weekend). However, I have found great success by just going to Advanced > Smart Reboot. I, too, don't have shared storage and I've found that this halts all the VMs and does a clean reboot. However, it only restarts the XO VM, so all other VMs must be resumed/started manually (something I've found to not be a problem to do).

Hope this helps ya, thanks.

@olivierlambert said in VM Failing to Reboot:

Quick check: is hardening is done via any MS virtualization technology? If yes, it could be a nested virt problem maybe (just throwing ideas)

Certainly could be.

@dinhngtu said in VM Failing to Reboot:

@kagbasi-ngc Could you apply the hardening controls now and see if the VM still works?

Yeah, I'll try doing that today to see what happens. We generally get an already hardened ISO from our upstream Security Team, so as to minimize the deployment time. However, I can manually harden myself - although, since it may not be the exact same process as followed by the upstream team, there's a chance I will not be testing apples-to-apples.

@dinhngtu Some positive news to report.

The new template and VM I created, using a vanilla ISO directly from Microsoft, is working flawlessly. I am able to control it from within the OS and by using the XO controls. I've performed many reboots and shutdowns, and thus far no BSOD.

Unfortunately, this suggests that there may be something with the hardening of the Secured Host Baseline image I'm using; which means my case for getting Vates VMS adopted is now weaker, until we identify the root cause of the BSOD.

@dinhngtu said in VM Failing to Reboot:

@kagbasi-ngc It seems like the debugger was attached a little late, after the Xen driver was loaded. Could you try attaching Windbg at the F8 menu instead?

As requested, I managed to attach Windbg at the F8 menu. Here's the output:

************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   NonInteractiveNuget : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true
   EnableRedirectToChakraJsProvider : false

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.032 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 42

Microsoft (R) Windows Debugger Version 10.0.27725.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Opened 10.0.10.12
Waiting to reconnect...
Connected to Windows 10 20348 x64 target at (Wed Jan 29 20:06:47.664 2025 (UTC - 5:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*
Executable search path is: 
Windows 10 Kernel Version 20348 MP (1 procs) Free x64
Edition build lab: 20348.859.amd64fre.fe_release_svc_prod2.220707-1832
Kernel base = 0xfffff802`6c600000 PsLoadedModuleList = 0xfffff802`6d233d20
System Uptime: 0 days 0:00:00.000
Unable to load image ntkrnlmp.exe, Win32 error 0n2
xen|LogReadLogLevel: fail1 (c0000034)
xen|LogReadLogLevel: fail1 (c0000034)
xen|DllInitialize: 9.1.9 (105) (10.09.2024)
xen|AcpiFindRsdp: 0x00000000000EA020
xen|SystemInitialize: CPUs 2 / 4
xen|SystemGetStartOptions:  NOEXECUTE=OPTOUT  BOOTLOG  DEBUGPORT=COM1  BAUDRATE=115200  FVEBOOT=2670592  NOVGA  DEBUG
xen|SystemGetVersionInformation: KERNEL: 10.0 (BUILD 20348) PLATFORM WIN32_NT (x64)
xen|SystemGetVersionInformation: SUITES:
xen|SystemGetVersionInformation: - TERMINAL
xen|SystemGetVersionInformation: - SINGLEUSERTS
xen|SystemGetVersionInformation: TYPE: SERVER
xen|SystemGetMemoryInformation: RANGE[0] 00000000.00001000 - 00000000.0009ffff
xen|SystemGetMemoryInformation: RANGE[1] 00000000.00100000 - 00000000.ee4fbfff
xen|SystemGetMemoryInformation: RANGE[2] 00000000.ee4fd000 - 00000000.ee510fff
xen|SystemGetMemoryInformation: RANGE[3] 00000000.ee51b000 - 00000000.ef59afff
xen|SystemGetMemoryInformation: RANGE[4] 00000000.ef5ff000 - 00000000.effdefff
xen|SystemGetMemoryInformation: RANGE[5] 00000000.effff000 - 00000000.efffffff
xen|SystemGetMemoryInformation: RANGE[6] 00000001.00000000 - 00000004.0f7fffff
xen|SystemGetMemoryInformation: MaximumPhysicalAddress = 00000004.0f7fffff
xen|AcpiGetXsdt: 0x00000000FC00A5E0
xen|SystemProcessorDpc: ====> (0:0)
xen|SystemViridianInformation: ====>
xen|SystemViridianInformation: Interface Identifier: Hv#1
xen|SystemViridianInformation: Hypervisor Features:
xen|SystemViridianInformation:  - Partition Reference Counter
xen|SystemViridianInformation:  - Basic SynIC MSRs
xen|SystemViridianInformation:  - Synthetic Timer MSRs
xen|SystemViridianInformation:  - APIC Access MSRs
xen|SystemViridianInformation:  - Hypercall MSRs
xen|SystemViridianInformation:  - Virtual Processor Index MSR
xen|SystemViridianInformation:  - Partition Reference TSC MSR
xen|SystemViridianInformation:  - Timer Frequency MSR
xen|SystemViridianInformation: Recommendations:
xen|SystemViridianInformation:  - EOI, ICR and TPR access via MSR
xen|SystemViridianInformation:  - Use relaxed timing
xen|SystemViridianInformation:  - Retry spinlocks 2047 times
xen|SystemViridianInformation: Hardware Features:
xen|SystemViridianInformation:  - APIC overlay assist
xen|SystemViridianInformation:  - MSR bitmaps
xen|SystemViridianInformation:  - Second Level Address Translation (SLAT)
xen|SystemViridianInformation: <====
xen|SystemProcessorInitialize: Manufacturer: GenuineIntel
xen|SystemProcessorInitialize: APIC ID: 00
xen|SystemProcessorInitialize: PROCESSOR ID: 00
xen|SystemProcessorDpc: <==== (0:0)
xen|SystemProcessorDpc: ====> (0:1)
xen|SystemProcessorInitialize: Manufacturer: GenuineIntel
xen|SystemProcessorInitialize: APIC ID: 02
xen|SystemProcessorInitialize: PROCESSOR ID: 01
xen|SystemProcessorDpc: <==== (0:1)
xen|SystemGetTimeInformation: FALSE
xen|ModuleAdd: FFFFF8026C600000 - FFFFF8026D646FFF [ntoskrnl.exe]
xen|ModuleAdd: FFFFF8026B910000 - FFFFF8026B915FFF [hal.dll]
xen|ModuleAdd: FFFFF8026B920000 - FFFFF8026B92DFFF [kdcom.dll]
xen|ModuleAdd: FFFFF8026B560000 - FFFFF8026B8E3FFF [mcupdate_GenuineIntel.dll]
xen|ModuleAdd: FFFFF8026B960000 - FFFFF8026B9CCFFF [CLFS.SYS]
xen|ModuleAdd: FFFFF8026B930000 - FFFFF8026B957FFF [tm.sys]
xen|ModuleAdd: FFFFF8026B9D0000 - FFFFF8026B9E9FFF [PSHED.dll]
xen|ModuleAdd: FFFFF8026B9F0000 - FFFFF8026B9FAFFF [BOOTVID.dll]
xen|ModuleAdd: FFFFF80270F20000 - FFFFF80270F90FFF [FLTMGR.SYS]
xen|ModuleAdd: FFFFF80270FD0000 - FFFFF80271031FFF [msrpc.sys]
xen|ModuleAdd: FFFFF80270FA0000 - FFFFF80270FC8FFF [ksecdd.sys]
xen|ModuleAdd: FFFFF80270E00000 - FFFFF80270F12FFF [clipsp.sys]
xen|ModuleAdd: FFFFF80271040000 - FFFFF8027104DFFF [cmimcext.sys]
xen|ModuleAdd: FFFFF80271050000 - FFFFF80271060FFF [werkernel.sys]
xen|ModuleAdd: FFFFF80271070000 - FFFFF8027107BFFF [ntosext.sys]
xen|ModuleAdd: FFFFF80271080000 - FFFFF80271164FFF [CI.dll]
xen|ModuleAdd: FFFFF80271170000 - FFFFF8027122AFFF [cng.sys]
xen|ModuleAdd: FFFFF80271230000 - FFFFF8027130CFFF [Wdf01000.sys]
xen|ModuleAdd: FFFFF80271310000 - FFFFF80271323FFF [WDFLDR.SYS]
xen|ModuleAdd: FFFFF80271330000 - FFFFF8027133CFFF [PRM.sys]
xen|ModuleAdd: FFFFF80271340000 - FFFFF80271365FFF [acpiex.sys]
xen|ModuleAdd: FFFFF80271370000 - FFFFF80271380FFF [WppRecorder.sys]
xen|ModuleAdd: FFFFF80271390000 - FFFFF8027139DFFF [msseccore.sys]
xen|ModuleAdd: FFFFF802713A0000 - FFFFF802713BAFFF [SgrmAgent.sys]
xen|ModuleAdd: FFFFF802713C0000 - FFFFF8027148AFFF [ACPI.sys]
xen|ModuleAdd: FFFFF80271490000 - FFFFF8027149BFFF [WMILIB.SYS]
xen|ModuleAdd: FFFFF802714A0000 - FFFFF802714B0FFF [WdBoot.sys]
xen|ModuleAdd: FFFFF802714C0000 - FFFFF802714D6FFF [WindowsTrustedRT.sys]
xen|ModuleAdd: FFFFF802714E0000 - FFFFF8027154AFFF [intelpep.sys]
xen|ModuleAdd: FFFFF80271550000 - FFFFF8027155AFFF [WindowsTrustedRTProxy.sys]
xen|ModuleAdd: FFFFF80271560000 - FFFFF8027156AFFF [IntelPMT.sys]
xen|ModuleAdd: FFFFF80271570000 - FFFFF80271584FFF [pcw.sys]
xen|ModuleAdd: FFFFF80271590000 - FFFFF802715ADFFF [sacdrv.sys]
xen|ModuleAdd: FFFFF80271650000 - FFFFF802717CBFFF [NDIS.SYS]
xen|ModuleAdd: FFFFF802715B0000 - FFFFF8027164DFFF [NETIO.SYS]
xen|ModuleAdd: FFFFF802717D0000 - FFFFF802717DAFFF [msisadrv.sys]
xen|ModuleAdd: FFFFF802717E0000 - FFFFF802717F5FFF [vdrvroot.sys]
xen|ModuleAdd: FFFFF80271800000 - FFFFF80271883FFF [pci.sys]
xen|ModuleAdd: FFFFF80271890000 - FFFFF802718C0FFF [xenbus.sys]
xen|ModuleAdd: FFFFF802718D0000 - FFFFF802718F0FFF [xen.sys]
xen|ModuleAdd: FFFFF80271900000 - FFFFF80271910FFF [xenfilt.sys]
xen|ModuleAdd: FFFFF80271920000 - FFFFF8027194EFFF [pdc.sys]
xen|ModuleAdd: FFFFF80271950000 - FFFFF80271967FFF [CEA.sys]
xen|ModuleAdd: FFFFF80271970000 - FFFFF802719A0FFF [partmgr.sys]
xen|ModuleAdd: FFFFF802719B0000 - FFFFF80271A7BFFF [spaceport.sys]
xen|ModuleAdd: FFFFF80271A80000 - FFFFF80271A99FFF [volmgr.sys]
xen|ModuleAdd: FFFFF80271AA0000 - FFFFF80271B02FFF [volmgrx.sys]
xen|ModuleAdd: FFFFF80271B10000 - FFFFF80271B1AFFF [intelide.sys]
xen|ModuleAdd: FFFFF80271B20000 - FFFFF80271B32FFF [PCIIDEX.SYS]
xen|ModuleAdd: FFFFF80271B40000 - FFFFF80271B5DFFF [mountmgr.sys]
xen|ModuleAdd: FFFFF80271B60000 - FFFFF80271B6CFFF [atapi.sys]
xen|ModuleAdd: FFFFF80271B70000 - FFFFF80271BABFFF [ataport.SYS]
xen|ModuleAdd: FFFFF80271BB0000 - FFFFF80271BE5FFF [stornvme.sys]
xen|ModuleAdd: FFFFF80271BF0000 - FFFFF80271CBAFFF [storport.sys]
xen|ModuleAdd: FFFFF80271CC0000 - FFFFF80271CDBFFF [xenvbd.sys]
xen|ModuleAdd: FFFFF80271CE0000 - FFFFF80271D19FFF [xencrsh.sys]
xen|ModuleAdd: FFFFF80271D20000 - FFFFF80271D2DFFF [xendisk.sys]
xen|ModuleAdd: FFFFF80271D30000 - FFFFF80271D51FFF [EhStorClass.sys]
xen|ModuleAdd: FFFFF80271D60000 - FFFFF80271DA1FFF [Wof.sys]
xen|ModuleAdd: FFFFF80271DB0000 - FFFFF80271E19FFF [WdFilter.sys]
xen|ModuleAdd: FFFFF80271E20000 - FFFFF80272130FFF [Ntfs.sys]
xen|ModuleAdd: FFFFF80272140000 - FFFFF8027214CFFF [Fs_Rec.sys]
xen|ModuleAdd: FFFFF80272150000 - FFFFF80272182FFF [ksecpkg.sys]
xen|ModuleAdd: FFFFF80272190000 - FFFFF802724AEFFF [tcpip.sys]
xen|ModuleAdd: FFFFF802724B0000 - FFFFF80272530FFF [fwpkclnt.sys]
xen|ModuleAdd: FFFFF80272540000 - FFFFF8027256FFFF [wfplwfs.sys]
xen|ModuleAdd: FFFFF80272580000 - FFFFF8027258AFFF [volume.sys]
xen|ModuleAdd: FFFFF80272590000 - FFFFF80272603FFF [volsnap.sys]
xen|ModuleAdd: FFFFF80272610000 - FFFFF80272635FFF [mup.sys]
xen|ModuleAdd: FFFFF80272640000 - FFFFF8027264FFFF [hwpolicy.sys]
xen|ModuleAdd: FFFFF80272660000 - FFFFF8027267EFFF [disk.sys]
xen|ModuleAdd: FFFFF80272680000 - FFFFF802726F4FFF [CLASSPNP.SYS]
xen|UnplugSetRequest: DISKS (0)
xen|UnplugSetRequest: NICS (0)
xenbus|DriverEntry: 9.1.9 (105) (10.09.2024)
xen|LogReadLogLevel: fail1 (c0000034)
xen|ConfigSetActive: PCI\VEN_5853&DEV_0002&SUBSYS_00015853&REV_01\18: \Registry\Machine\System\CurrentControlSet\Services\pci,65536,0,3,0
xenbus|FdoCreate: FFFF9A05CD4AB550 (XS0002 XENBUS) [ACTIVE]
xen|FiltersInstallClass: DEVCLASS_SYSTEM XENFILT
xen|FiltersInstallClass: DEVCLASS_HDC XENFILT
xenbus|FdoConnectInterrupt: FFFF9A05CD90E6E0: Shared LevelSensitive CPU 0:0 VECTOR a1
xenbus|FdoConnectInterrupt: FFFF9A05CD4AE6E0: DeviceExclusive Latched CPU 0:0 VECTOR b1
xenbus|FdoConnectInterrupt: FFFF9A05CD4AD6E0: DeviceExclusive Latched CPU 0:1 VECTOR b1
xenbus|FdoScan: ====>
xenbus|FdoSuspend: ====>
xenbus|FdoBalloon: ====>
xenbus|FdoPciHoleCreate: 000f0000 - 000f0fff
xenbus|EvtchnAbiAcquire: FIFO
xenbus|EvtchnInterruptEnable: CPU 0:0 (Vector = 177)
xenbus|EvtchnInterruptEnable: CPU 0:1 (Vector = 177)
xenbus|EvtchnInterruptEnable: CALLBACK VIA (Vector = 28)
xenbus|GnttabExpand: added references [00000020 - 000001ff]
xenbus|EvtchnFifoExpand: added ports [00000000 - 000003ff]
xenbus|__FdoVirqCreate: DEBUG: CPU 0:0
xenbus|PdoCreate: FFFF9A05CD8DB530 (VBD)
xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2
xenbus|PdoCreate: FFFF9A05CD8DA530 (VIF)
xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2
xenbus|PdoCreate: FFFF9A05CD8D94B0 (IFACE)
xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2
xenfilt|DriverEntry: 9.1.9 (105) (10.09.2024)
xenfilt|DriverAddDevice: FFFF9A05CD8E2360 UNKNOWN
xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE
xenfilt|DriverAddDevice: FFFF9A05CD8DE270 IDE
xenfilt|FdoCreate: FFFF9A05CD5DDDA0 (PCIIDE\IDEChannel\0)
xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE
xenfilt|DriverAddDevice: FFFF9A05CD8DE5A0 IDE
xenfilt|FdoCreate: FFFF9A05CD939DA0 (PCIIDE\IDEChannel\1)
xenfilt|PdoCreate: FFFF9A05CD97A750 (IDE\CdRomQEMU_QEMU_DVD-ROM_______________________2.5+____\1.1.0) 

Could you build a new template VM without any prebaked drivers to see if it's a driver or group policy issue?

I'm in the process of building a new template now using a vanilla ISO of Windows Server 2022. So far the VM is running fine; no BSOD after multiple reboots. I'll provide an update when patching is completed and I've turned the new VM to a template and built a VM from that.

@dinhngtu I can certainly do that. I will be back in the lab in about 2 hours and will retry.

Concerning the template, hmmm....I was having the same thoughts, so I've pulled down a Windows Server 2022 ISO directly from Microsoft and I'm gonna build a new template with it, to see if the problem resurfaces. It's possible there's something in the Secure Host Baseline that is causing problems with this platform (not sure if it's XCP-ng, or Xen). However, if that ends up being the case, it would be a bummer as that would weaken my case for adoption of the Vates VMS.

In any case, I'll report back in a couple of hours.

@dinhngtu My apologies for the delayed response. Got busy with several meetings.

So, I finally got the debugger to attach to the VM (after many tries). I was able to capture some logs, but was not able to get any response to the command you asked me to run: sxe -c "lm1mna (poi(rdx));g" ld

One thing I did was to re-install the xen drivers, so you'll see them get loaded in the debug logs.

Another thing to also point out, I cloned the snapshot of this VM to a new VM and observed the following:

• The cloned VM from snapshot booted up all the way into Windows, and I was able to login.

• I noticed that the Start Menu would not open (after several clicks), even though the Windows Explorer opens. So, I initiated a reboot - this time using the XO restart button.

• The VM went down for a reboot and got stuck in the same boot state (i.e., spinning wheel after the Windows splash logo). The VM's NIC comes up, however, because it starts responding to pings.

Anyway, here're the debug logs from the original VM:

************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   NonInteractiveNuget : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true
   EnableRedirectToChakraJsProvider : false

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.015 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 42

Microsoft (R) Windows Debugger Version 10.0.27725.1000 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

Opened 10.0.10.12
Waiting to reconnect...
Connected to Windows 10 20348 x64 target at (Tue Jan 28 20:04:11.011 2025 (UTC - 5:00)), ptr64 TRUE
Kernel Debugger connection established.
Symbol search path is: srv*
Executable search path is: 
Windows 10 Kernel Version 20348 MP (1 procs) Free x64
Edition build lab: 20348.859.amd64fre.fe_release_svc_prod2.220707-1832
Kernel base = 0xfffff803`1be00000 PsLoadedModuleList = 0xfffff803`1ca33d20
System Uptime: 0 days 0:00:00.000
Unable to load image ntkrnlmp.exe, Win32 error 0n2
xen|LogReadLogLevel: fail1 (c0000034)
xen|LogReadLogLevel: fail1 (c0000034)
xen|DllInitialize: 9.1.9 (105) (10.09.2024)
xen|AcpiFindRsdp: 0x00000000000EA020
xen|SystemInitialize: CPUs 2 / 4
xen|SystemGetStartOptions:  NOEXECUTE=OPTOUT  BOOTLOG  DEBUGPORT=COM1  BAUDRATE=115200  FVEBOOT=2670592  NOVGA  DEBUG
xen|SystemGetVersionInformation: KERNEL: 10.0 (BUILD 20348) PLATFORM WIN32_NT (x64)
xen|SystemGetVersionInformation: SUITES:
xen|SystemGetVersionInformation: - TERMINAL
xen|SystemGetVersionInformation: - SINGLEUSERTS
xen|SystemGetVersionInformation: TYPE: SERVER
xen|SystemGetMemoryInformation: RANGE[0] 00000000.00001000 - 00000000.0009ffff
xen|SystemGetMemoryInformation: RANGE[1] 00000000.00100000 - 00000000.ee4fbfff
xen|SystemGetMemoryInformation: RANGE[2] 00000000.ee4fd000 - 00000000.ee510fff
xen|SystemGetMemoryInformation: RANGE[3] 00000000.ee51b000 - 00000000.ef59afff
xen|SystemGetMemoryInformation: RANGE[4] 00000000.ef5ff000 - 00000000.effdefff
xen|SystemGetMemoryInformation: RANGE[5] 00000000.effff000 - 00000000.efffffff
xen|SystemGetMemoryInformation: RANGE[6] 00000001.00000000 - 00000004.0f7fffff
xen|SystemGetMemoryInformation: MaximumPhysicalAddress = 00000004.0f7fffff
xen|AcpiGetXsdt: 0x00000000FC00A5E0
xen|SystemProcessorDpc: ====> (0:0)
xen|SystemViridianInformation: ====>
xen|SystemViridianInformation: Interface Identifier: Hv#1
xen|SystemViridianInformation: Hypervisor Features:
xen|SystemViridianInformation:  - Partition Reference Counter
xen|SystemViridianInformation:  - Basic SynIC MSRs
xen|SystemViridianInformation:  - Synthetic Timer MSRs
xen|SystemViridianInformation:  - APIC Access MSRs
xen|SystemViridianInformation:  - Hypercall MSRs
xen|SystemViridianInformation:  - Virtual Processor Index MSR
xen|SystemViridianInformation:  - Partition Reference TSC MSR
xen|SystemViridianInformation:  - Timer Frequency MSR
xen|SystemViridianInformation: Recommendations:
xen|SystemViridianInformation:  - EOI, ICR and TPR access via MSR
xen|SystemViridianInformation:  - Use relaxed timing
xen|SystemViridianInformation:  - Retry spinlocks 2047 times
xen|SystemViridianInformation: Hardware Features:
xen|SystemViridianInformation:  - APIC overlay assist
xen|SystemViridianInformation:  - MSR bitmaps
xen|SystemViridianInformation:  - Second Level Address Translation (SLAT)
xen|SystemViridianInformation: <====
xen|SystemProcessorInitialize: Manufacturer: GenuineIntel
xen|SystemProcessorInitialize: APIC ID: 00
xen|SystemProcessorInitialize: PROCESSOR ID: 00
xen|SystemProcessorDpc: <==== (0:0)
xen|SystemProcessorDpc: ====> (0:1)
xen|SystemProcessorInitialize: Manufacturer: GenuineIntel
xen|SystemProcessorInitialize: APIC ID: 02
xen|SystemProcessorInitialize: PROCESSOR ID: 01
xen|SystemProcessorDpc: <==== (0:1)
xen|SystemGetTimeInformation: FALSE
xen|ModuleAdd: FFFFF8031BE00000 - FFFFF8031CE46FFF [ntoskrnl.exe]
xen|ModuleAdd: FFFFF80319DA0000 - FFFFF80319DA5FFF [hal.dll]
xen|ModuleAdd: FFFFF80319DB0000 - FFFFF80319DBDFFF [kdcom.dll]
xen|ModuleAdd: FFFFF803199F0000 - FFFFF80319D73FFF [mcupdate_GenuineIntel.dll]
xen|ModuleAdd: FFFFF8031F200000 - FFFFF8031F26CFFF [CLFS.SYS]
xen|ModuleAdd: FFFFF80319DC0000 - FFFFF80319DE7FFF [tm.sys]
xen|ModuleAdd: FFFFF8031F270000 - FFFFF8031F289FFF [PSHED.dll]
xen|ModuleAdd: FFFFF80319DF0000 - FFFFF80319DFAFFF [BOOTVID.dll]
xen|ModuleAdd: FFFFF8031F3B0000 - FFFFF8031F420FFF [FLTMGR.SYS]
xen|ModuleAdd: FFFFF8031F460000 - FFFFF8031F4C1FFF [msrpc.sys]
xen|ModuleAdd: FFFFF8031F430000 - FFFFF8031F458FFF [ksecdd.sys]
xen|ModuleAdd: FFFFF8031F290000 - FFFFF8031F3A2FFF [clipsp.sys]
xen|ModuleAdd: FFFFF8031F4D0000 - FFFFF8031F4DDFFF [cmimcext.sys]
xen|ModuleAdd: FFFFF8031F4E0000 - FFFFF8031F4F0FFF [werkernel.sys]
xen|ModuleAdd: FFFFF8031F500000 - FFFFF8031F50BFFF [ntosext.sys]
xen|ModuleAdd: FFFFF8031F510000 - FFFFF8031F5F4FFF [CI.dll]
xen|ModuleAdd: FFFFF8031F600000 - FFFFF8031F6BAFFF [cng.sys]
xen|ModuleAdd: FFFFF8031F6C0000 - FFFFF8031F79CFFF [Wdf01000.sys]
xen|ModuleAdd: FFFFF8031F7A0000 - FFFFF8031F7B3FFF [WDFLDR.SYS]
xen|ModuleAdd: FFFFF8031F7C0000 - FFFFF8031F7CCFFF [PRM.sys]
xen|ModuleAdd: FFFFF8031F7D0000 - FFFFF8031F7F5FFF [acpiex.sys]
xen|ModuleAdd: FFFFF8031F800000 - FFFFF8031F810FFF [WppRecorder.sys]
xen|ModuleAdd: FFFFF8031F820000 - FFFFF8031F82DFFF [msseccore.sys]
xen|ModuleAdd: FFFFF8031F830000 - FFFFF8031F84AFFF [SgrmAgent.sys]
xen|ModuleAdd: FFFFF8031F850000 - FFFFF8031F91AFFF [ACPI.sys]
xen|ModuleAdd: FFFFF8031F920000 - FFFFF8031F92BFFF [WMILIB.SYS]
xen|ModuleAdd: FFFFF8031F930000 - FFFFF8031F940FFF [WdBoot.sys]
xen|ModuleAdd: FFFFF8031F950000 - FFFFF8031F966FFF [WindowsTrustedRT.sys]
xen|ModuleAdd: FFFFF8031F970000 - FFFFF8031F9DAFFF [intelpep.sys]
xen|ModuleAdd: FFFFF8031F9E0000 - FFFFF8031F9EAFFF [WindowsTrustedRTProxy.sys]
xen|ModuleAdd: FFFFF8031F9F0000 - FFFFF8031F9FAFFF [IntelPMT.sys]
xen|ModuleAdd: FFFFF8031FA00000 - FFFFF8031FA14FFF [pcw.sys]
xen|ModuleAdd: FFFFF8031FA20000 - FFFFF8031FA3DFFF [sacdrv.sys]
xen|ModuleAdd: FFFFF8031FAE0000 - FFFFF8031FC5BFFF [NDIS.SYS]
xen|ModuleAdd: FFFFF8031FA40000 - FFFFF8031FADDFFF [NETIO.SYS]
xen|ModuleAdd: FFFFF8031FC60000 - FFFFF8031FC6AFFF [msisadrv.sys]
xen|ModuleAdd: FFFFF8031FC70000 - FFFFF8031FC85FFF [vdrvroot.sys]
xen|ModuleAdd: FFFFF8031FC90000 - FFFFF8031FD13FFF [pci.sys]
xen|ModuleAdd: FFFFF8031FD20000 - FFFFF8031FD50FFF [xenbus.sys]
xen|ModuleAdd: FFFFF8031FD60000 - FFFFF8031FD80FFF [xen.sys]
xen|ModuleAdd: FFFFF8031FD90000 - FFFFF8031FDA0FFF [xenfilt.sys]
xen|ModuleAdd: FFFFF8031FDB0000 - FFFFF8031FDDEFFF [pdc.sys]
xen|ModuleAdd: FFFFF8031FDE0000 - FFFFF8031FDF7FFF [CEA.sys]
xen|ModuleAdd: FFFFF8031FE00000 - FFFFF8031FE30FFF [partmgr.sys]
xen|ModuleAdd: FFFFF8031FE40000 - FFFFF8031FF0BFFF [spaceport.sys]
xen|ModuleAdd: FFFFF8031FF10000 - FFFFF8031FF29FFF [volmgr.sys]
xen|ModuleAdd: FFFFF8031FF30000 - FFFFF8031FF92FFF [volmgrx.sys]
xen|ModuleAdd: FFFFF8031FFA0000 - FFFFF8031FFAAFFF [intelide.sys]
xen|ModuleAdd: FFFFF8031FFB0000 - FFFFF8031FFC2FFF [PCIIDEX.SYS]
xen|ModuleAdd: FFFFF8031FFD0000 - FFFFF8031FFEDFFF [mountmgr.sys]
xen|ModuleAdd: FFFFF8031FFF0000 - FFFFF8031FFFCFFF [atapi.sys]
xen|ModuleAdd: FFFFF80320000000 - FFFFF8032003BFFF [ataport.SYS]
xen|ModuleAdd: FFFFF80320040000 - FFFFF80320075FFF [stornvme.sys]
xen|ModuleAdd: FFFFF80320080000 - FFFFF8032014AFFF [storport.sys]
xen|ModuleAdd: FFFFF80320150000 - FFFFF8032016BFFF [xenvbd.sys]
xen|ModuleAdd: FFFFF80320170000 - FFFFF803201A9FFF [xencrsh.sys]
xen|ModuleAdd: FFFFF803201B0000 - FFFFF803201BDFFF [xendisk.sys]
xen|ModuleAdd: FFFFF803201C0000 - FFFFF803201E1FFF [EhStorClass.sys]
xen|ModuleAdd: FFFFF803201F0000 - FFFFF80320231FFF [Wof.sys]
xen|ModuleAdd: FFFFF80320240000 - FFFFF803202A9FFF [WdFilter.sys]
xen|ModuleAdd: FFFFF803202B0000 - FFFFF803205C0FFF [Ntfs.sys]
xen|ModuleAdd: FFFFF803205D0000 - FFFFF803205DCFFF [Fs_Rec.sys]
xen|ModuleAdd: FFFFF803205E0000 - FFFFF80320612FFF [ksecpkg.sys]
xen|ModuleAdd: FFFFF80320620000 - FFFFF8032093EFFF [tcpip.sys]
xen|ModuleAdd: FFFFF80320940000 - FFFFF803209C0FFF [fwpkclnt.sys]
xen|ModuleAdd: FFFFF803209D0000 - FFFFF803209FFFFF [wfplwfs.sys]
xen|ModuleAdd: FFFFF80320A10000 - FFFFF80320A1AFFF [volume.sys]
xen|ModuleAdd: FFFFF80320A20000 - FFFFF80320A93FFF [volsnap.sys]
xen|ModuleAdd: FFFFF80320AA0000 - FFFFF80320AC5FFF [mup.sys]
xen|ModuleAdd: FFFFF80320AD0000 - FFFFF80320ADFFFF [hwpolicy.sys]
xen|ModuleAdd: FFFFF80320AF0000 - FFFFF80320B0EFFF [disk.sys]
xen|ModuleAdd: FFFFF80320B10000 - FFFFF80320B84FFF [CLASSPNP.SYS]
xen|UnplugSetRequest: DISKS (0)
xen|UnplugSetRequest: NICS (0)
xenbus|DriverEntry: 9.1.9 (105) (10.09.2024)
xen|LogReadLogLevel: fail1 (c0000034)
xen|ConfigSetActive: PCI\VEN_5853&DEV_0002&SUBSYS_00015853&REV_01\18: \Registry\Machine\System\CurrentControlSet\Services\pci,65536,0,3,0
xenbus|FdoCreate: FFFFB889490CBC00 (XS0002 XENBUS) [ACTIVE]
xen|FiltersInstallClass: DEVCLASS_SYSTEM XENFILT
xen|FiltersInstallClass: DEVCLASS_HDC XENFILT
xenbus|FdoConnectInterrupt: FFFFB889490D5910: Shared LevelSensitive CPU 0:0 VECTOR a1
xenbus|FdoConnectInterrupt: FFFFB889490D5B30: DeviceExclusive Latched CPU 0:0 VECTOR b1
xenbus|FdoConnectInterrupt: FFFFB889490D5D50: DeviceExclusive Latched CPU 0:1 VECTOR b1
xenbus|FdoScan: ====>
xenbus|FdoSuspend: ====>
xenbus|FdoBalloon: ====>
xenbus|FdoPciHoleCreate: 000f0000 - 000f0fff
xenbus|EvtchnAbiAcquire: FIFO
xenbus|EvtchnInterruptEnable: CPU 0:0 (Vector = 177)
xenbus|EvtchnInterruptEnable: CPU 0:1 (Vector = 177)
xenbus|EvtchnInterruptEnable: CALLBACK VIA (Vector = 28)
xenbus|GnttabExpand: added references [00000020 - 000001ff]
xenbus|EvtchnFifoExpand: added ports [00000000 - 000003ff]
xenbus|__FdoVirqCreate: DEBUG: CPU 0:0
xenbus|PdoCreate: FFFFB88948C7B540 (VBD)
xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2
xenbus|PdoCreate: FFFFB88948D7F540 (VIF)
xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2
xenbus|PdoCreate: FFFFB88948D7F870 (IFACE)
xenbus|PdoDumpRevisions: 09000008 -> SUSPEND v1 SHARED_INFO v3 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 09000009 -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v1 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000A -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v2 CONSOLE v1 EMULATED v2
xenbus|PdoDumpRevisions: 0900000B -> SUSPEND v1 SHARED_INFO v4 EVTCHN v9 DEBUG v1 STORE v2 RANGE_SET v1 CACHE v2 GNTTAB v4 UNPLUG v3 CONSOLE v1 EMULATED v2
xenfilt|DriverEntry: 9.1.9 (105) (10.09.2024)
xenfilt|DriverAddDevice: FFFFB8894910B360 UNKNOWN
xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE
xenfilt|DriverAddDevice: FFFFB8894916BAB0 IDE
xenfilt|FdoCreate: FFFFB88949170DA0 (PCIIDE\IDEChannel\0)
xenfilt|DriverGetEmulatedType: MATCH: Internal_IDE_Channel -> IDE
xenfilt|DriverAddDevice: FFFFB8894916C060 IDE
xenfilt|FdoCreate: FFFFB88949176DA0 (PCIIDE\IDEChannel\1)
xenfilt|PdoCreate: FFFFB889490E2510 (IDE\CdRomQEMU_QEMU_DVD-ROM_______________________2.5+____\1.1.0) 
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
*                                                                             *
*   You are seeing this message because you pressed either                    *
*       CTRL+C (if you run console kernel debugger) or,                       *
*       CTRL+BREAK (if you run GUI kernel debugger),                          *
*   on your debugger machine's keyboard.                                      *
*                                                                             *
*                   THIS IS NOT A BUG OR A SYSTEM CRASH                       *
*                                                                             *
* If you did not intend to break into the debugger, press the "g" key, then   *
* press the "Enter" key now.  This message might immediately reappear.  If it *
* does, press "g" and "Enter" again.                                          *
*                                                                             *
*******************************************************************************
nt!DbgBreakPointWithStatus:
fffff803`1c229560 cc              int     3
0: kd> sxe -c "lm1mna (poi(rdx));g" ld
0: kd> g