@kagbasi-ngc I figured out what the problem was, so wanted to post it for the benefit of the community.

The root cause for me ended up being a firewall issue on the Windows Server where I was running the NFS server role. While I hadn't made any changes to the firewall, the profile of the network interface connected to the management network had changed from Domain to Public/Private, thus, Windows Defender Firewall applied the most restrictive settings since it saw that the NIC wasn't connected to the Domain profile (even though it physically was).

This post on Serverfault (https://serverfault.com/a/647201/189248) clued me in and I looked at the Network Location Awareness service (NLA). It was set automatic and was running, so I restarted it and this reverted the profile of the NIC back to Domain and caused Windows Defender Firewall to apply the previously configured rules.

It is important to note that this situation won't be common, as most of you won't be running your NAS on a dual-homed domain controller that is also running the DHCP server role. Hope this helps someone.

@Prilly Looks like @lawrencesystems has a good explainer video on YouTube at https://youtu.be/weVoKm8kDb4?si=QH93rOKaXLglrIbS

Check it out when you get a chance, thanks.

@nathanael-h Yeah, I've already done that several times and no luck.

What I'm gonna try now is see if I can boot it with the Windows installation disk, go into recovery and access the WinPE command line, and see if I can run xenbootfix.exe to clean out the guest tools/drivers.

If that doesn't work, I'll just blow away the VM and start over. I just hate doing this, cos it doesn't give us the chance to figure out why this happened.

@olivierlambert Cool, I'll run the other two when I get back. I'll kick them off simultaneously.

@olivierlambert No sir, I have not. I'll try that and report back shortly.

@stormi Sorry for the delayed reply. I did as you requested, and below are the resulting log files. I started the VM at timestamp Dec 17 14:43:34 and stopped it about 2 minutes later.

• daemon.log
• xensource.log

If I can be of assistance, don't hesitate to ask please. Thank you.

@olivierlambert No, it's the other way around. Once the subject user is a memberof 3 or more groups, the problem occurs.

@manilx Yeah, not doing it in production, just in a test environment. Plus I only did it because I actually found a write-up by the Vates team on how to get it done, which confirmed to me that while they don't recommend it for production, they understand that it has some merit in a testing environment and want to see it working.

That said, I definitely hear your admonition and would never do it in a production setting.

@olivierlambert Certainly strange, then, that the same settings and same plugin works in XOA but not in XOCE. We could argue perhaps some configuration variances within the Domain Controllers, but given that I'm the same person that configured both environments - I'm highly confident that I stood up the DCs in the same way. Only things that would obviously be different would be the Domain Name and OU Structure.

I'll confirm the Forest and Domain functional levels later this evening and perhaps run an XOA trial and see if it works. I'll report back.

@olivierlambert So XOA is running on local storage of host #1 (VMH01) and all the VMs run off an NFS datastore which I've checked and is not in read-only mode or full (has about 60% free space).

What I haven't checked is whether the partition being used as the Local SR on VMH01 is perhaps the culprit. I ran xe check yesterday and the only three things it complained about were NTP, Updates, and not having Internet connectivity.

I should be in the office in about an hour, so I'll inspect and report back.

@dinhngtu Thanks, that helped. Download has started, once it completes, I'll get it installed and retry.

I'd also found this - https://gist.github.com/alexrintt/9502728c6d75d27e2885c9cdaab0e550

@dinhngtu Thanks for the instructions. I'm back in the lab now, and I'm going to try as you've suggested.

Because the lab is airgapped (disconnected from the Internet), I wasn't able to install the latest version of Windbg, which is why I opted to use the version from the Windows SDK ISO. I need to figure out how to get the latest Windbg installed in an offline system, then I'll be able to proceed.

@dinhngtu Thank you for breaking it down, this worked and I was able to get the commands to be accepted on the VM. Whether they actually took effect, I can't tell.

Here's what I observed:

• I prepped the host with the xe vm-param-set uuid=<uuid> platform:hvm_serial=tcp::<port>,server,nodelay,nowait command and opened the firewall (using tcp port 7001).

• I disabled Secure Boot on the VM, prepped a PuTTY session to connect to my host on TCP port 7001

• I noticed that whenever the VM starts booting, and I simultaneously initiate the putting session, it seems like a connection is made but no data is returned to the screen. The session terminates as soon as the VM runs into the BSOD and shuts down.

I recorded a video of it: https://photos.app.goo.gl/wzJSwfoJJJWwtGk57

Next thing with Windbg. This is my first time using it, so please pardon my ignorance. However, I am unable to attach to the kernel debug, as per the instructions you provided. I see a Kernel Debug menu item and when I click on it, I don't see any place where I can enter the connection string specified in the instructions you provided.

Here's another video capture: https://photos.app.goo.gl/ZnjVQjo5Hk589dQ96

We're making progress, and I really appreciate your patience, thank you.

@dinhngtu While following the instructions you shared on how to expose the VM's serial console, I'm having trouble with this command "bcdedit /dbgsettings serial debugport:1 baudrate:115200". I had to modify it slightly to specify the /store parameter (as indicated previously).

Since I don't have the ability to run this command inside the OS, I am running it from a command shell that I accessed via the Troubleshooting menu (from an installation disk). It accepted the first command "bcdedit /store S:\EFI\Microsoft\Boot\BCD /set {default} debug on" but won't accept "bcdedit /store S:\EFI\Microsoft\Boot\BCD /set {default} dbgsettings serial debugport:1 baudrate:115200". It throws the following error:

The element data type specified is not recognized or does not apply to the specified entry.

What am I missing here?

Second issue - I attempted to get the host ready by setting the parameters (as per the instructions) and opened the firewall port (I used 7001). However, when I attempt to connect to the host via SSH on tcp\7001, the connection is refused.

@dinhngtu Roger that, I'll try it now and see what happens.

@dinhngtu said in VM Failing to Reboot:

Also, one way to verify the issue is to install MediaEnable on another VM.

So, I built a new VM from the same template, renamed it and joined it to the domain (both actions required a reboot which was initiated from inside the OS). I then placed it in the correct OU and rebooted so it will grab the correct GPOs (this reboot action was also initiated from inside the OS). Finally, I installed MailEnable and when it finished I initiated a final reboot from inside the OS.

None of the reboots caused the OS to fail to boot, and it was built from the same template (which has the Citrix Guest Tools v9.4 installed). Below is a link to a video I took of the test.

https://photos.app.goo.gl/Uw7WgFRY1BEem8gA8

@dinhngtu said in VM Failing to Reboot:

@kagbasi-ngc There's no slash in debug while using the /set command:

I had already tried that, but I think I did it on {bootmgr} - so I'll retry on {default} when I get back in the lab.

Update as of 09:58AM EDT:
I tried the command against the {default} identifier, and it worked. However, I didn't see any change in behavior during the boot process. Should I continue with the instructions for exposing the VMs serial console?

@dinhngtu Thanks, I did that but getting an error message that The system cannot find the file specified

@dinhngtu Sure, it's possible - though not likely. However, I'm willing to entertain you and I would uninstall them, except that I can't boot into the OS.

Anyway, I tried what you suggested by attempting to enable BCD Debug and it didn't work - got an error (even though the path is correct) :

@dinhngtu I installed the Citrix Tools into my template and then built the VM from the template (as I've always done). However, immediately prior to the reboot, I had just finished installing MediaEnable (email server) and was just trying to reboot to get some Group Policies to take effect.

Prior to that, I'd already rebooted the VM many times and not had any issues, however, I was always rebooting using the buttons in XOA not directly from within the OS.

I'm gonna go try what you've suggested now and see what happens.