VM Failing to Reboot

kagbasi-ngc

Good-day Folks,

MY ENVIRONMENT:

XCP-ng: 3 nodes running v8.3.0
XOA: v5.98.1 Build 20241024 (air-apped)
NFS Server: Windows Server 2016
Networking: Management network of hosts exist within the same VLAN as NFS Server

THE ISSUE:
Today, while working with a newly created VM (created without Fast Copy checked), I initiated a reboot from inside the VM. This appears to have gone off without a hitch; the VM shutdown normally and went for a reboot, but now will not finish booting up the OS. It gets past the POST process but the OS won't load. Nothing, that I can see, appears out of the ordinary. The VDI is attached, secure boot is enabled (as it is on all other VMs), and the secure boot certs have been properly installed on the pool (as per documentation).

Here's a link to a video I captured of the incident - has anybody experienced this and can share some guidance?

https://photos.app.goo.gl/75GS8VdqG8kLuWrg6

nathanael-h

Hello, are the guest tools and drivers installed on this Windows VM https://docs.xcp-ng.org/vms/#windows ?

kagbasi-ngc

nathanael-h Yes they are. Normally, I initiate reboots and shutdown operations from within XOA. However, I didn't expect that doing it from within the OS of the VM would result in this. Makes me quite nervous.

I allowed the VM to run all weekend and it still hasn't booted - still spinning. Any suggestions on how to recover it?

nathanael-h

kagbasi-ngc I am not sure, but I would force shutdown it from XO, then boot it again. If it does not work I'd try some fail safe mode, but I did not used Windows VMs that much this last years. Maybe someone else would be more helpful...

kagbasi-ngc

nathanael-h Yeah, I've already done that several times and no luck.

What I'm gonna try now is see if I can boot it with the Windows installation disk, go into recovery and access the WinPE command line, and see if I can run xenbootfix.exe to clean out the guest tools/drivers.

If that doesn't work, I'll just blow away the VM and start over. I just hate doing this, cos it doesn't give us the chance to figure out why this happened.

kagbasi-ngc

nathanael-h So I just used the xenbootfix.exe to clean out the guest tools/drivers, however, that hasn't helped. At least now the boot process is failing to a BSOD, but what's interesting is that I get a different stop code.

I have a snapshot that I can revert to and start over, but I really don't want to (if there's any chance of fixing this).

BSOD (after a normal startup

BSOD (after a recovery startup):

nathanael-h

kagbasi-ngc it looks like the same issue you had in https://xcp-ng.org/forum/topic/10140/bsod-with-inaccessible-boot-device-after-uninstalling-the-xcp-ng-guest-tools/ is the solution suggested by dinhngtu works here?

kagbasi-ngc

nathanael-h Yes it seems so, except that this time I’m unable to boot successfully into Windows.

dinhngtu

kagbasi-ngc Can you attach a kernel debugger using the VM's serial console?

kagbasi-ngc

dinhngtu Ooo, sounds interesting. Not sure how I would do that, but certainly open to learning.

dinhngtu

kagbasi-ngc You'll find the instructions here: https://github.com/xcp-ng/xcp-ng-org/blob/wintools/docs/troubleshooting/windows-pv-tools.md#connecting-to-guests-using-serial-console

kagbasi-ngc

dinhngtu Thanks. I read through the instructions, and given that I'm unable to boot into Windows, I don't think I'll be able to complete this step:

To configure kernel debugging on Windows, disable Secure Boot and BitLocker then run the following commands within the VM:
bcdedit /debug on
bcdedit /dbgsettings serial debugport:1 baudrate:115200

Is there an alternative?

dinhngtu

kagbasi-ngc You should be able to get into debugging mode via the Advanced Boot Options menu (spam F8 at boot). You'll still need to enable Secure Boot. I'm not sure if you'll be able to connect without running the /dbgsettings command, but it's worth a try. Note that you'll need to have WinDbg ready and connect it as soon as you select debugging mode.

If all that fails, can you use Safe Mode or Last Known Good Configuration?

kagbasi-ngc

dinhngtu Okay, gonna give it a shot now. Will report back shortly.

kagbasi-ngc

dinhngtu Unfortunately, no amount of smashing of the F8 key got me into the Advanced Boot Options menu, so I gave up on that.

Instead, I've booted up with Hiren's Boot Disc, and I'm about to see if perhaps I can uninstall the guest tools this way and try again. Any pointers are welcome.

dinhngtu

kagbasi-ngc You can enable debugging from within Hiren's by mounting the Windows EFI system partition to e.g. S: then running bcdedit /store S:\EFI\Microsoft\Boot\BCD /debug on

You can also try renaming the .sys files but normally XenBootFix should have been sufficient to disable all Xen drivers.

I forgot to ask, how did you install guest tools onto your VM, and did you install anything else to it (e.g. some 3rd-party apps) before rebooting?

kagbasi-ngc

dinhngtu I installed the Citrix Tools into my template and then built the VM from the template (as I've always done). However, immediately prior to the reboot, I had just finished installing MediaEnable (email server) and was just trying to reboot to get some Group Policies to take effect.

Prior to that, I'd already rebooted the VM many times and not had any issues, however, I was always rebooting using the buttons in XOA not directly from within the OS.

I'm gonna go try what you've suggested now and see what happens.

dinhngtu

kagbasi-ngc Is it possible that your GPOs or MediaEnable that are causing the BSOD? Could you try to eliminate the causes?

kagbasi-ngc

dinhngtu Sure, it's possible - though not likely. However, I'm willing to entertain you and I would uninstall them, except that I can't boot into the OS.

Anyway, I tried what you suggested by attempting to enable BCD Debug and it didn't work - got an error (even though the path is correct) :

dinhngtu

kagbasi-ngc It's not the BCD database path but the BCD entry identifier. Try bcdedit /store ...\BCD /enum to get the entry's identifier, then bcdedit /store ...\BCD /set <identifier> /debug on