Secure Boot was not enabled in the VM. I enabled Secure Boot in the VM and was able to install KB5012170 without any problem.

I tested further to see if there were any issues related to enabling then disabling Secure Boot in the VM. I did not experience any problems booting the VM after disabling Secure Boot. There were no problems booting the VM after moving it to a pool where the default UEFI Certificates had not been installed.

For anyone wanting to resolve the KB5012170 update error, here are the steps I took:

• On the pool/host for the VM, install the UEFI Certificates with secureboot-certs install
• Shut down the problem VM
• Enable Secure Boot on the VM. I do this via Xen Orchestra but it can also be done with xe vm-param-set uuid=[uuid of VM] platform:secureboot=true
• Boot the VM
• Apply the KB5012170 update
• Shut down the VM
• Disable Secure Boot on the VM via XO or xe vm-param-set uuid=[uuid of VM] platform:secureboot=false
• Boot the VM

https://xcp-ng.org/docs/guides.html#guest-uefi-secure-boot is a very thorough guide on Secure Boot in XCP-ng.

Thanks for the help @stormi

I believe I have the definitive cause for this 'random host reboot' issue.

After 6 months of problem-free operation, I have experienced the host reboot issue again on this server. The host was running only Linux VMs, so the theory of Windows VMs on the host contributing to the reboot issue has proven false. As with each time before, there are no indications in any relevant log files that the host is going to reboot. I think at this point I can definitively say that the reboot is caused by a faulty SuperMicro motherboard.

I've learned my lesson: use HPE servers! This SuperMicro system will be melted down for scrap.

Since I last posted on this topic, I've found that the random reboots only occur when there are Windows Server VMs on the host (Tested with 2019 and 2022). The issue will not occur when running Linux VMs.

My issue seems very similar to the problem described (and solved) in https://xcp-ng.org/forum/topic/6683/windows-server-2019-sporadic-reboot/7

The difference is that in my case, the host restarted and in the other post, the poster reports that the VMs are restarting. Since the poster also tested RAM and found no problems but was able to solve the issue by replacing a suspected DIMM, that information may be useful in the host reboot scenario that I experience.

FYI, I have not replaced the RAM yet and may not actually do it since the server in question is aging and will likely be replaced (with HP hardware) soon.

I've not been able to find information on the signature used by the UEFI bootloader and if that is on the DBX update in KB5012170. Since my original post, Microsoft has updated the Known Issues documentation for KB5012170 and it seems that this problem is now 'known' and has a proposed resolution of, "We are presently investigating and will provide an update in an upcoming release."

So at this point, it appears Microsoft is investigating this as an issue Microsoft needs to resolve, not an issue with the bootloader itself.

Reference: https://support.microsoft.com/en-us/topic/kb5012170-security-update-for-secure-boot-dbx-august-9-2022-72ff5eed-25b4-47c7-be28-c42bd211bb15
(see the third issue listed in Known Issues, which didn't exist until after my initial post)