thanks for your tips - I have been trying some time and did not have any luck with it. However one time the whole xcp-ng host went down and rebooted and then I got an entry in the Supermicro IPMI with an uncorrectable ECC memory error. Memtest x86 did not report any faults but after changing that particular DIMM module the reboots and triple faults just stopped. So I think it is safe to assume that these error occured because of a hardware failure and the VM triple faulted because of that memory segment not being available.

I just wanted to report back for anyone encountering this. Thank you for your help.

thanks for your tips - I have been trying some time and did not have any luck with it. However one time the whole xcp-ng host went down and rebooted and then I got an entry in the Supermicro IPMI with an uncorrectable ECC memory error. Memtest x86 did not report any faults but after changing that particular DIMM module the reboots and triple faults just stopped. So I think it is safe to assume that these error occured because of a hardware failure and the VM triple faulted because of that memory segment not being available.

I just wanted to report back for anyone encountering this. Thank you for your help.

You're right, this obviously can be a bug in Citrix drivers.

To confirm this hypothesis you can desinstall Citrix Tools. There will be some impact on performances as you will run on emulated hardware and not paravirtualized, but normally this should work.

BACKUP all important data (snapshot would be a good idea) and follow this procedure https://xcp-ng.org/docs/guests.html#upgrade-from-citrix-xenserver-client-tools to desinstall and clean-up all Citrix add-on software. You would probably want to stop at step 5 if you're not using the scripts and proceeding with manual desinstall.

Hope this helps

thanks a lot for your insight. I think I can rule out the second option (normal reboot). One of the VMs (the one the crash dump above is from) is doing scheduled reboots every night. Also Windows updates do occur from time to time with regular reboots.

I think you are right with option 1. As far as my limited understanding of x86 assembly goes a triple fault is the second exception during interrupt handling. The interrupt table should be in the Windows kernel memory space, so only device drivers should be able to corrupt it (if I am not mistaken).
Does anybody know a way to identify driver problems in a Windows XEN VM? For example in the register dump above the instrcution pointer is present. Would it be possible to save Windows kernel memory space from time to time and when a fault occurs backreference the faulty driver? Actually, I am not sure about Adress Space Layout Randomization, but I think it is possible to turn it off.
The reboots are not very often, but they are a problem for me the last couple of months - so any help or ideas are greatly appreciated!

Cheers,
phipra

Well, this is the tricky one.

The triple fault could normally be two things:

• heavy memory corruption (the IDT was corrupted)
• the normal reboot (if ACPI reboot is not available)

My question is - can this Windows reboot be a planified Windows reboot (for example related with Windows update mechanism, or something like that ....)?

It can be obviously be a memory corruption (and possibly done by citrix-vm-tools drivers), but it'll be VERY hard to debug this from a memory dump (and actually AFAIK xen doesn't provide a guest memory dump).

My suggestions would be - try to enable/disable some Windows Services (disable the Update?) to see if there's some changes.

Sorry for this very poor insight, but this is related to my rather poor knowledge about Windows OS.

Invoking @andSmv to take a look on Monday if he can