HA causes reboot of xcp-ng nodes

[carloum70]

Hi all,

I need some help regarding HA.
I have a 3 node cluster running xcp-ng 8.3 (002 is the master)

[13:55 dacshyp002 ~]# xe host-list
uuid ( RO)                : 6b99f1ab-6f4c-4a8d-b766-d16a8a942bdf
          name-label ( RW): dacshyp003
    name-description ( RW): Default install


uuid ( RO)                : d99e150c-079a-4092-8909-ad1a36e07dec
          name-label ( RW): dacshyp001
    name-description ( RW): Default install


uuid ( RO)                : 41a2a448-a5dc-44c6-be44-c07540d75c60
          name-label ( RW): dacshyp002
    name-description ( RW): Default install

Last Thursday I enabled HA on the pool and also on some om the VM's. On Friday I also did an "rolling pool update"
Today I had a "spontaneous" reboot of the dacshyp001 and dacshyp003.
Boot time of dacshyp001:

[Mon Sep 21 11:21:31 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

Around that time I also see the following message in the xensource.log:

Sep 21 11:21:08 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:28 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:48 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead

I will attached all logging around this time.
I am trying to understand what triggered the reboot and how I can troubleshoot this.

I am running XO community commit 6a441 . I am aware this is not the latest version. Before upgrading I want to know the cause of the reboot.

Also ignore the error:

/var/lib/xcp/xapi|dispatch:VDI.get_by_uuid D:1f5b4c8d329e|backtrace] VDI.get_by_uuid D:9df5c0255224 failed with exception Db_exn.Read_missing_uuid("VDI", "", "9219113f-65e4-4368-b42d-1b8bdc7614a9")

I will create another ticket for this.
HA-problem.txt
Thanks in advance.
Carlo

[poddingue]

From what I read in the docs (HA isn't my strong suit), a host in an HA pool that loses its heartbeat in certain ways is designed to reboot itself, which they call self-fencing, so this may well be HA doing its job rather than something crashing.

The log you attached is from the master and starts at 11:21:00, and dacshyp001 is already marked as not live in the very first liveset at 11:21:08, so I think whatever triggered it happened just before that and isn't in this file.

The three Setting host dacshyp001 to dead lines look to me like one event being re-checked every 20 seconds while 001 was still coming back, though I could be misreading that.

I also suspect 003 went down at a different moment, because it still shows as alive in that same liveset. There's a doc section for this case, https://docs.xcp-ng.org/troubleshooting/troubleshooting-ha#my-host-rebooted-why-did-it-reboot, which points at /var/log/xha.log on the host that rebooted.

Could you post that file from dacshyp001 and dacshyp003 for the few minutes before each reboot, plus 003's boot time?

[carloum70]

dacshyp003 booted at

[Mon Sep 21 05:44:37 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

This is the part of the xensource logging of the dacshyp001 before the reboot:

Sep 21 11:17:52 dacshyp001 xapi: [debug||17 ha_monitor|HA monitor D:58d1715e0726|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
...
Same behavior on dacshyp003. 
I also attached the xha.log from both nodes.

[xha_log_001.txt](/forum/assets/uploads/files/1789999785333-xha_log_001.txt) [xha_log_003.txt](/forum/assets/uploads/files/1789999785339-xha_log_003.txt)