HA causes reboot of xcp-ng nodes

[carloum70]

Hi all,

I need some help regarding HA.
I have a 3 node cluster running xcp-ng 8.3 (002 is the master)

[13:55 dacshyp002 ~]# xe host-list
uuid ( RO)                : 6b99f1ab-6f4c-4a8d-b766-d16a8a942bdf
          name-label ( RW): dacshyp003
    name-description ( RW): Default install


uuid ( RO)                : d99e150c-079a-4092-8909-ad1a36e07dec
          name-label ( RW): dacshyp001
    name-description ( RW): Default install


uuid ( RO)                : 41a2a448-a5dc-44c6-be44-c07540d75c60
          name-label ( RW): dacshyp002
    name-description ( RW): Default install

Last Thursday I enabled HA on the pool and also on some om the VM's. On Friday I also did an "rolling pool update"
Today I had a "spontaneous" reboot of the dacshyp001 and dacshyp003.
Boot time of dacshyp001:

[Mon Sep 21 11:21:31 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

Around that time I also see the following message in the xensource.log:

Sep 21 11:21:08 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:28 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:48 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead

I will attached all logging around this time.
I am trying to understand what triggered the reboot and how I can troubleshoot this.

I am running XO community commit 6a441 . I am aware this is not the latest version. Before upgrading I want to know the cause of the reboot.

Also ignore the error:

/var/lib/xcp/xapi|dispatch:VDI.get_by_uuid D:1f5b4c8d329e|backtrace] VDI.get_by_uuid D:9df5c0255224 failed with exception Db_exn.Read_missing_uuid("VDI", "", "9219113f-65e4-4368-b42d-1b8bdc7614a9")

I will create another ticket for this.
HA-problem.txt
Thanks in advance.
Carlo

[poddingue]

From what I read in the docs (HA isn't my strong suit), a host in an HA pool that loses its heartbeat in certain ways is designed to reboot itself, which they call self-fencing, so this may well be HA doing its job rather than something crashing.

The log you attached is from the master and starts at 11:21:00, and dacshyp001 is already marked as not live in the very first liveset at 11:21:08, so I think whatever triggered it happened just before that and isn't in this file.

The three Setting host dacshyp001 to dead lines look to me like one event being re-checked every 20 seconds while 001 was still coming back, though I could be misreading that.

I also suspect 003 went down at a different moment, because it still shows as alive in that same liveset. There's a doc section for this case, https://docs.xcp-ng.org/troubleshooting/troubleshooting-ha#my-host-rebooted-why-did-it-reboot, which points at /var/log/xha.log on the host that rebooted.

Could you post that file from dacshyp001 and dacshyp003 for the few minutes before each reboot, plus 003's boot time?

[carloum70]

dacshyp003 booted at

[Mon Sep 21 05:44:37 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

This is the part of the xensource logging of the dacshyp001 before the reboot:

Sep 21 11:17:52 dacshyp001 xapi: [debug||17 ha_monitor|HA monitor D:58d1715e0726|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Same behavior on dacshyp003.
I also attached the xha.log from both nodes.

xha_log_001.txt xha_log_003.txt

[carloum70]

I also checked the xha.log on the dacshyp002:

Sep 21 11:39:49 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30408.
Sep 21 11:40:09 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:40:29 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=31466.
Sep 21 11:40:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:46:30 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=35178.
Sep 21 11:46:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:47:10 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33233.
Sep 21 11:47:30 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:55:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=40853.
Sep 21 11:56:11 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:56:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32914.
Sep 21 11:56:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:59:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33377.
Sep 21 11:59:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:01:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38588.
Sep 21 12:02:12 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=38125.
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38323.
Sep 21 12:10:32 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:19:33 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=43934.
Sep 21 12:19:53 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:20:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32989.
Sep 21 12:20:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=42077.
Sep 21 12:21:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:53 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31325.
Sep 21 12:22:13 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:25:54 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=34497.
Sep 21 12:26:14 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:26:34 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32558.
Sep 21 12:26:54 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:27:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30616.
Sep 21 12:27:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:31:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=33975.
Sep 21 12:31:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:38:15 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=34797.
Sep 21 12:38:35 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=37000.
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=40176.
Sep 21 12:54:16 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:16 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=36380.
Sep 21 12:56:36 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:57 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31439.
Sep 21 12:57:17 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 

I can conclude the heartbeat communication is generally unstable in my HA pool. Tomorrow I check the logging of our switch/router.