carloum70

Some update:

After removing the ip-address from eth4 the following messages disappeared

 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30408.
[info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 

By the way I forgot to mention that each time I enable HA the following warning appears in xha.log

Sep 29 13:31:56 CEST 2026 [warn] BM: cannot open bonding status file (/proc/net/bonding/bond0). (2)
Sep 29 13:31:56 CEST 2026 [info] BM: this is not bonded. Terminating bonding thread.

bond0 (eth2 + eth3) is the management interface and also used for the HA.

@bvitnik I think there is some confusing.
During the initial installation we have have used the eth4 (172.28.4.11) as the management interface. Later on we created a bond with eth2 and eth3 and changed the management interface to this bond using the xsconsole, and also used the same ip-address (172.28.4.11). We thought that the IP address had also been removed from eth4 during the reconfiguration, but we didn't check it afterwards. It turns out that this was not the case.
So now I removed the ip from eth4 using the xe pif-reconfigure-ip command.

Next step is to use the sysctl workaround @john.c mentioned in an earlier post and enable HA again.

(I also think it would be wise to use dedicated bond for the heartbeat.)

@john.c We are using a LCAP bond.
For my understanding, if I remove the IP settings from eth4, the problem will still not be solved because of the known issue with the bond that you explained in “Weak Host Model & HA Heartbeat Failures.”
(At the moment we have HA disabled.)

@john.c now I remember, during the installation we used the eth4 as the management interface.

[16:54 dacshyp002 ~]# cat /etc/firstboot.d/data/management.conf 
LABEL='eth4'
MODE='static'
IP='172.28.4.11'
NETMASK='255.255.252.0'
GATEWAY='172.28.4.1'
MODEV6='none'
DNS='8.8.8.8'

Afterwords we created a bond bond0 with eth2/eth3 and used this as the management interface.
The xe pif-list command shows the following:

device ( RO)                   : bond0
               management ( RO): true
    IP-configuration-mode ( RO): Static
                       IP ( RO): 172.28.4.11

device ( RO)                   : eth4
               management ( RO): false
    IP-configuration-mode ( RO): Static
                       IP ( RO): 172.28.4.11

Is it that easy to remove the IP address from the eth4 interface? Because it make no sense to have duplicate ip-addresses.
I think we've had duplicate IP addresses all along, but we only noticed the issue after enabling HA.
By the way what is the correct way to change the management interface? We used the xsconsole.

@tjkreidl

According to xsconsole the management interface is bond0

Current Management Interface          
                                              
Device           bond0                
MAC Address	 30:3e:a7:1d:b0:90    
DHCP/Static IP   Static               
IP address       172.28.4.11          
Netmask          255.255.252.0        
Gateway          172.28.4.1           
Hostname         dacshyp002  

According the output of xe pif-list

uuid ( RO)                  : 0761d887-268e-5cf0-401b-a08ad7c419aa
                device ( RO): bond0
                   MAC ( RO): 30:3e:a7:1d:b0:90
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
             host-uuid ( RO): 41a2a448-a5dc-44c6-be44-c07540d75c60

Let's check the network

# xe network-list uuid=28d51793-0226-cf3f-75d3-ed7c5a9d8b33
uuid ( RO)                : 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
          name-label ( RW): mgmt-bond
    name-description ( RW): 
              bridge ( RO): xapi1

At linux level:

# ip addr show xapi1
15: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xapi1
       valid_lft forever preferred_lft forever

xapi1 is a bond of et2 and eth3. So far so good.
But if I am running the xe-reset-networking command

Your network will be re-configured as follows:

Management interface:   eth4
Reset interface name rules: Yes
IP configuration mode:  dhcp
IPv6 configuration mode:none

If you want to change any of the above settings, type 'no' and re-run
the command with appropriate arguments (use --help for a list of options).

Type 'yes' to continue.
Type 'no' to cancel.
no

So for some reason xcp-ng thinks the eth4 is management interface, which also has the ip-address 172.28.4.11/22 .
By the way --reset-primary is not a valid option:

# xe-reset-networking --reset-primary
Usage: xe-reset-networking [options]

xe-reset-networking: error: no such option: --reset-primary

I did some further investigation and I think there is something wrong with the ip-settings of the management interface.

At the xsconsole I see the following:

Management Network Parameters    

Device        bond0         
address       172.28.4.11    
Netmask       255.255.252.0   
Gateway       172.28.4.1

So it's using the bond0 interface.

# xe pif-list host-uuid=41a2a448-a5dc-44c6-be44-c07540d75c60

uuid ( RO)                  : 0761d887-268e-5cf0-401b-a08ad7c419aa
                device ( RO): bond0
                   MAC ( RO): 30:3e:a7:1d:b0:90
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
             host-uuid ( RO): 41a2a448-a5dc-44c6-be44-c07540d75c60

This is part of the output of xe bond-list

# xe bond-list
uuid ( RO)      : 79c525c3-22f9-04ce-4a80-21d18ef65ebc
    master ( RO): 0761d887-268e-5cf0-401b-a08ad7c419aa
    slaves ( RO): 7ac18886-e0a5-057b-bac0-36b909588dba ; 2f18fc9a-8c1c-dfce-9416-fa8c657c5f63 

I already checked that:

7ac18886-e0a5-057b-bac0-36b909588dba --> eth3
2f18fc9a-8c1c-dfce-9416-fa8c657c5f63 --> eth2

And now the confusing part. This is part of the output of the ip a command:

2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:91 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff


13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
       valid_lft forever preferred_lft forever

As you can see the xenbr4 has also the management ip-address and has the same mac-address as the eth4 interface.
OK check the following commands:

[17:21 dacshyp002 ~]# ip addr show xapi1
15: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xapi1
       valid_lft forever preferred_lft forever
[17:21 dacshyp002 ~]# ip addr show xenbr4
13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
       valid_lft forever preferred_lft forever

And If I look at the routing table:

# ip route
default via 172.28.4.1 dev xapi1 
172.18.8.0/23 dev xenbr0 proto kernel scope link src 172.18.8.11 
172.18.10.0/23 dev xenbr7 proto kernel scope link src 172.18.10.11 
172.28.4.0/22 dev xenbr4 proto kernel scope link src 172.28.4.11 
172.28.4.0/22 dev xapi1 proto kernel scope link src 172.28.4.11 

According to me the node has 2 interfaces xenbr4 / xapi1 with the same ip-address and also 2 different routes for the same ip-range 172.28.4.0/22.
And that's why the heartbeat has some issues. Can someone confirm this ?
By the way I really don't know how this happened.

I also checked the xha.log on the dacshyp002:

Sep 21 11:39:49 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30408.
Sep 21 11:40:09 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:40:29 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=31466.
Sep 21 11:40:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:46:30 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=35178.
Sep 21 11:46:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:47:10 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33233.
Sep 21 11:47:30 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:55:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=40853.
Sep 21 11:56:11 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:56:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32914.
Sep 21 11:56:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:59:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33377.
Sep 21 11:59:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:01:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38588.
Sep 21 12:02:12 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=38125.
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38323.
Sep 21 12:10:32 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:19:33 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=43934.
Sep 21 12:19:53 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:20:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32989.
Sep 21 12:20:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=42077.
Sep 21 12:21:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:53 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31325.
Sep 21 12:22:13 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:25:54 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=34497.
Sep 21 12:26:14 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:26:34 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32558.
Sep 21 12:26:54 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:27:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30616.
Sep 21 12:27:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:31:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=33975.
Sep 21 12:31:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:38:15 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=34797.
Sep 21 12:38:35 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=37000.
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=40176.
Sep 21 12:54:16 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:16 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=36380.
Sep 21 12:56:36 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:57 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31439.
Sep 21 12:57:17 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 

I can conclude the heartbeat communication is generally unstable in my HA pool. Tomorrow I check the logging of our switch/router.

dacshyp003 booted at

[Mon Sep 21 05:44:37 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

This is the part of the xensource logging of the dacshyp001 before the reboot:

Sep 21 11:17:52 dacshyp001 xapi: [debug||17 ha_monitor|HA monitor D:58d1715e0726|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Same behavior on dacshyp003.
I also attached the xha.log from both nodes.

xha_log_001.txt xha_log_003.txt

Hi all,

I need some help regarding HA.
I have a 3 node cluster running xcp-ng 8.3 (002 is the master)

[13:55 dacshyp002 ~]# xe host-list
uuid ( RO)                : 6b99f1ab-6f4c-4a8d-b766-d16a8a942bdf
          name-label ( RW): dacshyp003
    name-description ( RW): Default install


uuid ( RO)                : d99e150c-079a-4092-8909-ad1a36e07dec
          name-label ( RW): dacshyp001
    name-description ( RW): Default install


uuid ( RO)                : 41a2a448-a5dc-44c6-be44-c07540d75c60
          name-label ( RW): dacshyp002
    name-description ( RW): Default install

Last Thursday I enabled HA on the pool and also on some om the VM's. On Friday I also did an "rolling pool update"
Today I had a "spontaneous" reboot of the dacshyp001 and dacshyp003.
Boot time of dacshyp001:

[Mon Sep 21 11:21:31 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

Around that time I also see the following message in the xensource.log:

Sep 21 11:21:08 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:28 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:48 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead

I will attached all logging around this time.
I am trying to understand what triggered the reboot and how I can troubleshoot this.

I am running XO community commit 6a441 . I am aware this is not the latest version. Before upgrading I want to know the cause of the reboot.

Also ignore the error:

/var/lib/xcp/xapi|dispatch:VDI.get_by_uuid D:1f5b4c8d329e|backtrace] VDI.get_by_uuid D:9df5c0255224 failed with exception Db_exn.Read_missing_uuid("VDI", "", "9219113f-65e4-4368-b42d-1b8bdc7614a9")

I will create another ticket for this.
HA-problem.txt
Thanks in advance.
Carlo

@poddingue https://github.com/vatesfr/xen-orchestra/issues/9779