I did some further investigation and I think there is something wrong with the ip-settings of the management interface.
At the xsconsole I see the following:
Management Network Parameters
Device bond0
address 172.28.4.11
Netmask 255.255.252.0
Gateway 172.28.4.1
So it's using the bond0 interface.
# xe pif-list host-uuid=41a2a448-a5dc-44c6-be44-c07540d75c60
uuid ( RO) : 0761d887-268e-5cf0-401b-a08ad7c419aa
device ( RO): bond0
MAC ( RO): 30:3e:a7:1d:b0:90
currently-attached ( RO): true
VLAN ( RO): -1
network-uuid ( RO): 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
host-uuid ( RO): 41a2a448-a5dc-44c6-be44-c07540d75c60
This is part of the output of xe bond-list
# xe bond-list
uuid ( RO) : 79c525c3-22f9-04ce-4a80-21d18ef65ebc
master ( RO): 0761d887-268e-5cf0-401b-a08ad7c419aa
slaves ( RO): 7ac18886-e0a5-057b-bac0-36b909588dba ; 2f18fc9a-8c1c-dfce-9416-fa8c657c5f63
I already checked that:
7ac18886-e0a5-057b-bac0-36b909588dba --> eth3
2f18fc9a-8c1c-dfce-9416-fa8c657c5f63 --> eth2
And now the confusing part. This is part of the output of the ip a command:
2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 30:3e:a7:1d:b0:91 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
valid_lft forever preferred_lft forever
As you can see the xenbr4 has also the management ip-address and has the same mac-address as the eth4 interface.
OK check the following commands:
[17:21 dacshyp002 ~]# ip addr show xapi1
15: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
inet 172.28.4.11/22 brd 172.28.7.255 scope global xapi1
valid_lft forever preferred_lft forever
[17:21 dacshyp002 ~]# ip addr show xenbr4
13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
valid_lft forever preferred_lft forever
And If I look at the routing table:
# ip route
default via 172.28.4.1 dev xapi1
172.18.8.0/23 dev xenbr0 proto kernel scope link src 172.18.8.11
172.18.10.0/23 dev xenbr7 proto kernel scope link src 172.18.10.11
172.28.4.0/22 dev xenbr4 proto kernel scope link src 172.28.4.11
172.28.4.0/22 dev xapi1 proto kernel scope link src 172.28.4.11
According to me the node has 2 interfaces xenbr4 / xapi1 with the same ip-address and also 2 different routes for the same ip-range 172.28.4.0/22.
And that's why the heartbeat has some issues. Can someone confirm this ?
By the way I really don't know how this happened.