XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    HA causes reboot of xcp-ng nodes

    Scheduled Pinned Locked Moved Unsolved Management
    7 Posts 3 Posters 208 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      carloum70
      last edited by carloum70

      Hi all,

      I need some help regarding HA.
      I have a 3 node cluster running xcp-ng 8.3 (002 is the master)

      [13:55 dacshyp002 ~]# xe host-list
uuid ( RO)                : 6b99f1ab-6f4c-4a8d-b766-d16a8a942bdf
          name-label ( RW): dacshyp003
    name-description ( RW): Default install


uuid ( RO)                : d99e150c-079a-4092-8909-ad1a36e07dec
          name-label ( RW): dacshyp001
    name-description ( RW): Default install


uuid ( RO)                : 41a2a448-a5dc-44c6-be44-c07540d75c60
          name-label ( RW): dacshyp002
    name-description ( RW): Default install

      Last Thursday I enabled HA on the pool and also on some om the VM's. On Friday I also did an "rolling pool update"
      Today I had a "spontaneous" reboot of the dacshyp001 and dacshyp003.
      Boot time of dacshyp001:

      [Mon Sep 21 11:21:31 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

      Around that time I also see the following message in the xensource.log:

      Sep 21 11:21:08 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:28 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead
Sep 21 11:21:48 dacshyp002 xapi: [debug||20398 ha_monitor|HA monitor D:7b5068fa6bfc|xapi_ha_vm_failover] Setting host dacshyp001 to dead

      I will attached all logging around this time.
      I am trying to understand what triggered the reboot and how I can troubleshoot this.

      I am running XO community commit 6a441 . I am aware this is not the latest version. Before upgrading I want to know the cause of the reboot.

      Also ignore the error:

      /var/lib/xcp/xapi|dispatch:VDI.get_by_uuid D:1f5b4c8d329e|backtrace] VDI.get_by_uuid D:9df5c0255224 failed with exception Db_exn.Read_missing_uuid("VDI", "", "9219113f-65e4-4368-b42d-1b8bdc7614a9")

      I will create another ticket for this.
      HA-problem.txt
      Thanks in advance.
      Carlo

      poddingueP tjkreidlT 2 Replies Last reply
      Reply Quote 0
      • poddingueP
        poddingue Vates 🪐 @carloum70
        last edited by

        From what I read in the docs (HA isn't my strong suit), a host in an HA pool that loses its heartbeat in certain ways is designed to reboot itself, which they call self-fencing, so this may well be HA doing its job rather than something crashing. 🤔

        The log you attached is from the master and starts at 11:21:00, and dacshyp001 is already marked as not live in the very first liveset at 11:21:08, so I think whatever triggered it happened just before that and isn't in this file. 🤷

        The three Setting host dacshyp001 to dead lines look to me like one event being re-checked every 20 seconds while 001 was still coming back, though I could be misreading that.

        I also suspect 003 went down at a different moment, because it still shows as alive in that same liveset. There's a doc section for this case, https://docs.xcp-ng.org/troubleshooting/troubleshooting-ha#my-host-rebooted-why-did-it-reboot, which points at /var/log/xha.log on the host that rebooted.

        Could you post that file from dacshyp001 and dacshyp003 for the few minutes before each reboot, plus 003's boot time?

        1 Reply Last reply
        Reply Quote 0
        • C
          carloum70
          last edited by carloum70

          dacshyp003 booted at

          [Mon Sep 21 05:44:37 2026] Linux version 4.19.0+1 (mockbuild@1efe98ffea2245bcb4c11893890153ac) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC)) #1 SMP Thu Aug 13 13:06:11 UTC 2026

          This is the part of the xensource logging of the dacshyp001 before the reboot:

          Sep 21 11:17:52 dacshyp001 xapi: [debug||17 ha_monitor|HA monitor D:58d1715e0726|xapi_ha] The node we think is the master is still alive and marked as master; this is OK
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

          Same behavior on dacshyp003.
          I also attached the xha.log from both nodes.

          xha_log_001.txt xha_log_003.txt

          1 Reply Last reply
          Reply Quote 1
          • C
            carloum70
            last edited by carloum70

            I also checked the xha.log on the dacshyp002:

            Sep 21 11:39:49 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30408.
Sep 21 11:40:09 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:40:29 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=31466.
Sep 21 11:40:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:46:30 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=35178.
Sep 21 11:46:50 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:47:10 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33233.
Sep 21 11:47:30 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:55:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=40853.
Sep 21 11:56:11 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:56:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32914.
Sep 21 11:56:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 11:59:31 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=33377.
Sep 21 11:59:51 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:01:51 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38588.
Sep 21 12:02:12 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=38125.
Sep 21 12:10:12 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=38323.
Sep 21 12:10:32 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:19:33 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=43934.
Sep 21 12:19:53 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:20:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32989.
Sep 21 12:20:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:13 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=42077.
Sep 21 12:21:33 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:21:53 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31325.
Sep 21 12:22:13 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:25:54 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=34497.
Sep 21 12:26:14 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:26:34 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=32558.
Sep 21 12:26:54 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:27:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=30616.
Sep 21 12:27:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:31:14 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=33975.
Sep 21 12:31:34 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:38:15 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=34797.
Sep 21 12:38:35 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[1].time_since_last_hb=37000.
Sep 21 12:53:56 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=40176.
Sep 21 12:54:16 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:16 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=36380.
Sep 21 12:56:36 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE 
Sep 21 12:56:57 CEST 2026 [warn] SC: (script_service_do_query_liveset) reporting "heartbeat approaching timeout". host[2].time_since_last_hb=31439.
Sep 21 12:57:17 CEST 2026 [info] SC: (script_service_do_query_liveset) "Heartbeat approaching timeout" turned FALSE

            I can conclude the heartbeat communication is generally unstable in my HA pool. Tomorrow I check the logging of our switch/router.

            1 Reply Last reply
            Reply Quote 0
            • poddingueP poddingue marked this topic as a question
            • tjkreidlT
              tjkreidl Ambassador @carloum70
              last edited by tjkreidl

              @carloum70 Are all your hosts properly time sychronized to NTP or chronyc? Check each host for offsets. They need to be really close in time with each other. And what HA heartbeat client setup are you using? As long as you have a quorum, HA should continue to work fine. WHen you get down to two hosts and one cannot communicate with the other is when things get critical (unless running HA-Lizard on a two-host pool).

              1 Reply Last reply
              Reply Quote 0
              • C
                carloum70
                last edited by

                I did some further investigation and I think there is something wrong with the ip-settings of the management interface.

                At the xsconsole I see the following:

                Management Network Parameters    

Device        bond0         
address       172.28.4.11    
Netmask       255.255.252.0   
Gateway       172.28.4.1

                So it's using the bond0 interface.

                # xe pif-list host-uuid=41a2a448-a5dc-44c6-be44-c07540d75c60

uuid ( RO)                  : 0761d887-268e-5cf0-401b-a08ad7c419aa
                device ( RO): bond0
                   MAC ( RO): 30:3e:a7:1d:b0:90
    currently-attached ( RO): true
                  VLAN ( RO): -1
          network-uuid ( RO): 28d51793-0226-cf3f-75d3-ed7c5a9d8b33
             host-uuid ( RO): 41a2a448-a5dc-44c6-be44-c07540d75c60

                This is part of the output of xe bond-list

                # xe bond-list
uuid ( RO)      : 79c525c3-22f9-04ce-4a80-21d18ef65ebc
    master ( RO): 0761d887-268e-5cf0-401b-a08ad7c419aa
    slaves ( RO): 7ac18886-e0a5-057b-bac0-36b909588dba ; 2f18fc9a-8c1c-dfce-9416-fa8c657c5f63

                I already checked that:

                7ac18886-e0a5-057b-bac0-36b909588dba --> eth3
2f18fc9a-8c1c-dfce-9416-fa8c657c5f63 --> eth2

                And now the confusing part. This is part of the output of the ip a command:

                2: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
3: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:91 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff


13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
       valid_lft forever preferred_lft forever

                As you can see the xenbr4 has also the management ip-address and has the same mac-address as the eth4 interface.
                OK check the following commands:

                [17:21 dacshyp002 ~]# ip addr show xapi1
15: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:90 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xapi1
       valid_lft forever preferred_lft forever
[17:21 dacshyp002 ~]# ip addr show xenbr4
13: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 30:3e:a7:1d:b0:92 brd ff:ff:ff:ff:ff:ff
    inet 172.28.4.11/22 brd 172.28.7.255 scope global xenbr4
       valid_lft forever preferred_lft forever

                And If I look at the routing table:

                # ip route
default via 172.28.4.1 dev xapi1 
172.18.8.0/23 dev xenbr0 proto kernel scope link src 172.18.8.11 
172.18.10.0/23 dev xenbr7 proto kernel scope link src 172.18.10.11 
172.28.4.0/22 dev xenbr4 proto kernel scope link src 172.28.4.11 
172.28.4.0/22 dev xapi1 proto kernel scope link src 172.28.4.11

                According to me the node has 2 interfaces xenbr4 / xapi1 with the same ip-address and also 2 different routes for the same ip-range 172.28.4.0/22.
                And that's why the heartbeat has some issues. Can someone confirm this ?
                By the way I really don't know how this happened.

                tjkreidlT 1 Reply Last reply
                Reply Quote 0
                • tjkreidlT
                  tjkreidl Ambassador @carloum70
                  last edited by

                  @carloum70 If you are willing to try this, you can reset the primary management interface (PMI) on a bond, WIthin the bash shell:
                  :
                  xe pif-list
                  This will show the PIFs and their current network associations.

                  Identify the bond master PIF
                  If you have a bonded network, the master PIF is the one that represents the bond. You can list it with:

                  xe pif-list network-uuid=<bond-uuid>
                  The master PIF is the one that will be used as the primary interface for the bond.

                  Reset the management interface to the bond master
                  Use the xe-reset-networking command with the --reset-primary option:

                  xe-reset-networking --reset-primary
                  This will move the management interface to the bond master PIF, which is the correct way to reassign it when the bond is created or reconfigured.

                  1 Reply Last reply
                  Reply Quote 0

                  Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                  Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                  With your input, this post could be even better 💗

                  Register Login
                  • First post
                    Last post