Hello!

I am Unable to find logs in XenCenter or Xen Orchestra.

@sushant-diwakar

@Danp said in XAPI servic failed:

Is this server part of a pool? If yes, is it the pool master?
Have there been any recent updates to the pool (ie: patches applied, network settings, etc)?
What version of XCP-ng is the server running?

Is this server part of a pool? If yes, is it the pool master?
Yes this host is a part of a pool
Have there been any recent updates to the pool (ie: patches applied, network settings, etc)?
**No we have not installed any updates but we were trying to make this host as pool master **
What version of XCP-ng is the server running?
XCP-NG 8.2.1

@olivierlambert appologies i will close that ticket.

@sushant-diwakar
below one more error

Error: Connection refused (calling connect )

@sushant-diwakar said in XAPI servic failed:

server_init D:67172903d9f5 failed with exception Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] Raised Pool_role.This_host_is_broken

server_init D:67172903d9f5 failed with exception Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] Raised Pool_role.This_host_is_broken

Hi one of our host is giving xapi service failed, unable to start VM on this host.
please help.

Feb 17 08:49:25 sr0002 xapi: [ info||0 |bringing up management interface D:be80e045f243|xapi_mgmt_iface] Restarting stunnel (accepting connections on :::443)
Feb 17 08:49:25 sr0002 xapi: [debug||9 ||helpers] about to call script: /usr/bin/systemctl
Feb 17 08:49:25 sr0002 xapi: [debug||9 ||helpers] /usr/bin/systemctl is-enabled stunnel@xapi exited with code 1 [stdout = 'disabled\x0A'; stderr = '']
Feb 17 08:49:25 sr0002 xapi: [debug||9 ||helpers] about to call script: /usr/bin/systemctl
Feb 17 08:49:25 sr0002 xapi: [debug||0 |bringing up management interface D:be80e045f243|xapi] Management IP address is: 192.168.4.22
Feb 17 08:49:25 sr0002 xapi: [debug||0 |bringing up management interface D:be80e045f243|xapi] Caught exception bringing up management interface: INTERNAL_ERROR: [ Pool_role.This_host_is_broken ]
Feb 17 08:49:25 sr0002 xapi: [debug||11 ||dummytaskhelper] task dom0 networking update D:0bf503b1bc04 created by task D:67172903d9f5
Feb 17 08:49:25 sr0002 xapi: [debug||0 ||startup] task [starting thread Starting periodic scheduler]
Feb 17 08:49:25 sr0002 xapi: [debug||0 ||startup] task [Synchronising host configuration files]
Feb 17 08:49:25 sr0002 xapi: [debug||12 ||dummytaskhelper] task Starting periodic scheduler D:2aca8b4a1da1 created by task D:67172903d9f5
Feb 17 08:49:25 sr0002 xapi: [debug||11 |dom0 networking update D:0bf503b1bc04|xapi_mgmt_iface] Checking to see if hostname or management IP has changed
Feb 17 08:49:25 sr0002 xapi: [debug||12 |Starting periodic scheduler D:2aca8b4a1da1|backgroundscheduler] Periodic scheduler started
Feb 17 08:49:25 sr0002 xapi: [debug||0 ||dummytaskhelper] task Synchronising host configuration files D:2741cf86fe9c created by task D:67172903d9f5
Feb 17 08:49:25 sr0002 xapi: [debug||11 |dom0 networking update D:0bf503b1bc04|helpers] Updating IP addresses in DB for DHCP and autoconf PIFs
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] Synchronising host configuration files D:2741cf86fe9c failed with exception Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||11 ||backtrace] dom0 networking update D:0bf503b1bc04 failed with exception Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] Raised Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||11 ||backtrace] Raised Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 1/12 xapi Raised at file ocaml/xapi-aux/pool_role.ml, line 112
Feb 17 08:49:25 sr0002 xapi: [error||11 ||backtrace] 1/1 xapi Raised at file (Thread 11 has no backtrace table. Was with_backtraces called?, line 0
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 2/12 xapi Called from file ocaml/database/master_connection.ml, line 326
Feb 17 08:49:25 sr0002 xapi: [error||11 ||backtrace]
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 3/12 xapi Called from file ocaml/database/db_rpc_client_v1.ml, line 60
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 4/12 xapi Called from file ocaml/xapi/db_actions.ml, line 8179
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 5/12 xapi Called from file ocaml/xapi/xapi_host_helpers.ml, line 434
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 6/12 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 7/12 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 8/12 xapi Called from file ocaml/xapi/server_helpers.ml, line 87
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 9/12 xapi Called from file ocaml/xapi/server_helpers.ml, line 113
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 10/12 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 11/12 xapi Called from file map.ml, line 135
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 12/12 xapi Called from file src/sexp_conv.ml, line 156
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace]
Feb 17 08:49:25 sr0002 xapi: [ warn||0 ||startup] task [Synchronising host configuration files] exception: Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] server_init D:67172903d9f5 failed with exception Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] Raised Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 1/1 xapi Raised at file (Thread 0 has no backtrace table. Was with_backtraces called?, line 0
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace]
Feb 17 08:49:25 sr0002 xapi: [debug||0 ||xapi] xapi top-level caught exception: INTERNAL_ERROR: [ Pool_role.This_host_is_broken ]
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] Raised Pool_role.This_host_is_broken
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace] 1/1 xapi Raised at file (Thread 0 has no backtrace table. Was with_backtraces called?, line 0
Feb 17 08:49:25 sr0002 xapi: [error||0 ||backtrace]

@olivierlambert
I have create ticket on support.

Ticket#7721836

@olivierlambert
I will take forward this to my management regarding support.

I have found a solution should i do this ?
or is there any other way to start the xapi service ?
https://xcp-ng.org/forum/topic/4721/pool-master-went-down-all-other-nodes-claims-not-to-be-in-a-pool/4?_=1708076813182

@olivierlambert
Hi we have XOA premium license,

Can you help us ?

We have some hardware maintainence on current pool master so we change the pool master but unfortunalty on new pool master xapi service is in fail state.
And i am not able to access any of Virtual machines.
What could be solution. I am not able to access pool now there are 20 xcp-ng host on this pool and 150+ VM.

Feb 16 14:53:37 sr0002 xenopsd-xc: [debug||36 |events|xenops_server] VM f83416d8-1c6f-b814-bee1-a958fb22b986 is not requesting any attention
Feb 16 14:53:37 sr0002 xenopsd-xc: [debug||36 |events|xenops_server] VM_DB.signal f83416d8-1c6f-b814-bee1-a958fb22b986
Feb 16 14:53:37 sr0002 xenopsd-xc: [debug||36 |events|task_server] Task 144 completed; duration = 0
Feb 16 14:53:37 sr0002 xenopsd-xc: [debug||36 ||xenops_server] TASK.signal 144 (object deleted)
Feb 16 14:53:51 sr0002 xcp-rrdd: [ info||9 ||rrdd_main] memfree has changed to 13621020 in domain 1
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||5 ||xenops_server] Received an event on managed VM 851ded34-a5af-4de6-2226-2eb3cd14b5db
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||5 |queue|xenops_server] Queue.push ["VM_check_state","851ded34-a5af-4de6-2226-2eb3cd14b5db"] onto 851ded34-a5af-4de6-2226-2eb3cd14b5db:[  ]
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||11 ||xenops_server] Queue.pop returned ["VM_check_state","851ded34-a5af-4de6-2226-2eb3cd14b5db"]
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||11 |events|xenops_server] Task 145 reference events: ["VM_check_state","851ded34-a5af-4de6-2226-2eb3cd14b5db"]
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||11 |events|xenops_server] VM 851ded34-a5af-4de6-2226-2eb3cd14b5db is not requesting any attention
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||11 |events|xenops_server] VM_DB.signal 851ded34-a5af-4de6-2226-2eb3cd14b5db
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||11 |events|task_server] Task 145 completed; duration = 0
Feb 16 14:53:51 sr0002 xenopsd-xc: [debug||11 ||xenops_server] TASK.signal 145 (object deleted)
Feb 16 14:54:21 sr0002 xcp-rrdd: [ info||9 ||rrdd_main] memfree has changed to 34375344 in domain 3
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||5 ||xenops_server] Received an event on managed VM bf3352b8-117b-8882-834f-6933dd8486ad
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||5 |queue|xenops_server] Queue.push ["VM_check_state","bf3352b8-117b-8882-834f-6933dd8486ad"] onto bf3352b8-117b-8882-834f-6933dd8486ad:[  ]
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||25 ||xenops_server] Queue.pop returned ["VM_check_state","bf3352b8-117b-8882-834f-6933dd8486ad"]
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||25 |events|xenops_server] Task 146 reference events: ["VM_check_state","bf3352b8-117b-8882-834f-6933dd8486ad"]
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||25 |events|xenops_server] VM bf3352b8-117b-8882-834f-6933dd8486ad is not requesting any attention
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||25 |events|xenops_server] VM_DB.signal bf3352b8-117b-8882-834f-6933dd8486ad
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||25 |events|task_server] Task 146 completed; duration = 0
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||25 ||xenops_server] TASK.signal 146 (object deleted)
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||5 ||xenops_server] Received an event on managed VM 1f856394-5cb5-6446-0cb1-7b88871a8a7f
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||5 |queue|xenops_server] Queue.push ["VM_check_state","1f856394-5cb5-6446-0cb1-7b88871a8a7f"] onto 1f856394-5cb5-6446-0cb1-7b88871a8a7f:[  ]
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||21 ||xenops_server] Queue.pop returned ["VM_check_state","1f856394-5cb5-6446-0cb1-7b88871a8a7f"]
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||21 |events|xenops_server] Task 147 reference events: ["VM_check_state","1f856394-5cb5-6446-0cb1-7b88871a8a7f"]
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||21 |events|xenops_server] VM 1f856394-5cb5-6446-0cb1-7b88871a8a7f is not requesting any attention
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||21 |events|xenops_server] VM_DB.signal 1f856394-5cb5-6446-0cb1-7b88871a8a7f
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||21 |events|task_server] Task 147 completed; duration = 0
Feb 16 14:54:21 sr0002 xenopsd-xc: [debug||21 ||xenops_server] TASK.signal 147 (object deleted)
Feb 16 14:54:30 sr0002 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 748272
Feb 16 14:54:30 sr0002 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1554432
Feb 16 14:54:30 sr0002 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 806088
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||5 ||xenops_server] Received an event on managed VM f83416d8-1c6f-b814-bee1-a958fb22b986
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||5 |queue|xenops_server] Queue.push ["VM_check_state","f83416d8-1c6f-b814-bee1-a958fb22b986"] onto f83416d8-1c6f-b814-bee1-a958fb22b986:[  ]
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||23 ||xenops_server] Queue.pop returned ["VM_check_state","f83416d8-1c6f-b814-bee1-a958fb22b986"]
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||23 |events|xenops_server] Task 148 reference events: ["VM_check_state","f83416d8-1c6f-b814-bee1-a958fb22b986"]
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||23 |events|xenops_server] VM f83416d8-1c6f-b814-bee1-a958fb22b986 is not requesting any attention
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||23 |events|xenops_server] VM_DB.signal f83416d8-1c6f-b814-bee1-a958fb22b986
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||23 |events|task_server] Task 148 completed; duration = 0
Feb 16 14:54:37 sr0002 xenopsd-xc: [debug||23 ||xenops_server] TASK.signal 148 (object deleted)
Feb 16 14:54:45 sr0002 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||5 ||xenops_server] Received an event on managed VM 851ded34-a5af-4de6-2226-2eb3cd14b5db
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||5 |queue|xenops_server] Queue.push ["VM_check_state","851ded34-a5af-4de6-2226-2eb3cd14b5db"] onto 851ded34-a5af-4de6-2226-2eb3cd14b5db:[  ]
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||27 ||xenops_server] Queue.pop returned ["VM_check_state","851ded34-a5af-4de6-2226-2eb3cd14b5db"]
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||27 |events|xenops_server] Task 149 reference events: ["VM_check_state","851ded34-a5af-4de6-2226-2eb3cd14b5db"]
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||27 |events|xenops_server] VM 851ded34-a5af-4de6-2226-2eb3cd14b5db is not requesting any attention
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||27 |events|xenops_server] VM_DB.signal 851ded34-a5af-4de6-2226-2eb3cd14b5db
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||27 |events|task_server] Task 149 completed; duration = 0
Feb 16 14:54:52 sr0002 xenopsd-xc: [debug||27 ||xenops_server] TASK.signal 149 (object deleted)
[14:55 sr0002 ~]#

Please consider this as urgent.