XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. francescomaria-c
    F
    Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 1
    • Posts 2
    • Groups 0

    francescomaria-c

    @francescomaria-c

    0
    Reputation
    1
    Profile views
    2
    Posts
    0
    Followers
    0
    Following
    Joined
    Last Online

    francescomaria-c Unfollow Follow

    Latest posts made by francescomaria-c

    • RE: XCP-NG 8.2.1 slave node of cluster unresponsive. after hardware reboot (with software reboot the system hang for 20min) the node is unable to contact the master node...

      @Danp yes I ran "yum update" to be sure but "nothing to upgrade" on the pool-master.
      I try with storage (iscsi) NIC configured and without Storage NIC configured but the pool join freeze.
      Seems that persist some "SESSION" (may be referred to the slave host previously configured?) or some incoerence in the pool database...
      from /var/log/xensource of the slave host when try to join the pool:
      "session_check D:520c5b4e5b36 failed with exception Server_error(SESSION_INVALID, "

      posted in XCP-ng
      F
      francescomaria-c
    • XCP-NG 8.2.1 slave node of cluster unresponsive. after hardware reboot (with software reboot the system hang for 20min) the node is unable to contact the master node...

      After the update of the firmware of the switch that provide connectio to the "frontend" NIC the node 2 (slave) of my cluster became unresponsive.
      with xentop i see the VM running but I'm unable to stop o reboot the machines.
      I try to reboot the node but the system hang during the reboot process (more than 20min without progress), so I hard reboot the Server.
      After the reboot the slave server say that it's unable to contact the master node but He can ping the management interface o master node.
      Try to restore this node I try to eject the node from pool without success so i decide to use "declare-dead" command and eject.
      A this point the VM previously running on slave node are available to run on the master and I boot them on this node restore the service and I start a clean reinstallation of XCP8.2.1 on the slave node.
      after clean installation i update (yum update) the node and I set the Management and storage NICs (every interface ping the resources that they need). at this poin I try to add the host to the pool and ste host remain in maintenance mode without completing the join.
      After more than 1hour I decide to "systemctl restart xapi" and "xe-toolstart-restart" on master node but nothing solve (now the master is itself in maintenance mode).
      All VM on the master are still running but I'm now in a very fragile scenario.

      this is the xensource log of slave after the try to join the pool

      Jun 18 06:08:05 xcp10b xapi: [debug||0 ||startup] task [updating the vswitch controller]
      Jun 18 06:08:05 xcp10b xapi: [debug||0 ||dummytaskhelper] task updating the vswitch controller D:a70803078e12 created by task R:baf5d34323d8
      Jun 18 06:08:05 xcp10b xapi: [debug||0 |updating the vswitch controller D:a70803078e12|xmlrpc_client] stunnel pid: 2393 (cached = true) connected to 192.168.17.149:443
      Jun 18 06:08:05 xcp10b xapi: [debug||0 |updating the vswitch controller D:a70803078e12|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=2393
      Jun 18 06:08:05 xcp10b xapi: [debug||0 |updating the vswitch controller D:a70803078e12|xmlrpc_client] stunnel pid: 2393 (cached = true) returned stunnel to cache
      Jun 18 06:08:05 xcp10b xapi: [debug||0 |updating the vswitch controller D:a70803078e12|xmlrpc_client] stunnel pid: 2393 (cached = true) connected to 192.168.17.149:443
      Jun 18 06:08:05 xcp10b xapi: [debug||0 |updating the vswitch controller D:a70803078e12|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=2393
      Jun 18 06:08:05 xcp10b xapi: [debug||56 :::80||dummytaskhelper] task dispatch:host.call_plugin D:838536edbcc0 created by task R:2edbb10d3b4e
      Jun 18 06:08:05 xcp10b xapi: [ info||56 :::80||taskhelper] task host.call_plugin R:2edbb10d3b4e forwarded (trackid=d87b3b96d0c894b651ca6f1968aed6ed)
      Jun 18 06:08:05 xcp10b xapi: [debug||59 /var/lib/xcp/xapi||stunnel] get_reusable_stunnel: stunnel cache is empty; creating a fresh connection to 192.168.17.149:443
      Jun 18 06:08:05 xcp10b xapi: [debug||59 /var/lib/xcp/xapi||xmlrpc_client] stunnel pid: 3222 (cached = true) connected to 192.168.17.149:443
      Jun 18 06:08:05 xcp10b xapi: [debug||59 /var/lib/xcp/xapi||xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=3222
      Jun 18 06:08:05 xcp10b xapi: [debug||59 /var/lib/xcp/xapi||xmlrpc_client] stunnel pid: 3222 (cached = true) returned stunnel to cache
      Jun 18 06:08:05 xcp10b xapi: [debug||60 /var/lib/xcp/xapi||xmlrpc_client] stunnel pid: 3222 (cached = true) connected to 192.168.17.149:443
      Jun 18 06:08:05 xcp10b xapi: [debug||60 /var/lib/xcp/xapi||xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=3222
      Jun 18 06:08:12 xcp10b xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 466179
      Jun 18 06:08:12 xcp10b xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1752576
      Jun 18 06:08:12 xcp10b xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1285649
      Jun 18 06:08:14 xcp10b xapi: [debug||61 HTTPS 192.168.15.75->:::80|Get RRD updates. D:178a99215315|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
      Jun 18 06:08:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|stunnel] stunnel start
      Jun 18 06:08:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] stunnel pid: 3304 (cached = false) connected to 192.168.17.149:443
      Jun 18 06:08:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=3304
      Jun 18 06:08:29 xcp10b xapi: [debug||62 :::80||dummytaskhelper] task dispatch:host.call_plugin D:27d26e5ad597 created by task R:12bcd992f32d
      Jun 18 06:08:29 xcp10b xapi: [ info||62 :::80||taskhelper] task host.call_plugin R:12bcd992f32d forwarded (trackid=e37ef839eaed8066f1a1d06844349f73)
      Jun 18 06:08:36 xcp10b xapi: [debug||63 :::80|session_check D:6e2c65dd6253|session_check] Session check failed: missing reference; tbl = session, ref = OpaqueRef:44273968-a560-47e0-bbde-3a2f7742eacd
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] session_check D:6e2c65dd6253 failed with exception Server_error(SESSION_INVALID, [ OpaqueRef:44273968-a560-47e0-bbde-3a2f7742eacd ])
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] Raised Server_error(SESSION_INVALID, [ OpaqueRef:44273968-a560-47e0-bbde-3a2f7742eacd ])
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 1/6 xapi Raised at file ocaml/xapi/session_check.ml, line 65
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 2/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 87
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 3/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 113
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 4/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 5/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 6/6 xapi Called from file lib/backtrace.ml, line 177
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace]
      Jun 18 06:08:36 xcp10b xapi: [debug||63 :::80|session_check D:520c5b4e5b36|session_check] Session check failed: missing reference; tbl = session, ref = OpaqueRef:44273968-a560-47e0-bbde-3a2f7742eacd
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] session_check D:520c5b4e5b36 failed with exception Server_error(SESSION_INVALID, [ OpaqueRef:44273968-a560-47e0-bbde-3a2f7742eacd ])
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] Raised Server_error(SESSION_INVALID, [ OpaqueRef:44273968-a560-47e0-bbde-3a2f7742eacd ])
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 1/6 xapi Raised at file ocaml/xapi/session_check.ml, line 65
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 2/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 87
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 3/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 113
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 4/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 5/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 6/6 xapi Called from file lib/backtrace.ml, line 177
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace]
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] session.login_with_password D:3e88be8a745e failed with exception Server_error(HOST_IS_SLAVE, [ 192.168.17.149 ])
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] Raised Server_error(HOST_IS_SLAVE, [ 192.168.17.149 ])
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 1/8 xapi Raised at file ocaml/xapi/xapi_session.ml, line 517
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 2/8 xapi Called from file ocaml/xapi/xapi_session.ml, line 53
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 3/8 xapi Called from file ocaml/xapi/xapi_session.ml, line 55
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 4/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 87
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 5/8 xapi Called from file ocaml/xapi/server_helpers.ml, line 113
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 6/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 7/8 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace] 8/8 xapi Called from file lib/backtrace.ml, line 177
      Jun 18 06:08:36 xcp10b xapi: [error||63 :::80||backtrace]
      Jun 18 06:08:44 xcp10b xapi: [debug||64 HTTPS 192.168.15.75->:::80|Get RRD updates. D:a1e798adef76|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
      Jun 18 06:08:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|stunnel] stunnel start
      Jun 18 06:08:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] stunnel pid: 3517 (cached = false) connected to 192.168.17.149:443
      Jun 18 06:08:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=3517
      Jun 18 06:09:15 xcp10b xapi: [debug||65 HTTPS 192.168.15.75->:::80|Get RRD updates. D:5b35f4f6e7ed|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
      Jun 18 06:09:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|stunnel] stunnel start
      Jun 18 06:09:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] stunnel pid: 3713 (cached = false) connected to 192.168.17.149:443
      Jun 18 06:09:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=3713
      Jun 18 06:09:30 xcp10b xapi: [debug||62 :::80||dummytaskhelper] task dispatch:host.call_plugin D:ba1c60cfdeeb created by task R:2ec34cb8ff79
      Jun 18 06:09:30 xcp10b xapi: [ info||62 :::80||taskhelper] task host.call_plugin R:2ec34cb8ff79 forwarded (trackid=9a40c5a9a22b9abebd0e3171f44baa04)
      Jun 18 06:09:45 xcp10b xapi: [debug||66 HTTPS 192.168.15.75->:::80|Get RRD updates. D:49bda44d443c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
      Jun 18 06:09:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|stunnel] stunnel start
      Jun 18 06:09:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] stunnel pid: 3904 (cached = false) connected to 192.168.17.149:443
      Jun 18 06:09:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=3904
      Jun 18 06:10:15 xcp10b xapi: [debug||67 HTTPS 192.168.15.75->:::80|Get RRD updates. D:79fa8bc4fc1b|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
      Jun 18 06:10:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|stunnel] stunnel start
      Jun 18 06:10:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] stunnel pid: 4104 (cached = false) connected to 192.168.17.149:443
      Jun 18 06:10:19 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=4104
      Jun 18 06:10:32 xcp10b xapi: [debug||62 :::80||dummytaskhelper] task dispatch:host.call_plugin D:c3a094597e6f created by task R:d73ed11ba5f3
      Jun 18 06:10:32 xcp10b xapi: [ info||62 :::80||taskhelper] task host.call_plugin R:d73ed11ba5f3 forwarded (trackid=b3c7354a7eb983bdc722812ac49747ce)
      Jun 18 06:10:46 xcp10b xapi: [debug||68 HTTPS 192.168.15.75->:::80|Get RRD updates. D:c0de3061473b|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
      Jun 18 06:10:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|stunnel] stunnel start
      Jun 18 06:10:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] stunnel pid: 4296 (cached = false) connected to 192.168.17.149:443
      Jun 18 06:10:49 xcp10b xapi: [debug||57 heartbeat|Heartbeat D:97b3d62acc66|xmlrpc_client] with_recorded_stunnelpid task_opt=None s_pid=4296
      
      this il the same log on the master node:
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] host.request_backup D:b0a42260ce02 failed with exception Server_error(HOST_OFFLINE, [ OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012 ])
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] Raised Server_error(HOST_OFFLINE, [ OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012 ])
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] 1/6 xapi Raised at file ocaml/xapi/rbac.ml, line 205
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] 2/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 95
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] 3/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 113
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] 4/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] 5/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace] 6/6 xapi Called from file lib/backtrace.ml, line 177
      Jun 18 10:04:03 xcp9 xapi: [error||2405 /var/lib/xcp/xapi||backtrace]
      Jun 18 10:04:03 xcp9 xapi: [error||113 pool_db_backup_thread|Pool DB sync D:2620d99f7346|pool_db_sync] Failed to synchronise DB with host OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012: Server_error(HOST_OFFLINE, [ OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012 ])
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] host.request_backup D:080608935c86 failed with exception Server_error(HOST_OFFLINE, [ OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012 ])
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] Raised Server_error(HOST_OFFLINE, [ OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012 ])
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] 1/6 xapi Raised at file ocaml/xapi/rbac.ml, line 205
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] 2/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 95
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] 3/6 xapi Called from file ocaml/xapi/server_helpers.ml, line 113
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] 4/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] 5/6 xapi Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace] 6/6 xapi Called from file lib/backtrace.ml, line 177
      Jun 18 10:09:03 xcp9 xapi: [error||2485 /var/lib/xcp/xapi||backtrace]
      Jun 18 10:09:03 xcp9 xapi: [error||113 pool_db_backup_thread|Pool DB sync D:2620d99f7346|pool_db_sync] Failed to synchronise DB with host OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012: Server_error(HOST_OFFLINE, [ OpaqueRef:8497c8bc-409e-4d89-bfbc-382e404f8012 ])
      
      posted in XCP-ng
      F
      francescomaria-c