@ph7 This update only covers the security issue described above. Fix for the stats issue will roll out later.
andriy.sultanov
@andriy.sultanov
Best posts made by andriy.sultanov
-
RE: XCP-ng 8.3 updates announcements and testing
-
RE: XCP-ng 8.3 updates announcements and testing
@Greg_E Thanks, but that will not be necessary - I think I've figured out where the problem lies now. Good luck with the move
-
RE: Unable to add host to pool (db mismatch)
Could you please provide the output of (for the master and the candidate host):
xe host-list params=software-version
It does seem odd....
-
RE: "Download System logs" tgz-file does not work
I've been able to reproduce this on 8.3 without XO involved (just by querying the
/host_logs_download
endpoint).The only error I can see in
xensource.log
is at the end of a successful download call:xapi: [debug||6246 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: HANDLE_INVALID: [ task; OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d ] xapi: [error||6246 :::80||backtrace] get_host_logs_download failed with exception Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d") xapi: [error||6246 :::80||backtrace] Raised Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d") xapi: [error||6246 :::80||backtrace] 1/19 xapi Raised at file ocaml/database/db_rpc_client_v1.ml, line 33 xapi: [error||6246 :::80||backtrace] 2/19 xapi Called from file ocaml/xapi/db_actions.ml, line 2577 xapi: [error||6246 :::80||backtrace] 3/19 xapi Called from file ocaml/xapi/taskHelper.ml, line 272 xapi: [error||6246 :::80||backtrace] 4/19 xapi Called from file ocaml/xapi/context.ml, line 515 xapi: [error||6246 :::80||backtrace] 5/19 xapi Called from file ocaml/xapi/context.ml, line 522 xapi: [error||6246 :::80||backtrace] 6/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6246 :::80||backtrace] 7/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6246 :::80||backtrace] 8/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 108 xapi: [error||6246 :::80||backtrace] 9/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6246 :::80||backtrace] 10/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6246 :::80||backtrace] 11/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6246 :::80||backtrace] 12/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6246 :::80||backtrace] 13/19 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6246 :::80||backtrace] 14/19 xapi Called from file ocaml/libs/log/debug.ml, line 267 xapi: [error||6246 :::80||backtrace] 15/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6246 :::80||backtrace] 16/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6246 :::80||backtrace] 17/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 364 xapi: [error||6246 :::80||backtrace] 18/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 370 xapi: [error||6246 :::80||backtrace] 19/19 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6246 :::80||backtrace] xapi: [error||6246 :::80||http_internal_errors] Responding with 500 Internal Error due to Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d") xapi: [debug||6246 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, lineApr 3 15:09:04 xcp-gtn-ip12 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] xapi: [ info||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] task Xapi#getResource /host_logs_download R:83146129fe1c forwarded xapi: [debug||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] the status of R:83146129fe1c is failure; cannot set it to `failure xapi: [debug||6372 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: INTERNAL_ERROR: [ Subprocess exited with unexpected code 1 ] xapi: [error||6372 :::80||backtrace] get_host_logs_download failed with exception Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80||backtrace] Raised Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80||backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80||backtrace] 2/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80||backtrace] 3/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 114 xapi: [error||6372 :::80||backtrace] 4/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6372 :::80||backtrace] 5/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6372 :::80||backtrace] 6/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80||backtrace] 7/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80||backtrace] 8/14 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80||backtrace] 9/14 xapi Called from file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80||backtrace] 10/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80||backtrace] 11/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80||backtrace] 12/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 364 xapi: [error||6372 :::80||backtrace] 13/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 370 xapi: [error||6372 :::80||backtrace] 14/14 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80||backtrace] xapi: [error||6372 :::80||http_internal_errors] Responding with 500 Internal Error due to Forkhelpers.Subprocess_failed(1) xapi: [debug||6372 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n
And then the subsequent download stalls without any indication. I will investigate this further but just to confirm, can any of you see these or similar errors in your logs? Do you have occasional successful downloads followed by failures?
Latest posts made by andriy.sultanov
-
RE: XCP-ng 8.3 updates announcements and testing
@ph7 This update only covers the security issue described above. Fix for the stats issue will roll out later.
-
RE: XCP-ng 8.3 updates announcements and testing
@Greg_E Thanks, but that will not be necessary - I think I've figured out where the problem lies now. Good luck with the move
-
RE: XCP-ng 8.3 updates announcements and testing
@bufanda Thanks!
Hmm, can't see anything suspicious either in your logs or @abudef'sSince you said it reproduces if you reboot the host again, I'd really appreciate if you could send the result of
rrd2csv
running for 30 seconds or so on a system that doesn't have the stats working - so before any toolstack restarts. -
RE: XCP-ng 8.3 updates announcements and testing
Could you please attach
/var/log/{xensource.log,daemon.log,xcp-rrdd-plugins.log}
from the time after the reboot and before the toolstack restart?Also: does this reproduce if you reboot again - would you still have no metrics until toolstack restart? Because if not, this is entering paranormal territory
UPD: I've just updated xapi from 24.39 to 25.6 myself - stats are working after the reboot and I can see them in XenOrchestra, no additional toolstack restart required.
-
RE: Unable to add host to pool (db mismatch)
Could you please provide the output of (for the master and the candidate host):
xe host-list params=software-version
It does seem odd....
-
RE: "Download System logs" tgz-file does not work
I've been able to reproduce this on 8.3 without XO involved (just by querying the
/host_logs_download
endpoint).The only error I can see in
xensource.log
is at the end of a successful download call:xapi: [debug||6246 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: HANDLE_INVALID: [ task; OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d ] xapi: [error||6246 :::80||backtrace] get_host_logs_download failed with exception Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d") xapi: [error||6246 :::80||backtrace] Raised Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d") xapi: [error||6246 :::80||backtrace] 1/19 xapi Raised at file ocaml/database/db_rpc_client_v1.ml, line 33 xapi: [error||6246 :::80||backtrace] 2/19 xapi Called from file ocaml/xapi/db_actions.ml, line 2577 xapi: [error||6246 :::80||backtrace] 3/19 xapi Called from file ocaml/xapi/taskHelper.ml, line 272 xapi: [error||6246 :::80||backtrace] 4/19 xapi Called from file ocaml/xapi/context.ml, line 515 xapi: [error||6246 :::80||backtrace] 5/19 xapi Called from file ocaml/xapi/context.ml, line 522 xapi: [error||6246 :::80||backtrace] 6/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6246 :::80||backtrace] 7/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6246 :::80||backtrace] 8/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 108 xapi: [error||6246 :::80||backtrace] 9/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6246 :::80||backtrace] 10/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6246 :::80||backtrace] 11/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6246 :::80||backtrace] 12/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6246 :::80||backtrace] 13/19 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6246 :::80||backtrace] 14/19 xapi Called from file ocaml/libs/log/debug.ml, line 267 xapi: [error||6246 :::80||backtrace] 15/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6246 :::80||backtrace] 16/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6246 :::80||backtrace] 17/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 364 xapi: [error||6246 :::80||backtrace] 18/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 370 xapi: [error||6246 :::80||backtrace] 19/19 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6246 :::80||backtrace] xapi: [error||6246 :::80||http_internal_errors] Responding with 500 Internal Error due to Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d") xapi: [debug||6246 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, lineApr 3 15:09:04 xcp-gtn-ip12 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] xapi: [ info||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] task Xapi#getResource /host_logs_download R:83146129fe1c forwarded xapi: [debug||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] the status of R:83146129fe1c is failure; cannot set it to `failure xapi: [debug||6372 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: INTERNAL_ERROR: [ Subprocess exited with unexpected code 1 ] xapi: [error||6372 :::80||backtrace] get_host_logs_download failed with exception Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80||backtrace] Raised Forkhelpers.Subprocess_failed(1) xapi: [error||6372 :::80||backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80||backtrace] 2/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80||backtrace] 3/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 114 xapi: [error||6372 :::80||backtrace] 4/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 69 xapi: [error||6372 :::80||backtrace] 5/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 96 xapi: [error||6372 :::80||backtrace] 6/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80||backtrace] 7/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80||backtrace] 8/14 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80||backtrace] 9/14 xapi Called from file ocaml/libs/log/debug.ml, line 267 xapi: [error||6372 :::80||backtrace] 10/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24 xapi: [error||6372 :::80||backtrace] 11/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39 xapi: [error||6372 :::80||backtrace] 12/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 364 xapi: [error||6372 :::80||backtrace] 13/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 370 xapi: [error||6372 :::80||backtrace] 14/14 xapi Called from file ocaml/libs/log/debug.ml, line 250 xapi: [error||6372 :::80||backtrace] xapi: [error||6372 :::80||http_internal_errors] Responding with 500 Internal Error due to Forkhelpers.Subprocess_failed(1) xapi: [debug||6372 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n
And then the subsequent download stalls without any indication. I will investigate this further but just to confirm, can any of you see these or similar errors in your logs? Do you have occasional successful downloads followed by failures?