XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    "Download System logs" tgz-file does not work

    Scheduled Pinned Locked Moved Xen Orchestra
    17 Posts 8 Posters 505 Views 8 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      peder @Chemikant784
      last edited by

      Same problem here. In my case it downloads about 30MB before it stalls.

      XCP-ng version : 8.2.1 with March updates
      XO : I tried both 9c27b53df and the older 708b9b5bc
      Browser : Firefox and Chrome in Linux

      1 Reply Last reply Reply Quote 0
      • P Offline
        ph7 @olivierlambert
        last edited by ph7

        @olivierlambert
        Same in 8.3 and 37fbb
        edit: And in 5.103.1

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Online
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Ping @lsouai-vates it seems to be a bug, IDK if it's on XO or XCP-ng's side

          lsouai-vatesL 1 Reply Last reply Reply Quote 0
          • lsouai-vatesL Offline
            lsouai-vates Vates 🪐 XO Team @olivierlambert
            last edited by

            @Chemikant784 Hello, and thanks for your bug report.
            I'am currently asking the XO and XCP-NG team about it, and I keep you in touch when I have more information...

            lsouai-vatesL 1 Reply Last reply Reply Quote 0
            • lsouai-vatesL Offline
              lsouai-vates Vates 🪐 XO Team @lsouai-vates
              last edited by

              @stormi can you help on this subject on XCP-NG team side?

              1 Reply Last reply Reply Quote 0
              • stormiS Offline
                stormi Vates 🪐 XCP-ng Team
                last edited by stormi

                I'd start with checking the usual logs (notably xensource.log) when the issue happens.

                See https://docs.xcp-ng.org/troubleshooting/log-files/

                Yes, I know, getting logs about an issue getting logs. Of course, I mean checking them via CLI on XCP-ng hosts :).

                C P 2 Replies Last reply Reply Quote 0
                • C Offline
                  Chemikant784 @stormi
                  last edited by

                  @stormi indeed not free of irony 😄

                  1 Reply Last reply Reply Quote 0
                  • P Offline
                    peder @stormi
                    last edited by

                    @stormi Any hint on what to look for?

                    I see these entries when the download starts but no subsequent errors or anything else that stands out when the download stalls

                    xapi: [ info||1520711 HTTPS 10.x.y.z->:::80|task.create D:b06a0108fbb7|taskhelper] task Xapi#getResource /host_logs_download R:8ea91ee79d9a (uuid:a-b-c-d-e) created (trackid=18e32f90af09a295c704ca6099aaef26) by task D:b06a0108fbb7
xapi: [debug||1520711 :::80||xapi_logs_download] running logs-download handler
xapi: [ info||1520711 :::80||taskhelper] task Xapi#getResource /host_logs_download R:8ea91ee79d9a forwarded (trackid=18e32f90af09a295c704ca6099aaef26)
xapi: [debug||1520711 HTTPS 10.x.y.z->:::80|Xapi#getResource /host_logs_download R:8ea91ee79d9a|xapi_logs_download] send the http headers
                    stormiS 1 Reply Last reply Reply Quote 0
                    • stormiS Offline
                      stormi Vates 🪐 XCP-ng Team @peder
                      last edited by

                      @peder No, but let me ping our XAPI developers @gthvn1 and @andriy.sultanov

                      A 1 Reply Last reply Reply Quote 0
                      • A Offline
                        andriy.sultanov Vates 🪐 XAPI & Network Team @stormi
                        last edited by andriy.sultanov

                        I've been able to reproduce this on 8.3 without XO involved (just by querying the /host_logs_download endpoint).

                        The only error I can see in xensource.log is at the end of a successful download call:

                        xapi: [debug||6246 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: HANDLE_INVALID: [ task; OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d ]
xapi: [error||6246 :::80||backtrace] get_host_logs_download failed with exception Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] Raised Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] 1/19 xapi Raised at file ocaml/database/db_rpc_client_v1.ml, line 33
xapi: [error||6246 :::80||backtrace] 2/19 xapi Called from file ocaml/xapi/db_actions.ml, line 2577
xapi: [error||6246 :::80||backtrace] 3/19 xapi Called from file ocaml/xapi/taskHelper.ml, line 272
xapi: [error||6246 :::80||backtrace] 4/19 xapi Called from file ocaml/xapi/context.ml, line 515
 xapi: [error||6246 :::80||backtrace] 5/19 xapi Called from file ocaml/xapi/context.ml, line 522
xapi: [error||6246 :::80||backtrace] 6/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 7/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 8/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 108
xapi: [error||6246 :::80||backtrace] 9/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6246 :::80||backtrace] 10/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6246 :::80||backtrace] 11/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 12/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 13/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace] 14/19 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6246 :::80||backtrace] 15/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 16/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 17/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6246 :::80||backtrace] 18/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6246 :::80||backtrace] 19/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace]
xapi: [error||6246 :::80||http_internal_errors] Responding with 500 Internal Error due to Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [debug||6246 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, lineApr  3 15:09:04 xcp-gtn-ip12 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace]
xapi: [ info||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] task Xapi#getResource /host_logs_download R:83146129fe1c forwarded
xapi: [debug||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] the status of R:83146129fe1c is failure; cannot set it to `failure
xapi: [debug||6372 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: INTERNAL_ERROR: [ Subprocess exited with unexpected code 1 ]
xapi: [error||6372 :::80||backtrace] get_host_logs_download failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 2/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 3/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 114
xapi: [error||6372 :::80||backtrace] 4/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80||backtrace] 5/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80||backtrace] 6/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 7/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 8/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace] 9/14 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 10/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 11/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 12/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6372 :::80||backtrace] 13/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6372 :::80||backtrace] 14/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace]
xapi: [error||6372 :::80||http_internal_errors] Responding with 500 Internal Error due to Forkhelpers.Subprocess_failed(1)
xapi: [debug||6372 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n

                        And then the subsequent download stalls without any indication. I will investigate this further but just to confirm, can any of you see these or similar errors in your logs? Do you have occasional successful downloads followed by failures?

                        P 1 Reply Last reply Reply Quote 1
                        • P Offline
                          peder @andriy.sultanov
                          last edited by

                          @andriy.sultanov I'm not seeing any errors after it stalls.

                          One funny thing is that the pool master usually stalls after approximately 135MB and the slave stalls after 230MB
                          /var/log on master is half the size of slave's so that may be why.

                          Another funny thing is that when I tried the slave once more it actually finished the download at 230MB but my subsequent "tar -xf" complained about "gzip: stdin: decompression OK, trailing garbage ignored"
                          And for some reason the files it extracted were 528MB whereas they're 498MB on the server.
                          Maybe that's a matter of my using XFS vs the server's ext3, though.

                          That "successful" download resulted in

                          xapi: [error||307417 :::80||backtrace] Xapi#getResource /host_logs_download R:51bd50ee890d failed with exception Forkhelpers.Subprocess_failed(1)

                          in xensource.log.
                          So it seems to fail to properly close the transfer session most of the time and the file compression seems to be somewhat buggy as well.
                          Maybe those things are related somehow.

                          P 1 Reply Last reply Reply Quote 0
                          • P Offline
                            peder @peder
                            last edited by

                            I managed to download from master now as well, with the same gzip complaint and the same "Forkhelpers.Subprocess_failed(1)" in the log

                            gthvn1G 1 Reply Last reply Reply Quote 0
                            • gthvn1G Offline
                              gthvn1 Vates 🪐 XCP-ng Team @peder
                              last edited by

                              Does it fails using directly the xe CLI: xe host-logs-download file-name="/tmp/logs.gz" uuid=<hostuuid>
                              The host uuid can be any host of the pool. I didn't reproduce yet with 8.3 by there are not many logs on my hosts.

                              P 1 Reply Last reply Reply Quote 0
                              • P Offline
                                peder @gthvn1
                                last edited by

                                @gthvn1 Yeah, that works.
                                I can get the logs both from the same host and from the other one in the pool.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post