XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    "Download System logs" tgz-file does not work

    Scheduled Pinned Locked Moved Xen Orchestra
    17 Posts 8 Posters 505 Views 8 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      Chemikant784 @olivierlambert
      last edited by

      @olivierlambert

      Hi, thanks for your reply.

      To your questions:

      1. XO Commit 3aa36f9
      2. XCP-ng 8.2.1 LTS with latest updates (updated last saturday)

      Regards

      P 1 Reply Last reply Reply Quote 0
      • P Offline
        peder @Chemikant784
        last edited by

        Same problem here. In my case it downloads about 30MB before it stalls.

        XCP-ng version : 8.2.1 with March updates
        XO : I tried both 9c27b53df and the older 708b9b5bc
        Browser : Firefox and Chrome in Linux

        1 Reply Last reply Reply Quote 0
        • P Offline
          ph7 @olivierlambert
          last edited by ph7

          @olivierlambert
          Same in 8.3 and 37fbb
          edit: And in 5.103.1

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Ping @lsouai-vates it seems to be a bug, IDK if it's on XO or XCP-ng's side

            lsouai-vatesL 1 Reply Last reply Reply Quote 0
            • lsouai-vatesL Offline
              lsouai-vates Vates 🪐 XO Team @olivierlambert
              last edited by

              @Chemikant784 Hello, and thanks for your bug report.
              I'am currently asking the XO and XCP-NG team about it, and I keep you in touch when I have more information...

              lsouai-vatesL 1 Reply Last reply Reply Quote 0
              • lsouai-vatesL Offline
                lsouai-vates Vates 🪐 XO Team @lsouai-vates
                last edited by

                @stormi can you help on this subject on XCP-NG team side?

                1 Reply Last reply Reply Quote 0
                • stormiS Offline
                  stormi Vates 🪐 XCP-ng Team
                  last edited by stormi

                  I'd start with checking the usual logs (notably xensource.log) when the issue happens.

                  See https://docs.xcp-ng.org/troubleshooting/log-files/

                  Yes, I know, getting logs about an issue getting logs. Of course, I mean checking them via CLI on XCP-ng hosts :).

                  C P 2 Replies Last reply Reply Quote 0
                  • C Offline
                    Chemikant784 @stormi
                    last edited by

                    @stormi indeed not free of irony 😄

                    1 Reply Last reply Reply Quote 0
                    • P Offline
                      peder @stormi
                      last edited by

                      @stormi Any hint on what to look for?

                      I see these entries when the download starts but no subsequent errors or anything else that stands out when the download stalls

                      xapi: [ info||1520711 HTTPS 10.x.y.z->:::80|task.create D:b06a0108fbb7|taskhelper] task Xapi#getResource /host_logs_download R:8ea91ee79d9a (uuid:a-b-c-d-e) created (trackid=18e32f90af09a295c704ca6099aaef26) by task D:b06a0108fbb7
xapi: [debug||1520711 :::80||xapi_logs_download] running logs-download handler
xapi: [ info||1520711 :::80||taskhelper] task Xapi#getResource /host_logs_download R:8ea91ee79d9a forwarded (trackid=18e32f90af09a295c704ca6099aaef26)
xapi: [debug||1520711 HTTPS 10.x.y.z->:::80|Xapi#getResource /host_logs_download R:8ea91ee79d9a|xapi_logs_download] send the http headers
                      stormiS 1 Reply Last reply Reply Quote 0
                      • stormiS Offline
                        stormi Vates 🪐 XCP-ng Team @peder
                        last edited by

                        @peder No, but let me ping our XAPI developers @gthvn1 and @andriy.sultanov

                        A 1 Reply Last reply Reply Quote 0
                        • A Offline
                          andriy.sultanov Vates 🪐 XAPI & Network Team @stormi
                          last edited by andriy.sultanov

                          I've been able to reproduce this on 8.3 without XO involved (just by querying the /host_logs_download endpoint).

                          The only error I can see in xensource.log is at the end of a successful download call:

                          xapi: [debug||6246 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: HANDLE_INVALID: [ task; OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d ]
xapi: [error||6246 :::80||backtrace] get_host_logs_download failed with exception Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] Raised Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] 1/19 xapi Raised at file ocaml/database/db_rpc_client_v1.ml, line 33
xapi: [error||6246 :::80||backtrace] 2/19 xapi Called from file ocaml/xapi/db_actions.ml, line 2577
xapi: [error||6246 :::80||backtrace] 3/19 xapi Called from file ocaml/xapi/taskHelper.ml, line 272
xapi: [error||6246 :::80||backtrace] 4/19 xapi Called from file ocaml/xapi/context.ml, line 515
 xapi: [error||6246 :::80||backtrace] 5/19 xapi Called from file ocaml/xapi/context.ml, line 522
xapi: [error||6246 :::80||backtrace] 6/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 7/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 8/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 108
xapi: [error||6246 :::80||backtrace] 9/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6246 :::80||backtrace] 10/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6246 :::80||backtrace] 11/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 12/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 13/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace] 14/19 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6246 :::80||backtrace] 15/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 16/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 17/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6246 :::80||backtrace] 18/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6246 :::80||backtrace] 19/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace]
xapi: [error||6246 :::80||http_internal_errors] Responding with 500 Internal Error due to Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [debug||6246 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, lineApr  3 15:09:04 xcp-gtn-ip12 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace]
xapi: [ info||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] task Xapi#getResource /host_logs_download R:83146129fe1c forwarded
xapi: [debug||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] the status of R:83146129fe1c is failure; cannot set it to `failure
xapi: [debug||6372 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: INTERNAL_ERROR: [ Subprocess exited with unexpected code 1 ]
xapi: [error||6372 :::80||backtrace] get_host_logs_download failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 2/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 3/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 114
xapi: [error||6372 :::80||backtrace] 4/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80||backtrace] 5/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80||backtrace] 6/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 7/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 8/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace] 9/14 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 10/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 11/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 12/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6372 :::80||backtrace] 13/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6372 :::80||backtrace] 14/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace]
xapi: [error||6372 :::80||http_internal_errors] Responding with 500 Internal Error due to Forkhelpers.Subprocess_failed(1)
xapi: [debug||6372 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n

                          And then the subsequent download stalls without any indication. I will investigate this further but just to confirm, can any of you see these or similar errors in your logs? Do you have occasional successful downloads followed by failures?

                          P 1 Reply Last reply Reply Quote 1
                          • P Offline
                            peder @andriy.sultanov
                            last edited by

                            @andriy.sultanov I'm not seeing any errors after it stalls.

                            One funny thing is that the pool master usually stalls after approximately 135MB and the slave stalls after 230MB
                            /var/log on master is half the size of slave's so that may be why.

                            Another funny thing is that when I tried the slave once more it actually finished the download at 230MB but my subsequent "tar -xf" complained about "gzip: stdin: decompression OK, trailing garbage ignored"
                            And for some reason the files it extracted were 528MB whereas they're 498MB on the server.
                            Maybe that's a matter of my using XFS vs the server's ext3, though.

                            That "successful" download resulted in

                            xapi: [error||307417 :::80||backtrace] Xapi#getResource /host_logs_download R:51bd50ee890d failed with exception Forkhelpers.Subprocess_failed(1)

                            in xensource.log.
                            So it seems to fail to properly close the transfer session most of the time and the file compression seems to be somewhat buggy as well.
                            Maybe those things are related somehow.

                            P 1 Reply Last reply Reply Quote 0
                            • P Offline
                              peder @peder
                              last edited by

                              I managed to download from master now as well, with the same gzip complaint and the same "Forkhelpers.Subprocess_failed(1)" in the log

                              gthvn1G 1 Reply Last reply Reply Quote 0
                              • gthvn1G Offline
                                gthvn1 Vates 🪐 XCP-ng Team @peder
                                last edited by

                                Does it fails using directly the xe CLI: xe host-logs-download file-name="/tmp/logs.gz" uuid=<hostuuid>
                                The host uuid can be any host of the pool. I didn't reproduce yet with 8.3 by there are not many logs on my hosts.

                                P 1 Reply Last reply Reply Quote 0
                                • P Offline
                                  peder @gthvn1
                                  last edited by

                                  @gthvn1 Yeah, that works.
                                  I can get the logs both from the same host and from the other one in the pool.

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post