XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    "Download System logs" tgz-file does not work

    Scheduled Pinned Locked Moved Xen Orchestra
    17 Posts 8 Posters 294 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      Chemikant784
      last edited by Chemikant784

      Hi All

      I am just curious if i am the only one who is facing this.

      I have an actual XenOrchestra which is build from the sources. Recently i've noticed, that i'm not able to download the systemlogs tgz file from Hosts -> Host -> Advanced Tab -> yellow button "Download system logs".

      I have two Hosts in a pool, they both behave the same with any browser:

      1. Download starts without estimation but at high speed
      2. Some Megabytes are downloaded (~180MiB when trying on masterhost while 90MiB on slavehost)
      3. Download speed suddenly stop an stuck at 0 for some minutes
      4. Browser considered the download as failed
      5. The file "logs.tgz" is corrupted an cannot be read.

      With google search i did not found anything useful. Am i missing something or is there currently a problem with this feature?

      Kind regards and have a nice day!

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Hi!

        We need more info to assist:

        1. XO commit
        2. XCP-ng version (and must be up to date)
        C P 2 Replies Last reply Reply Quote 0
        • C Offline
          Chemikant784 @olivierlambert
          last edited by

          @olivierlambert

          Hi, thanks for your reply.

          To your questions:

          1. XO Commit 3aa36f9
          2. XCP-ng 8.2.1 LTS with latest updates (updated last saturday)

          Regards

          P 1 Reply Last reply Reply Quote 0
          • P Offline
            peder @Chemikant784
            last edited by

            Same problem here. In my case it downloads about 30MB before it stalls.

            XCP-ng version : 8.2.1 with March updates
            XO : I tried both 9c27b53df and the older 708b9b5bc
            Browser : Firefox and Chrome in Linux

            1 Reply Last reply Reply Quote 0
            • P Offline
              ph7 @olivierlambert
              last edited by ph7

              @olivierlambert
              Same in 8.3 and 37fbb
              edit: And in 5.103.1

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Ping @lsouai-vates it seems to be a bug, IDK if it's on XO or XCP-ng's side

                lsouai-vatesL 1 Reply Last reply Reply Quote 0
                • lsouai-vatesL Offline
                  lsouai-vates Vates 🪐 XO Team @olivierlambert
                  last edited by

                  @Chemikant784 Hello, and thanks for your bug report.
                  I'am currently asking the XO and XCP-NG team about it, and I keep you in touch when I have more information...

                  lsouai-vatesL 1 Reply Last reply Reply Quote 0
                  • lsouai-vatesL Offline
                    lsouai-vates Vates 🪐 XO Team @lsouai-vates
                    last edited by

                    @stormi can you help on this subject on XCP-NG team side?

                    1 Reply Last reply Reply Quote 0
                    • stormiS Offline
                      stormi Vates 🪐 XCP-ng Team
                      last edited by stormi

                      I'd start with checking the usual logs (notably xensource.log) when the issue happens.

                      See https://docs.xcp-ng.org/troubleshooting/log-files/

                      Yes, I know, getting logs about an issue getting logs. Of course, I mean checking them via CLI on XCP-ng hosts :).

                      C P 2 Replies Last reply Reply Quote 0
                      • C Offline
                        Chemikant784 @stormi
                        last edited by

                        @stormi indeed not free of irony 😄

                        1 Reply Last reply Reply Quote 0
                        • P Offline
                          peder @stormi
                          last edited by

                          @stormi Any hint on what to look for?

                          I see these entries when the download starts but no subsequent errors or anything else that stands out when the download stalls

                          xapi: [ info||1520711 HTTPS 10.x.y.z->:::80|task.create D:b06a0108fbb7|taskhelper] task Xapi#getResource /host_logs_download R:8ea91ee79d9a (uuid:a-b-c-d-e) created (trackid=18e32f90af09a295c704ca6099aaef26) by task D:b06a0108fbb7
xapi: [debug||1520711 :::80||xapi_logs_download] running logs-download handler
xapi: [ info||1520711 :::80||taskhelper] task Xapi#getResource /host_logs_download R:8ea91ee79d9a forwarded (trackid=18e32f90af09a295c704ca6099aaef26)
xapi: [debug||1520711 HTTPS 10.x.y.z->:::80|Xapi#getResource /host_logs_download R:8ea91ee79d9a|xapi_logs_download] send the http headers
                          stormiS 1 Reply Last reply Reply Quote 0
                          • stormiS Offline
                            stormi Vates 🪐 XCP-ng Team @peder
                            last edited by

                            @peder No, but let me ping our XAPI developers @gthvn1 and @andriy.sultanov

                            A 1 Reply Last reply Reply Quote 0
                            • A Offline
                              andriy.sultanov XAPI Vizier @stormi
                              last edited by andriy.sultanov

                              I've been able to reproduce this on 8.3 without XO involved (just by querying the /host_logs_download endpoint).

                              The only error I can see in xensource.log is at the end of a successful download call:

                              xapi: [debug||6246 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: HANDLE_INVALID: [ task; OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d ]
xapi: [error||6246 :::80||backtrace] get_host_logs_download failed with exception Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] Raised Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [error||6246 :::80||backtrace] 1/19 xapi Raised at file ocaml/database/db_rpc_client_v1.ml, line 33
xapi: [error||6246 :::80||backtrace] 2/19 xapi Called from file ocaml/xapi/db_actions.ml, line 2577
xapi: [error||6246 :::80||backtrace] 3/19 xapi Called from file ocaml/xapi/taskHelper.ml, line 272
xapi: [error||6246 :::80||backtrace] 4/19 xapi Called from file ocaml/xapi/context.ml, line 515
 xapi: [error||6246 :::80||backtrace] 5/19 xapi Called from file ocaml/xapi/context.ml, line 522
xapi: [error||6246 :::80||backtrace] 6/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 7/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 8/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 108
xapi: [error||6246 :::80||backtrace] 9/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6246 :::80||backtrace] 10/19 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6246 :::80||backtrace] 11/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 12/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 13/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace] 14/19 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6246 :::80||backtrace] 15/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6246 :::80||backtrace] 16/19 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6246 :::80||backtrace] 17/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6246 :::80||backtrace] 18/19 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6246 :::80||backtrace] 19/19 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6246 :::80||backtrace]
xapi: [error||6246 :::80||http_internal_errors] Responding with 500 Internal Error due to Db_exn.DBCache_NotFound("missing row", "task", "OpaqueRef:b40bd070-307e-f05d-91b5-562dc2f71e6d")
xapi: [debug||6246 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, lineApr  3 15:09:04 xcp-gtn-ip12 xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 4/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 5/6 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace] 6/6 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|handler:http/get_host_logs_download D:4e4d887163d8|backtrace]
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] handler:http/get_host_logs_download D:4e4d887163d8 failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 1/13 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 2/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 3/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 4/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 5/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 6/13 xapi Called from file ocaml/xapi/xapi_http.ml, line 257
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 7/13 xapi Called from file ocaml/xapi/rbac.ml, line 188
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 8/13 xapi Called from file ocaml/xapi/rbac.ml, line 197
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 9/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 10/13 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 12/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace] 13/13 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80|xapi_http_session_check D:7cc5eb59b433|backtrace]
xapi: [ info||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] task Xapi#getResource /host_logs_download R:83146129fe1c forwarded
xapi: [debug||6372 :::80|xapi_http_session_check D:7cc5eb59b433|taskhelper] the status of R:83146129fe1c is failure; cannot set it to `failure
xapi: [debug||6372 HTTPS 10.1.0.100->:::80|get_host_logs_download|xapi_http] Leaving RBAC-handler in xapi_http after: INTERNAL_ERROR: [ Subprocess exited with unexpected code 1 ]
xapi: [error||6372 :::80||backtrace] get_host_logs_download failed with exception Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] Raised Forkhelpers.Subprocess_failed(1)
xapi: [error||6372 :::80||backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 2/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 3/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 114
xapi: [error||6372 :::80||backtrace] 4/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 69
xapi: [error||6372 :::80||backtrace] 5/14 xapi Called from file ocaml/xapi/server_helpers.ml, line 96
xapi: [error||6372 :::80||backtrace] 6/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 7/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 8/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace] 9/14 xapi Called from file ocaml/libs/log/debug.ml, line 267
xapi: [error||6372 :::80||backtrace] 10/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error||6372 :::80||backtrace] 11/14 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
xapi: [error||6372 :::80||backtrace] 12/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 364
xapi: [error||6372 :::80||backtrace] 13/14 xapi Called from file ocaml/xapi/xapi_http.ml, line 370
xapi: [error||6372 :::80||backtrace] 14/14 xapi Called from file ocaml/libs/log/debug.ml, line 250
xapi: [error||6372 :::80||backtrace]
xapi: [error||6372 :::80||http_internal_errors] Responding with 500 Internal Error due to Forkhelpers.Subprocess_failed(1)
xapi: [debug||6372 :::80||http_internal_errors] Raised at Debug.with_thread_associated in file \\"ocaml/libs/log/debug.ml\\", line 267, characters 6-15\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 509, characters 4-32\\nCalled from Http_svr.let@ in file \\"ocaml/libs/http-lib/http_svr.ml\\" (inlined), line 46, characters 19-22\\nCalled from Http_svr.handle_one in file \\"ocaml/libs/http-lib/http_svr.ml\\", line 508, characters 4-195\\n

                              And then the subsequent download stalls without any indication. I will investigate this further but just to confirm, can any of you see these or similar errors in your logs? Do you have occasional successful downloads followed by failures?

                              P 1 Reply Last reply Reply Quote 1
                              • P Offline
                                peder @andriy.sultanov
                                last edited by

                                @andriy.sultanov I'm not seeing any errors after it stalls.

                                One funny thing is that the pool master usually stalls after approximately 135MB and the slave stalls after 230MB
                                /var/log on master is half the size of slave's so that may be why.

                                Another funny thing is that when I tried the slave once more it actually finished the download at 230MB but my subsequent "tar -xf" complained about "gzip: stdin: decompression OK, trailing garbage ignored"
                                And for some reason the files it extracted were 528MB whereas they're 498MB on the server.
                                Maybe that's a matter of my using XFS vs the server's ext3, though.

                                That "successful" download resulted in

                                xapi: [error||307417 :::80||backtrace] Xapi#getResource /host_logs_download R:51bd50ee890d failed with exception Forkhelpers.Subprocess_failed(1)

                                in xensource.log.
                                So it seems to fail to properly close the transfer session most of the time and the file compression seems to be somewhat buggy as well.
                                Maybe those things are related somehow.

                                P 1 Reply Last reply Reply Quote 0
                                • P Offline
                                  peder @peder
                                  last edited by

                                  I managed to download from master now as well, with the same gzip complaint and the same "Forkhelpers.Subprocess_failed(1)" in the log

                                  gthvn1G 1 Reply Last reply Reply Quote 0
                                  • gthvn1G Offline
                                    gthvn1 Vates 🪐 XCP-ng Team @peder
                                    last edited by

                                    Does it fails using directly the xe CLI: xe host-logs-download file-name="/tmp/logs.gz" uuid=<hostuuid>
                                    The host uuid can be any host of the pool. I didn't reproduce yet with 8.3 by there are not many logs on my hosts.

                                    P 1 Reply Last reply Reply Quote 0
                                    • P Offline
                                      peder @gthvn1
                                      last edited by

                                      @gthvn1 Yeah, that works.
                                      I can get the logs both from the same host and from the other one in the pool.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post