XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Endless Xapi#getResource /rrd_updates in tasks list

    Scheduled Pinned Locked Moved XCP-ng
    85 Posts 10 Posters 25.2k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • 1 Offline
      14wkinnersley
      last edited by 14wkinnersley

      I am also having this same issue after updating all the patches and XO. XO from sources. Commit 214b8. Was there a solution to this? Logged in today, and had close to 2000 tasks.

      MathieuM 1 Reply Last reply Reply Quote 0
      • MathieuM Offline
        Mathieu @14wkinnersley
        last edited by Mathieu

        @14wkinnersley
        That issue is definitely coming from XO from source, it does not appear with XOA.

        I revert my XO to commit 0ccfd4b and now everything is back to normal.

        I don't know if our issue is also related to self-signed cert, but there's been other issues with latest commit of XO, we should stick to that version until a solution is implemented.

        1 Reply Last reply Reply Quote 1
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Adding @julien-f in the loop

          1 Reply Last reply Reply Quote 0
          • 1 Offline
            14wkinnersley
            last edited by 14wkinnersley

            I never installed any certificates on my instance. I can rollback unless anyone needs me to stay on the latest and for any testing. Up to about 1500 tasks in the past 20 hours since I rebooted all my hosts and xo last night.

            1 Reply Last reply Reply Quote 0
            • MathieuM Offline
              Mathieu
              last edited by

              I tried latest commit 583c7.
              It solved restore, health check and disaster recovery certificate issues, but unfortunately, I still got the Xapi#getResource /rrd_updates tasks growing in the list.

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by olivierlambert

                @Mathieu have you restarted to toolstack on the XCP-ng side? It could be also unrelated to the bug fixed recently. Is your host with the issue is behind a NAT or something?

                MathieuM 1 Reply Last reply Reply Quote 0
                • MathieuM Offline
                  Mathieu @olivierlambert
                  last edited by

                  @olivierlambert
                  Yes, toolstack restarted several times. It clears the task, and then, after a while, the task is appearing again.

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    In the host view, "General" tab, is the IP of the host visible matches the IP added in Settings/server of XO?

                    MathieuM 1 Reply Last reply Reply Quote 0
                    • MathieuM Offline
                      Mathieu @olivierlambert
                      last edited by

                      @olivierlambert
                      Yes, the IP is matching. There has been no change in IP address, and there's no routing involved, no proxy or whatever, XO and the host are in the same subnet.

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        That's weird then 🤔 That would require a bit more investigation on the host to understand why the task is stuck 🤔

                        MathieuM 1 Reply Last reply Reply Quote 0
                        • MathieuM Offline
                          Mathieu @olivierlambert
                          last edited by

                          @olivierlambert
                          I would be happy to provide any extra info you need, no worries.
                          The config on the host is pretty straightforward, just XCP-ng 8.2.1 with latest patch, no extra package except netdata-ui.

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            I would check the xensource.log I think, to see if I can spot any error.

                            MathieuM 1 Reply Last reply Reply Quote 0
                            • MathieuM Offline
                              Mathieu @olivierlambert
                              last edited by Mathieu

                              @olivierlambert
                              I'm not good at reading logs, but I don't see errors if I look around the time of the stuck task.

                              All I see is a lot of this, which doesn't seem to be an error:

                              Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 HTTPS "XO-IPv4-address"->:::80|task.create D:8c2a9729cdc7|taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c (uuid:fc9308de-a60e-0647-de77-5affe59f0d65) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:8c2a9729cdc7
Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 :::80||taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124062 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:0ce72cef0c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||87 |session_revalidation_func D:226b44cedeab|xapi_session] revalidating all external sessions in the local host
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 HTTPS "XO-IPv4-address"->:::80|task.create D:44cbbd1eaa07|taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 (uuid:ff57e9c2-0efe-1442-b914-f0fc03ad4298) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:44cbbd1eaa07
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 :::80||taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||123681 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:c02e53b83d06|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:d9a843aad4e0|taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 (uuid:0cf536c9-4aca-345a-21af-5b50517850a1) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d9a843aad4e0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 HTTPS "XO-IPv4-address"->:::80|task.create D:d5fe32e3d49d|taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 (uuid:136e5de2-7cff-e064-3889-1e53e94bc39a) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d5fe32e3d49d
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 HTTPS "XO-IPv4-address"->:::80|task.create D:a93c0ae92fcb|taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 (uuid:6d9f76fd-a1a4-a276-cf09-c48bda2c1bda) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a93c0ae92fcb
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 HTTPS "XO-IPv4-address"->:::80|task.create D:43d6297edb7a|taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 (uuid:d9c42476-a14c-5d22-7dad-07b4bc1beb34) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:43d6297edb7a
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:54f29cd94a60|taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c (uuid:8a409f1b-05fb-a649-dd1c-43995e9b083b) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:54f29cd94a60
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:22d06e0331f0|taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b (uuid:618903d7-0d37-4048-0d87-cc43ae2715d9) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:22d06e0331f0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124070 HTTPS "XO-IPv4-address"->:::80|task.create D:4b078a1f60f8|taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc (uuid:a853806a-c374-bd5a-4a70-0b5e653355ba) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:4b078a1f60f8
Mar 26 15:38:00 xcp-angelina xapi: [ info||124071 HTTPS "XO-IPv4-address"->:::80|task.create D:d3c88668426e|taskhelper] task Xapi#getResource /rrd_updates R:29c0be84d819 (uuid:96afdfa1-c411-1010-a02b-898f763ea2e7) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d3c88668426e
Mar 26 15:38:00 xcp-angelina xapi: [ info||124068 :::80||taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124068 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:204df6c50cf5|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 :::80||taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 :::80||taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124065 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:60d06c209765|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124064 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:45d68614eed1|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:df8c7a898e42|taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd (uuid:57e238cf-4990-e698-f92e-4db87790c025) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:df8c7a898e42
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 :::80||taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124063 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:14f097fd8f89|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 :::80||taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124066 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:96c553abff8b|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124069 :::80||taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 :::80||taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124069 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:d604b8b2b8fc|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124067 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:761a0f164b4c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124072 :::80||taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124072 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:1e5f14157edd|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded

                              And if I track one of the stuck task by its UUID, there's only one occurrence in the logs:

                              Mar 26 12:37:00 xcp-angelina xapi: [ info||55610 HTTPS "XO-IPv4-address"->:::80|task.create D:a75213c5e38a|taskhelper] task Xapi#getResource /rrd_updates R:24f7cb51c61e (uuid:02585f09-7fae-fac6-8e9c-d261ba19587d) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a75213c5e38a

                              I can share more of the logs if needed.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                I wonder if XO can reach that stats URL or if there's something blocking it. In this very case, the usual suspect is a network issue between XO and the host, but if it's in the same network with nothing in the middle, IDK. Have you changed the MTU?

                                MathieuM 1 Reply Last reply Reply Quote 0
                                • MathieuM Offline
                                  Mathieu @olivierlambert
                                  last edited by

                                  @olivierlambert
                                  MTU is standard @1500.
                                  There's been no change in the network.
                                  That issue is only there with commit newer than 0ccfd4b.

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    You mean you don't have this issue since which commit exactly?

                                    MathieuM 1 Reply Last reply Reply Quote 0
                                    • MathieuM Offline
                                      Mathieu @olivierlambert
                                      last edited by

                                      @olivierlambert
                                      I mean if use commit 0ccfd4b, there is no issue, if I use a more recent one, like the latest 583c7, the issue is appearing.

                                      1 Reply Last reply Reply Quote 1
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        Can you find the exact one causing the issue? That might be very helpful 🙂

                                        1 2 Replies Last reply Reply Quote 0
                                        • 1 Offline
                                          14wkinnersley @olivierlambert
                                          last edited by

                                          @olivierlambert
                                          I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build.

                                          1 Reply Last reply Reply Quote 0
                                          • 1 Offline
                                            14wkinnersley @olivierlambert
                                            last edited by

                                            @olivierlambert
                                            Appears the issue started on commit 6c16055 for fix(xen-api): various undici fixes

                                            c6451cf - Mar 14 - Working, no hanging logs
                                            6c16055 - Mar 15 - Broken - Hanging logs for Xapi#getResource /rrd_update

                                            julien-fJ 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post