XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Endless Xapi#getResource /rrd_updates in tasks list

    Scheduled Pinned Locked Moved XCP-ng
    85 Posts 10 Posters 25.2k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      That's weird then 🤔 That would require a bit more investigation on the host to understand why the task is stuck 🤔

      MathieuM 1 Reply Last reply Reply Quote 0
      • MathieuM Offline
        Mathieu @olivierlambert
        last edited by

        @olivierlambert
        I would be happy to provide any extra info you need, no worries.
        The config on the host is pretty straightforward, just XCP-ng 8.2.1 with latest patch, no extra package except netdata-ui.

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          I would check the xensource.log I think, to see if I can spot any error.

          MathieuM 1 Reply Last reply Reply Quote 0
          • MathieuM Offline
            Mathieu @olivierlambert
            last edited by Mathieu

            @olivierlambert
            I'm not good at reading logs, but I don't see errors if I look around the time of the stuck task.

            All I see is a lot of this, which doesn't seem to be an error:

            Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 HTTPS "XO-IPv4-address"->:::80|task.create D:8c2a9729cdc7|taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c (uuid:fc9308de-a60e-0647-de77-5affe59f0d65) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:8c2a9729cdc7
Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 :::80||taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124062 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:0ce72cef0c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||87 |session_revalidation_func D:226b44cedeab|xapi_session] revalidating all external sessions in the local host
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 HTTPS "XO-IPv4-address"->:::80|task.create D:44cbbd1eaa07|taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 (uuid:ff57e9c2-0efe-1442-b914-f0fc03ad4298) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:44cbbd1eaa07
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 :::80||taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||123681 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:c02e53b83d06|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:d9a843aad4e0|taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 (uuid:0cf536c9-4aca-345a-21af-5b50517850a1) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d9a843aad4e0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 HTTPS "XO-IPv4-address"->:::80|task.create D:d5fe32e3d49d|taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 (uuid:136e5de2-7cff-e064-3889-1e53e94bc39a) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d5fe32e3d49d
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 HTTPS "XO-IPv4-address"->:::80|task.create D:a93c0ae92fcb|taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 (uuid:6d9f76fd-a1a4-a276-cf09-c48bda2c1bda) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a93c0ae92fcb
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 HTTPS "XO-IPv4-address"->:::80|task.create D:43d6297edb7a|taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 (uuid:d9c42476-a14c-5d22-7dad-07b4bc1beb34) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:43d6297edb7a
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:54f29cd94a60|taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c (uuid:8a409f1b-05fb-a649-dd1c-43995e9b083b) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:54f29cd94a60
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:22d06e0331f0|taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b (uuid:618903d7-0d37-4048-0d87-cc43ae2715d9) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:22d06e0331f0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124070 HTTPS "XO-IPv4-address"->:::80|task.create D:4b078a1f60f8|taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc (uuid:a853806a-c374-bd5a-4a70-0b5e653355ba) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:4b078a1f60f8
Mar 26 15:38:00 xcp-angelina xapi: [ info||124071 HTTPS "XO-IPv4-address"->:::80|task.create D:d3c88668426e|taskhelper] task Xapi#getResource /rrd_updates R:29c0be84d819 (uuid:96afdfa1-c411-1010-a02b-898f763ea2e7) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d3c88668426e
Mar 26 15:38:00 xcp-angelina xapi: [ info||124068 :::80||taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124068 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:204df6c50cf5|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 :::80||taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 :::80||taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124065 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:60d06c209765|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124064 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:45d68614eed1|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:df8c7a898e42|taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd (uuid:57e238cf-4990-e698-f92e-4db87790c025) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:df8c7a898e42
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 :::80||taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124063 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:14f097fd8f89|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 :::80||taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124066 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:96c553abff8b|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124069 :::80||taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 :::80||taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124069 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:d604b8b2b8fc|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124067 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:761a0f164b4c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124072 :::80||taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124072 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:1e5f14157edd|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded

            And if I track one of the stuck task by its UUID, there's only one occurrence in the logs:

            Mar 26 12:37:00 xcp-angelina xapi: [ info||55610 HTTPS "XO-IPv4-address"->:::80|task.create D:a75213c5e38a|taskhelper] task Xapi#getResource /rrd_updates R:24f7cb51c61e (uuid:02585f09-7fae-fac6-8e9c-d261ba19587d) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a75213c5e38a

            I can share more of the logs if needed.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              I wonder if XO can reach that stats URL or if there's something blocking it. In this very case, the usual suspect is a network issue between XO and the host, but if it's in the same network with nothing in the middle, IDK. Have you changed the MTU?

              MathieuM 1 Reply Last reply Reply Quote 0
              • MathieuM Offline
                Mathieu @olivierlambert
                last edited by

                @olivierlambert
                MTU is standard @1500.
                There's been no change in the network.
                That issue is only there with commit newer than 0ccfd4b.

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  You mean you don't have this issue since which commit exactly?

                  MathieuM 1 Reply Last reply Reply Quote 0
                  • MathieuM Offline
                    Mathieu @olivierlambert
                    last edited by

                    @olivierlambert
                    I mean if use commit 0ccfd4b, there is no issue, if I use a more recent one, like the latest 583c7, the issue is appearing.

                    1 Reply Last reply Reply Quote 1
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      Can you find the exact one causing the issue? That might be very helpful 🙂

                      1 2 Replies Last reply Reply Quote 0
                      • 1 Offline
                        14wkinnersley @olivierlambert
                        last edited by

                        @olivierlambert
                        I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build.

                        1 Reply Last reply Reply Quote 0
                        • 1 Offline
                          14wkinnersley @olivierlambert
                          last edited by

                          @olivierlambert
                          Appears the issue started on commit 6c16055 for fix(xen-api): various undici fixes

                          c6451cf - Mar 14 - Working, no hanging logs
                          6c16055 - Mar 15 - Broken - Hanging logs for Xapi#getResource /rrd_update

                          julien-fJ 1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Ping @julien-f

                            1 Reply Last reply Reply Quote 0
                            • julien-fJ Offline
                              julien-f Vates 🪐 Co-Founder XO Team @14wkinnersley
                              last edited by

                              @14wkinnersley Commit 583c728 should have fixed the issue, please test with latest master.

                              1 1 Reply Last reply Reply Quote 1
                              • 1 Offline
                                14wkinnersley @julien-f
                                last edited by

                                @julien-f I updated to master, but still seeing hanging logs for Xapi#getResource /rrd_update

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  Even after restarting the toolstack to get rid of the previous tasks?

                                  1 1 Reply Last reply Reply Quote 0
                                  • 1 Offline
                                    14wkinnersley @olivierlambert
                                    last edited by

                                    @olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.

                                    1 Reply Last reply Reply Quote 0
                                    • 1 Offline
                                      14wkinnersley
                                      last edited by 14wkinnersley

                                      Just wanted to provide some additional details/screenshots to help in anyway possible.

                                      My configuration has the master host at 192.168.1.4
                                      I have 3 hosts in the cluster.
                                      XO is on a Ubuntu VM at 192.168.1.15
                                      Firewall at my router is allowing all traffic in that same VLAN.
                                      Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.

                                      Task List:
                                      Screenshot 2024-03-27 081251.png

                                      Host - Master:
                                      Screenshot 2024-03-27 081306.png

                                      Commit:
                                      Screenshot 2024-03-27 081333.png

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        Can you try to destroy your XO folder, re-pull, rebuid to be sure there's no leftovers?

                                        1 2 Replies Last reply Reply Quote 0
                                        • 1 Offline
                                          14wkinnersley @olivierlambert
                                          last edited by

                                          @olivierlambert Will try it now and report back

                                          1 Reply Last reply Reply Quote 0
                                          • 1 Offline
                                            14wkinnersley @olivierlambert
                                            last edited by

                                            @olivierlambert
                                            No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post