XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Endless Xapi#getResource /rrd_updates in tasks list

    Scheduled Pinned Locked Moved XCP-ng
    85 Posts 10 Posters 27.6k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • MathieuM Online
      Mathieu @olivierlambert
      last edited by

      @olivierlambert
      I would be happy to provide any extra info you need, no worries.
      The config on the host is pretty straightforward, just XCP-ng 8.2.1 with latest patch, no extra package except netdata-ui.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        I would check the xensource.log I think, to see if I can spot any error.

        MathieuM 1 Reply Last reply Reply Quote 0
        • MathieuM Online
          Mathieu @olivierlambert
          last edited by Mathieu

          @olivierlambert
          I'm not good at reading logs, but I don't see errors if I look around the time of the stuck task.

          All I see is a lot of this, which doesn't seem to be an error:

          Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 HTTPS "XO-IPv4-address"->:::80|task.create D:8c2a9729cdc7|taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c (uuid:fc9308de-a60e-0647-de77-5affe59f0d65) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:8c2a9729cdc7
Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 :::80||taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124062 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:0ce72cef0c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||87 |session_revalidation_func D:226b44cedeab|xapi_session] revalidating all external sessions in the local host
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 HTTPS "XO-IPv4-address"->:::80|task.create D:44cbbd1eaa07|taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 (uuid:ff57e9c2-0efe-1442-b914-f0fc03ad4298) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:44cbbd1eaa07
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 :::80||taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||123681 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:c02e53b83d06|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:d9a843aad4e0|taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 (uuid:0cf536c9-4aca-345a-21af-5b50517850a1) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d9a843aad4e0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 HTTPS "XO-IPv4-address"->:::80|task.create D:d5fe32e3d49d|taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 (uuid:136e5de2-7cff-e064-3889-1e53e94bc39a) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d5fe32e3d49d
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 HTTPS "XO-IPv4-address"->:::80|task.create D:a93c0ae92fcb|taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 (uuid:6d9f76fd-a1a4-a276-cf09-c48bda2c1bda) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a93c0ae92fcb
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 HTTPS "XO-IPv4-address"->:::80|task.create D:43d6297edb7a|taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 (uuid:d9c42476-a14c-5d22-7dad-07b4bc1beb34) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:43d6297edb7a
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:54f29cd94a60|taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c (uuid:8a409f1b-05fb-a649-dd1c-43995e9b083b) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:54f29cd94a60
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:22d06e0331f0|taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b (uuid:618903d7-0d37-4048-0d87-cc43ae2715d9) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:22d06e0331f0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124070 HTTPS "XO-IPv4-address"->:::80|task.create D:4b078a1f60f8|taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc (uuid:a853806a-c374-bd5a-4a70-0b5e653355ba) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:4b078a1f60f8
Mar 26 15:38:00 xcp-angelina xapi: [ info||124071 HTTPS "XO-IPv4-address"->:::80|task.create D:d3c88668426e|taskhelper] task Xapi#getResource /rrd_updates R:29c0be84d819 (uuid:96afdfa1-c411-1010-a02b-898f763ea2e7) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d3c88668426e
Mar 26 15:38:00 xcp-angelina xapi: [ info||124068 :::80||taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124068 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:204df6c50cf5|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 :::80||taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 :::80||taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124065 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:60d06c209765|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124064 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:45d68614eed1|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:df8c7a898e42|taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd (uuid:57e238cf-4990-e698-f92e-4db87790c025) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:df8c7a898e42
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 :::80||taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124063 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:14f097fd8f89|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 :::80||taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124066 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:96c553abff8b|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124069 :::80||taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 :::80||taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124069 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:d604b8b2b8fc|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124067 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:761a0f164b4c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124072 :::80||taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124072 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:1e5f14157edd|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded

          And if I track one of the stuck task by its UUID, there's only one occurrence in the logs:

          Mar 26 12:37:00 xcp-angelina xapi: [ info||55610 HTTPS "XO-IPv4-address"->:::80|task.create D:a75213c5e38a|taskhelper] task Xapi#getResource /rrd_updates R:24f7cb51c61e (uuid:02585f09-7fae-fac6-8e9c-d261ba19587d) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a75213c5e38a

          I can share more of the logs if needed.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            I wonder if XO can reach that stats URL or if there's something blocking it. In this very case, the usual suspect is a network issue between XO and the host, but if it's in the same network with nothing in the middle, IDK. Have you changed the MTU?

            MathieuM 1 Reply Last reply Reply Quote 0
            • MathieuM Online
              Mathieu @olivierlambert
              last edited by

              @olivierlambert
              MTU is standard @1500.
              There's been no change in the network.
              That issue is only there with commit newer than 0ccfd4b.

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                You mean you don't have this issue since which commit exactly?

                MathieuM 1 Reply Last reply Reply Quote 0
                • MathieuM Online
                  Mathieu @olivierlambert
                  last edited by

                  @olivierlambert
                  I mean if use commit 0ccfd4b, there is no issue, if I use a more recent one, like the latest 583c7, the issue is appearing.

                  1 Reply Last reply Reply Quote 1
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    Can you find the exact one causing the issue? That might be very helpful 🙂

                    1 2 Replies Last reply Reply Quote 0
                    • 1 Offline
                      14wkinnersley @olivierlambert
                      last edited by

                      @olivierlambert
                      I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build.

                      1 Reply Last reply Reply Quote 0
                      • 1 Offline
                        14wkinnersley @olivierlambert
                        last edited by

                        @olivierlambert
                        Appears the issue started on commit 6c16055 for fix(xen-api): various undici fixes

                        c6451cf - Mar 14 - Working, no hanging logs
                        6c16055 - Mar 15 - Broken - Hanging logs for Xapi#getResource /rrd_update

                        julien-fJ 1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          Ping @julien-f

                          1 Reply Last reply Reply Quote 0
                          • julien-fJ Offline
                            julien-f Vates 🪐 Co-Founder XO Team @14wkinnersley
                            last edited by

                            @14wkinnersley Commit 583c728 should have fixed the issue, please test with latest master.

                            1 1 Reply Last reply Reply Quote 1
                            • 1 Offline
                              14wkinnersley @julien-f
                              last edited by

                              @julien-f I updated to master, but still seeing hanging logs for Xapi#getResource /rrd_update

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Even after restarting the toolstack to get rid of the previous tasks?

                                1 1 Reply Last reply Reply Quote 0
                                • 1 Offline
                                  14wkinnersley @olivierlambert
                                  last edited by

                                  @olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.

                                  1 Reply Last reply Reply Quote 0
                                  • 1 Offline
                                    14wkinnersley
                                    last edited by 14wkinnersley

                                    Just wanted to provide some additional details/screenshots to help in anyway possible.

                                    My configuration has the master host at 192.168.1.4
                                    I have 3 hosts in the cluster.
                                    XO is on a Ubuntu VM at 192.168.1.15
                                    Firewall at my router is allowing all traffic in that same VLAN.
                                    Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.

                                    Task List:
                                    Screenshot 2024-03-27 081251.png

                                    Host - Master:
                                    Screenshot 2024-03-27 081306.png

                                    Commit:
                                    Screenshot 2024-03-27 081333.png

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      Can you try to destroy your XO folder, re-pull, rebuid to be sure there's no leftovers?

                                      1 2 Replies Last reply Reply Quote 0
                                      • 1 Offline
                                        14wkinnersley @olivierlambert
                                        last edited by

                                        @olivierlambert Will try it now and report back

                                        1 Reply Last reply Reply Quote 0
                                        • 1 Offline
                                          14wkinnersley @olivierlambert
                                          last edited by

                                          @olivierlambert
                                          No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building.

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by

                                            Okay thanks, we'll try to see if we can reproduce it here.

                                            MathieuM 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post