XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Stuck tasks within XO (rrd_updates)?

    Scheduled Pinned Locked Moved
    Xen Orchestra
    4
    32
    2.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • T
      tanjix @jedimarcus
      last edited by

      @jedimarcus said in Stuck tasks within XO (rrd_updates)?:

      @tanjix Check audit.log and xensource.log in /var/log

      If you see any errors there?

      [13:52 vmcluster01 ~]# cat /var/log/audit.log | grep rrd
(...)
Jan 25 13:27:02 vmcluster01 xapi: [20210125T12:27:02.152Z|audit||252297 INET :::80|handler:http/rrd_updates D:750a7f18dcb2|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:32:09 vmcluster01 xapi: [20210125T12:32:09.367Z|audit||253127 INET :::80|handler:http/rrd_updates D:ef6cdab4202a|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:37:17 vmcluster01 xapi: [20210125T12:37:17.497Z|audit||254056 INET :::80|handler:http/rrd_updates D:901de089dd50|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:42:30 vmcluster01 xapi: [20210125T12:42:30.082Z|audit||254876 INET :::80|handler:http/rrd_updates D:e9be8bc70f19|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:47:31 vmcluster01 xapi: [20210125T12:47:31.442Z|audit||255789 INET :::80|handler:http/rrd_updates D:f29b7c671ac1|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
[13:52 vmcluster01 ~]#

      Nothing suspicious here.

      [13:54 vmcluster01 ~]# cat /var/log/xensource.log | grep rrd
(...)
Jan 25 13:27:02 vmcluster01 xapi: [debug||252297 INET :::80|Get RRD updates. D:42a316633c15|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:27:56 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 644162
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1372160
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 727987
Jan 25 13:32:09 vmcluster01 xapi: [debug||253127 INET :::80|Get RRD updates. D:c5389f6a1534|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 811402
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1372160
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 560736
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 580752
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 997217
Jan 25 13:37:17 vmcluster01 xapi: [debug||254056 INET :::80|Get RRD updates. D:3badea6de893|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 506708
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1071264
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 837407
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 740562
Jan 25 13:42:30 vmcluster01 xapi: [debug||254876 INET :::80|Get RRD updates. D:5cdf089c4396|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505863
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072109
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505881
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072083
Jan 25 13:47:31 vmcluster01 xapi: [debug||255789 INET :::80|Get RRD updates. D:49078b4a7579|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:49:22 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 641704
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 936269
Jan 25 13:52:40 vmcluster01 xapi: [debug||256729 INET :::80|Get RRD updates. D:d0acbc896c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505866
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072106

      Here we have some more interesting things:

      Jan 25 13:49:22 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))

      No idea, what to do here?

      jedimarcusJ 1 Reply Last reply Reply Quote 0
      • jedimarcusJ
        jedimarcus @tanjix
        last edited by

        I have the same entries in my 8.2 installation but I don't have the stuck tasks.

        1 Reply Last reply Reply Quote 0
        • DanpD
          Danp Vates 🪐 Pro Support Team
          last edited by

          @tanjix Have you applied the ca-certificates update from the xcp-ng-testing repo? If so, maybe that is the source of your problem.

          T 1 Reply Last reply Reply Quote 0
          • T
            tanjix @Danp
            last edited by

            @danp

            @danp said in Stuck tasks within XO (rrd_updates)?:

            @tanjix Have you applied the ca-certificates update from the xcp-ng-testing repo? If so, maybe that is the source of your problem.

            Negative, I will do that and report later on. Thanks!

            DanpD 1 Reply Last reply Reply Quote 0
            • DanpD
              Danp Vates 🪐 Pro Support Team @tanjix
              last edited by

              @tanjix I would hold off installing it for now. My concern was that the update was the source of the issue, but you ruled that out since it hasn't been installed yet.

              1 Reply Last reply Reply Quote 0
              • T
                tanjix
                last edited by

                @olivierlambert

                Maybe I found the problem.
                In XO's log section I found an entry like:

                Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list:

                The detailed log says at the bottom:

                "code": "ERR_TLS_CERT_ALTNAME_INVALID",
"url": "https://a.b.c.d/rrd_updates?cf=AVERAGE&host=true&interval=5&json=true&start=1611607291&session_id=OpaqueRef%3A7601e913-a96c-419e-a5ab-be65255ab3d7&task_id=OpaqueRef%3Aecd831ec-bd4f-4a40-82c4-f15059ffb377",
"message": "Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list: ",

                a.b.c.d is in this case the ip address of my pool master.

                I installed TLS certificates (a wildcard certificate, not self signed but from a trusted authority), so that all of my hosts are reachable ssl-secured (done that through XO --> Home --> Hosts --> <host> --> Advanced and there at the bottom).

                Of course, this only works if requests are made with the hostname and not with the ip address.

                If I open the URL from above and replace the ip address through the hostname of the pool master, then it looks like it works.

                So, how can this be fixed that these rrd_things make their calls with the hostname instead of the ip address?
                Or did I do anything wrong with the certificates?

                Thanks!

                T 1 Reply Last reply Reply Quote 0
                • T
                  tanjix @tanjix
                  last edited by

                  @olivierlambert is there any update on this issue available?

                  Thanks!

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    This is a normal XAPI behavior. I don't think there's a simple fix for that.

                    T 1 Reply Last reply Reply Quote 0
                    • T
                      tanjix @olivierlambert
                      last edited by

                      @olivierlambert So, how could I get rid of these error messages and the fact, that I have plenty of these tasks running after a while?
                      I guess, restarting the toolstack every hour is not the preferred way... 😂

                      Thanks!

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Be sure you are correctly connected to your host (no NAT or private IP you couldn't reach)

                        T 1 Reply Last reply Reply Quote 0
                        • T
                          tanjix @olivierlambert
                          last edited by

                          @olivierlambert What are you referring to? What should I do? There is no NAT, private network etc involved. XO connected to the XCP-NG host (pool master) via its public hostname (for which the SSL certificate is issued) and NOT with its ip address only.

                          Thanks!

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Is the host using the same IP address internally? (than the public one)

                            T 1 Reply Last reply Reply Quote 0
                            • T
                              tanjix @olivierlambert
                              last edited by

                              @olivierlambert What do you mean with "internally"? There is only one IP in use in my environment, which is the public IP. So yes, it's the same IP address.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Can you do a xe host-param-get param-name=address uuid=<HOST_UUID>?

                                T 1 Reply Last reply Reply Quote 0
                                • T
                                  tanjix @olivierlambert
                                  last edited by

                                  @olivierlambert said in Stuck tasks within XO (rrd_updates)?:

                                  xe host-param-get param-name=address uuid=

                                  Sure, that one is returning one ip address, the public one I am talking about.

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    And it's your management interface, right?

                                    T 1 Reply Last reply Reply Quote 0
                                    • T
                                      tanjix @olivierlambert
                                      last edited by

                                      @olivierlambert correct

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        tanjix
                                        last edited by

                                        Any news available?

                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          Sorry I'm completely snowed under with too many different things 😞

                                          To my knowledge, those tasks should disappear after a bit, if there's no NAT I don't see any reason for them to stack.

                                          T 1 Reply Last reply Reply Quote 0
                                          • T
                                            tanjix @olivierlambert
                                            last edited by

                                            Hi Olivier,

                                            thanks, but as I described initially, those tasks are not disappearing and summing up after a while.

                                            Did you also consider my report --> here?

                                            Thanks!

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post