XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Stuck tasks within XO (rrd_updates)?

    Scheduled Pinned Locked Moved Xen Orchestra
    32 Posts 4 Posters 8.6k Views 1 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jedimarcusJ Offline
      jedimarcus @tanjix
      last edited by

      @tanjix Check audit.log and xensource.log in /var/log

      If you see any errors there?

      T 1 Reply Last reply Reply Quote 0
      • T Offline
        tanjix @jedimarcus
        last edited by

        @jedimarcus said in Stuck tasks within XO (rrd_updates)?:

        @tanjix Check audit.log and xensource.log in /var/log

        If you see any errors there?

        [13:52 vmcluster01 ~]# cat /var/log/audit.log | grep rrd
(...)
Jan 25 13:27:02 vmcluster01 xapi: [20210125T12:27:02.152Z|audit||252297 INET :::80|handler:http/rrd_updates D:750a7f18dcb2|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:32:09 vmcluster01 xapi: [20210125T12:32:09.367Z|audit||253127 INET :::80|handler:http/rrd_updates D:ef6cdab4202a|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:37:17 vmcluster01 xapi: [20210125T12:37:17.497Z|audit||254056 INET :::80|handler:http/rrd_updates D:901de089dd50|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:42:30 vmcluster01 xapi: [20210125T12:42:30.082Z|audit||254876 INET :::80|handler:http/rrd_updates D:e9be8bc70f19|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:47:31 vmcluster01 xapi: [20210125T12:47:31.442Z|audit||255789 INET :::80|handler:http/rrd_updates D:f29b7c671ac1|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
[13:52 vmcluster01 ~]#

        Nothing suspicious here.

        [13:54 vmcluster01 ~]# cat /var/log/xensource.log | grep rrd
(...)
Jan 25 13:27:02 vmcluster01 xapi: [debug||252297 INET :::80|Get RRD updates. D:42a316633c15|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:27:56 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 644162
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1372160
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 727987
Jan 25 13:32:09 vmcluster01 xapi: [debug||253127 INET :::80|Get RRD updates. D:c5389f6a1534|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 811402
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1372160
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 560736
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 580752
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 997217
Jan 25 13:37:17 vmcluster01 xapi: [debug||254056 INET :::80|Get RRD updates. D:3badea6de893|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 506708
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1071264
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 837407
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 740562
Jan 25 13:42:30 vmcluster01 xapi: [debug||254876 INET :::80|Get RRD updates. D:5cdf089c4396|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505863
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072109
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505881
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072083
Jan 25 13:47:31 vmcluster01 xapi: [debug||255789 INET :::80|Get RRD updates. D:49078b4a7579|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:49:22 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 641704
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 936269
Jan 25 13:52:40 vmcluster01 xapi: [debug||256729 INET :::80|Get RRD updates. D:d0acbc896c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505866
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072106

        Here we have some more interesting things:

        Jan 25 13:49:22 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))

        No idea, what to do here?

        jedimarcusJ 1 Reply Last reply Reply Quote 0
        • jedimarcusJ Offline
          jedimarcus @tanjix
          last edited by

          I have the same entries in my 8.2 installation but I don't have the stuck tasks.

          1 Reply Last reply Reply Quote 0
          • DanpD Online
            Danp Pro Support Team
            last edited by

            @tanjix Have you applied the ca-certificates update from the xcp-ng-testing repo? If so, maybe that is the source of your problem.

            T 1 Reply Last reply Reply Quote 0
            • T Offline
              tanjix @Danp
              last edited by

              @danp

              @danp said in Stuck tasks within XO (rrd_updates)?:

              @tanjix Have you applied the ca-certificates update from the xcp-ng-testing repo? If so, maybe that is the source of your problem.

              Negative, I will do that and report later on. Thanks!

              DanpD 1 Reply Last reply Reply Quote 0
              • DanpD Online
                Danp Pro Support Team @tanjix
                last edited by

                @tanjix I would hold off installing it for now. My concern was that the update was the source of the issue, but you ruled that out since it hasn't been installed yet.

                1 Reply Last reply Reply Quote 0
                • T Offline
                  tanjix
                  last edited by

                  @olivierlambert

                  Maybe I found the problem.
                  In XO's log section I found an entry like:

                  Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list:

                  The detailed log says at the bottom:

                  "code": "ERR_TLS_CERT_ALTNAME_INVALID",
"url": "https://a.b.c.d/rrd_updates?cf=AVERAGE&host=true&interval=5&json=true&start=1611607291&session_id=OpaqueRef%3A7601e913-a96c-419e-a5ab-be65255ab3d7&task_id=OpaqueRef%3Aecd831ec-bd4f-4a40-82c4-f15059ffb377",
"message": "Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list: ",

                  a.b.c.d is in this case the ip address of my pool master.

                  I installed TLS certificates (a wildcard certificate, not self signed but from a trusted authority), so that all of my hosts are reachable ssl-secured (done that through XO --> Home --> Hosts --> <host> --> Advanced and there at the bottom).

                  Of course, this only works if requests are made with the hostname and not with the ip address.

                  If I open the URL from above and replace the ip address through the hostname of the pool master, then it looks like it works.

                  So, how can this be fixed that these rrd_things make their calls with the hostname instead of the ip address?
                  Or did I do anything wrong with the certificates?

                  Thanks!

                  T 1 Reply Last reply Reply Quote 0
                  • T Offline
                    tanjix @tanjix
                    last edited by

                    @olivierlambert is there any update on this issue available?

                    Thanks!

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      This is a normal XAPI behavior. I don't think there's a simple fix for that.

                      T 1 Reply Last reply Reply Quote 0
                      • T Offline
                        tanjix @olivierlambert
                        last edited by

                        @olivierlambert So, how could I get rid of these error messages and the fact, that I have plenty of these tasks running after a while?
                        I guess, restarting the toolstack every hour is not the preferred way... 😂

                        Thanks!

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          Be sure you are correctly connected to your host (no NAT or private IP you couldn't reach)

                          T 1 Reply Last reply Reply Quote 0
                          • T Offline
                            tanjix @olivierlambert
                            last edited by

                            @olivierlambert What are you referring to? What should I do? There is no NAT, private network etc involved. XO connected to the XCP-NG host (pool master) via its public hostname (for which the SSL certificate is issued) and NOT with its ip address only.

                            Thanks!

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Is the host using the same IP address internally? (than the public one)

                              T 1 Reply Last reply Reply Quote 0
                              • T Offline
                                tanjix @olivierlambert
                                last edited by

                                @olivierlambert What do you mean with "internally"? There is only one IP in use in my environment, which is the public IP. So yes, it's the same IP address.

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  Can you do a xe host-param-get param-name=address uuid=<HOST_UUID>?

                                  T 1 Reply Last reply Reply Quote 0
                                  • T Offline
                                    tanjix @olivierlambert
                                    last edited by

                                    @olivierlambert said in Stuck tasks within XO (rrd_updates)?:

                                    xe host-param-get param-name=address uuid=

                                    Sure, that one is returning one ip address, the public one I am talking about.

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      And it's your management interface, right?

                                      T 1 Reply Last reply Reply Quote 0
                                      • T Offline
                                        tanjix @olivierlambert
                                        last edited by

                                        @olivierlambert correct

                                        1 Reply Last reply Reply Quote 0
                                        • T Offline
                                          tanjix
                                          last edited by

                                          Any news available?

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by

                                            Sorry I'm completely snowed under with too many different things 😞

                                            To my knowledge, those tasks should disappear after a bit, if there's no NAT I don't see any reason for them to stack.

                                            T 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post