Stuck tasks within XO (rrd_updates)?

tanjix

@jedimarcus said in Stuck tasks within XO (rrd_updates)?:

@tanjix Check audit.log and xensource.log in /var/log

If you see any errors there?

[13:52 vmcluster01 ~]# cat /var/log/audit.log | grep rrd
(...)
Jan 25 13:27:02 vmcluster01 xapi: [20210125T12:27:02.152Z|audit||252297 INET :::80|handler:http/rrd_updates D:750a7f18dcb2|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:32:09 vmcluster01 xapi: [20210125T12:32:09.367Z|audit||253127 INET :::80|handler:http/rrd_updates D:ef6cdab4202a|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:37:17 vmcluster01 xapi: [20210125T12:37:17.497Z|audit||254056 INET :::80|handler:http/rrd_updates D:901de089dd50|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:42:30 vmcluster01 xapi: [20210125T12:42:30.082Z|audit||254876 INET :::80|handler:http/rrd_updates D:e9be8bc70f19|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
Jan 25 13:47:31 vmcluster01 xapi: [20210125T12:47:31.442Z|audit||255789 INET :::80|handler:http/rrd_updates D:f29b7c671ac1|audit] ('trackid=c62d355204466eab132e8a9db671552d' 'LOCAL_SUPERUSER' '' 'ALLOWED' 'OK' 'HTTP' 'http/rrd_updates' ())
[13:52 vmcluster01 ~]#

Nothing suspicious here.

[13:54 vmcluster01 ~]# cat /var/log/xensource.log | grep rrd
(...)
Jan 25 13:27:02 vmcluster01 xapi: [debug||252297 INET :::80|Get RRD updates. D:42a316633c15|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:27:56 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 644162
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1372160
Jan 25 13:29:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 727987
Jan 25 13:32:09 vmcluster01 xapi: [debug||253127 INET :::80|Get RRD updates. D:c5389f6a1534|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 811402
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1372160
Jan 25 13:32:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 560736
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 580752
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:35:09 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 997217
Jan 25 13:37:17 vmcluster01 xapi: [debug||254056 INET :::80|Get RRD updates. D:3badea6de893|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 506708
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:38:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1071264
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 837407
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:41:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 740562
Jan 25 13:42:30 vmcluster01 xapi: [debug||254876 INET :::80|Get RRD updates. D:5cdf089c4396|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505863
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:44:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072109
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505881
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:47:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072083
Jan 25 13:47:31 vmcluster01 xapi: [debug||255789 INET :::80|Get RRD updates. D:49078b4a7579|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:49:22 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 641704
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:50:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 936269
Jan 25 13:52:40 vmcluster01 xapi: [debug||256729 INET :::80|Get RRD updates. D:d0acbc896c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC live_words = 505866
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC heap_words = 1577984
Jan 25 13:53:10 vmcluster01 xcp-rrdd: [ info||7 ||rrdd_main] GC free_words = 1072106

Here we have some more interesting things:

Jan 25 13:49:22 vmcluster01 xcp-rrdd: [ info||0 monitor_write|main|rrdd_server] Failed to process plugin metrics file: xcp-rrdd-gpumon ((Invalid_argument\x0A  "Cstruct.blit_to_bytes src=[0,0](0) dst=[11] src-off=0 len=11"))

No idea, what to do here?

jedimarcus

I have the same entries in my 8.2 installation but I don't have the stuck tasks.

Danp

@tanjix Have you applied the ca-certificates update from the xcp-ng-testing repo? If so, maybe that is the source of your problem.

tanjix

@danp

@danp said in Stuck tasks within XO (rrd_updates)?:

@tanjix Have you applied the ca-certificates update from the xcp-ng-testing repo? If so, maybe that is the source of your problem.

Negative, I will do that and report later on. Thanks!

Danp

@tanjix I would hold off installing it for now. My concern was that the update was the source of the issue, but you ruled that out since it hasn't been installed yet.

tanjix

@olivierlambert

Maybe I found the problem.
In XO's log section I found an entry like:

Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list:

The detailed log says at the bottom:

"code": "ERR_TLS_CERT_ALTNAME_INVALID",
"url": "https://a.b.c.d/rrd_updates?cf=AVERAGE&host=true&interval=5&json=true&start=1611607291&session_id=OpaqueRef%3A7601e913-a96c-419e-a5ab-be65255ab3d7&task_id=OpaqueRef%3Aecd831ec-bd4f-4a40-82c4-f15059ffb377",
"message": "Hostname/IP does not match certificate's altnames: IP: a.b.c.d is not in the cert's list: ",

a.b.c.d is in this case the ip address of my pool master.

I installed TLS certificates (a wildcard certificate, not self signed but from a trusted authority), so that all of my hosts are reachable ssl-secured (done that through XO --> Home --> Hosts --> <host> --> Advanced and there at the bottom).

Of course, this only works if requests are made with the hostname and not with the ip address.

If I open the URL from above and replace the ip address through the hostname of the pool master, then it looks like it works.

So, how can this be fixed that these rrd_things make their calls with the hostname instead of the ip address?
Or did I do anything wrong with the certificates?

Thanks!

tanjix

@olivierlambert is there any update on this issue available?

Thanks!

olivierlambert

This is a normal XAPI behavior. I don't think there's a simple fix for that.

tanjix

@olivierlambert So, how could I get rid of these error messages and the fact, that I have plenty of these tasks running after a while?
I guess, restarting the toolstack every hour is not the preferred way...

Thanks!

olivierlambert

Be sure you are correctly connected to your host (no NAT or private IP you couldn't reach)

tanjix

@olivierlambert What are you referring to? What should I do? There is no NAT, private network etc involved. XO connected to the XCP-NG host (pool master) via its public hostname (for which the SSL certificate is issued) and NOT with its ip address only.

Thanks!

olivierlambert

Is the host using the same IP address internally? (than the public one)

tanjix

@olivierlambert What do you mean with "internally"? There is only one IP in use in my environment, which is the public IP. So yes, it's the same IP address.

olivierlambert

Can you do a xe host-param-get param-name=address uuid=<HOST_UUID>?

tanjix

@olivierlambert said in Stuck tasks within XO (rrd_updates)?:

xe host-param-get param-name=address uuid=

Sure, that one is returning one ip address, the public one I am talking about.

olivierlambert

And it's your management interface, right?

tanjix

@olivierlambert correct

tanjix

Any news available?

olivierlambert

Sorry I'm completely snowed under with too many different things

To my knowledge, those tasks should disappear after a bit, if there's no NAT I don't see any reason for them to stack.

tanjix

Hi Olivier,

thanks, but as I described initially, those tasks are not disappearing and summing up after a while.

Did you also consider my report --> here?

Thanks!