Endless Xapi#getResource /rrd_updates in tasks list

Mathieu

Yes, I'm using XO from source, commit e52b1 (hosted on a different host).

And all my hosts have been rebooted after the last round of patches.
I've reset one more time the host with the issue, but that didn't change anything.

I also rebooted my XO instance.
I'm gonna try to switch off the XO instance and only let the XOA running, to see if that does change something.

14wkinnersley

I am also having this same issue after updating all the patches and XO. XO from sources. Commit 214b8. Was there a solution to this? Logged in today, and had close to 2000 tasks.

Mathieu

@14wkinnersley
That issue is definitely coming from XO from source, it does not appear with XOA.

I revert my XO to commit 0ccfd4b and now everything is back to normal.

I don't know if our issue is also related to self-signed cert, but there's been other issues with latest commit of XO, we should stick to that version until a solution is implemented.

olivierlambert

Adding @julien-f in the loop

14wkinnersley

I never installed any certificates on my instance. I can rollback unless anyone needs me to stay on the latest and for any testing. Up to about 1500 tasks in the past 20 hours since I rebooted all my hosts and xo last night.

Mathieu

I tried latest commit 583c7.
It solved restore, health check and disaster recovery certificate issues, but unfortunately, I still got the Xapi#getResource /rrd_updates tasks growing in the list.

olivierlambert

@Mathieu have you restarted to toolstack on the XCP-ng side? It could be also unrelated to the bug fixed recently. Is your host with the issue is behind a NAT or something?

Mathieu

@olivierlambert
Yes, toolstack restarted several times. It clears the task, and then, after a while, the task is appearing again.

olivierlambert

In the host view, "General" tab, is the IP of the host visible matches the IP added in Settings/server of XO?

Mathieu

@olivierlambert
Yes, the IP is matching. There has been no change in IP address, and there's no routing involved, no proxy or whatever, XO and the host are in the same subnet.

olivierlambert

That's weird then That would require a bit more investigation on the host to understand why the task is stuck

Mathieu

@olivierlambert
I would be happy to provide any extra info you need, no worries.
The config on the host is pretty straightforward, just XCP-ng 8.2.1 with latest patch, no extra package except netdata-ui.

olivierlambert

I would check the xensource.log I think, to see if I can spot any error.

Mathieu

@olivierlambert
I'm not good at reading logs, but I don't see errors if I look around the time of the stuck task.

All I see is a lot of this, which doesn't seem to be an error:

Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 HTTPS "XO-IPv4-address"->:::80|task.create D:8c2a9729cdc7|taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c (uuid:fc9308de-a60e-0647-de77-5affe59f0d65) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:8c2a9729cdc7
Mar 26 15:38:00 xcp-angelina xapi: [ info||124062 :::80||taskhelper] task Xapi#getResource /rrd_updates R:0ce72cef0c0c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124062 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:0ce72cef0c0c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||87 |session_revalidation_func D:226b44cedeab|xapi_session] revalidating all external sessions in the local host
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 HTTPS "XO-IPv4-address"->:::80|task.create D:44cbbd1eaa07|taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 (uuid:ff57e9c2-0efe-1442-b914-f0fc03ad4298) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:44cbbd1eaa07
Mar 26 15:38:00 xcp-angelina xapi: [ info||123681 :::80||taskhelper] task Xapi#getResource /rrd_updates R:c02e53b83d06 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||123681 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:c02e53b83d06|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:d9a843aad4e0|taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 (uuid:0cf536c9-4aca-345a-21af-5b50517850a1) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d9a843aad4e0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 HTTPS "XO-IPv4-address"->:::80|task.create D:d5fe32e3d49d|taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 (uuid:136e5de2-7cff-e064-3889-1e53e94bc39a) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d5fe32e3d49d
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 HTTPS "XO-IPv4-address"->:::80|task.create D:a93c0ae92fcb|taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 (uuid:6d9f76fd-a1a4-a276-cf09-c48bda2c1bda) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a93c0ae92fcb
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 HTTPS "XO-IPv4-address"->:::80|task.create D:43d6297edb7a|taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 (uuid:d9c42476-a14c-5d22-7dad-07b4bc1beb34) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:43d6297edb7a
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:54f29cd94a60|taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c (uuid:8a409f1b-05fb-a649-dd1c-43995e9b083b) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:54f29cd94a60
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 HTTPS "XO-IPv4-address"->:::80|task.create D:22d06e0331f0|taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b (uuid:618903d7-0d37-4048-0d87-cc43ae2715d9) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:22d06e0331f0
Mar 26 15:38:00 xcp-angelina xapi: [ info||124070 HTTPS "XO-IPv4-address"->:::80|task.create D:4b078a1f60f8|taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc (uuid:a853806a-c374-bd5a-4a70-0b5e653355ba) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:4b078a1f60f8
Mar 26 15:38:00 xcp-angelina xapi: [ info||124071 HTTPS "XO-IPv4-address"->:::80|task.create D:d3c88668426e|taskhelper] task Xapi#getResource /rrd_updates R:29c0be84d819 (uuid:96afdfa1-c411-1010-a02b-898f763ea2e7) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:d3c88668426e
Mar 26 15:38:00 xcp-angelina xapi: [ info||124068 :::80||taskhelper] task Xapi#getResource /rrd_updates R:204df6c50cf5 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124068 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:204df6c50cf5|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124065 :::80||taskhelper] task Xapi#getResource /rrd_updates R:60d06c209765 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124064 :::80||taskhelper] task Xapi#getResource /rrd_updates R:45d68614eed1 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124065 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:60d06c209765|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124064 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:45d68614eed1|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 HTTPS "XO-IPv4-address"->:::80|task.create D:df8c7a898e42|taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd (uuid:57e238cf-4990-e698-f92e-4db87790c025) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:df8c7a898e42
Mar 26 15:38:00 xcp-angelina xapi: [ info||124063 :::80||taskhelper] task Xapi#getResource /rrd_updates R:14f097fd8f89 forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124063 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:14f097fd8f89|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124066 :::80||taskhelper] task Xapi#getResource /rrd_updates R:96c553abff8b forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124066 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:96c553abff8b|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124069 :::80||taskhelper] task Xapi#getResource /rrd_updates R:d604b8b2b8fc forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [ info||124067 :::80||taskhelper] task Xapi#getResource /rrd_updates R:761a0f164b4c forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124069 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:d604b8b2b8fc|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [debug||124067 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:761a0f164b4c|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded
Mar 26 15:38:00 xcp-angelina xapi: [ info||124072 :::80||taskhelper] task Xapi#getResource /rrd_updates R:1e5f14157edd forwarded (trackid=b5f6258df8b1ce99f2e2e1e340704adf)
Mar 26 15:38:00 xcp-angelina xapi: [debug||124072 HTTPS "XO-IPv4-address"->:::80|Xapi#getResource /rrd_updates R:1e5f14157edd|xapi_services] hand_over_connection GET /rrd_updates to /var/lib/xcp/xcp-rrdd.forwarded

And if I track one of the stuck task by its UUID, there's only one occurrence in the logs:

Mar 26 12:37:00 xcp-angelina xapi: [ info||55610 HTTPS "XO-IPv4-address"->:::80|task.create D:a75213c5e38a|taskhelper] task Xapi#getResource /rrd_updates R:24f7cb51c61e (uuid:02585f09-7fae-fac6-8e9c-d261ba19587d) created (trackid=b5f6258df8b1ce99f2e2e1e340704adf) by task D:a75213c5e38a

I can share more of the logs if needed.

olivierlambert

I wonder if XO can reach that stats URL or if there's something blocking it. In this very case, the usual suspect is a network issue between XO and the host, but if it's in the same network with nothing in the middle, IDK. Have you changed the MTU?

Mathieu

@olivierlambert
MTU is standard @1500.
There's been no change in the network.
That issue is only there with commit newer than 0ccfd4b.

olivierlambert

You mean you don't have this issue since which commit exactly?

Mathieu

@olivierlambert
I mean if use commit 0ccfd4b, there is no issue, if I use a more recent one, like the latest 583c7, the issue is appearing.

olivierlambert

Can you find the exact one causing the issue? That might be very helpful

14wkinnersley

@olivierlambert
I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build.