Endless Xapi#getResource /rrd_updates in tasks list

[olivierlambert]

I wonder if XO can reach that stats URL or if there's something blocking it. In this very case, the usual suspect is a network issue between XO and the host, but if it's in the same network with nothing in the middle, IDK. Have you changed the MTU?

[Mathieu]

@olivierlambert
MTU is standard @1500.
There's been no change in the network.
That issue is only there with commit newer than 0ccfd4b.

[olivierlambert]

You mean you don't have this issue since which commit exactly?

[Mathieu]

@olivierlambert
I mean if use commit 0ccfd4b, there is no issue, if I use a more recent one, like the latest 583c7, the issue is appearing.

[olivierlambert]

Can you find the exact one causing the issue? That might be very helpful

[14wkinnersley]

@olivierlambert
I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build.

[14wkinnersley]

@olivierlambert
Appears the issue started on commit 6c16055 for fix(xen-api): various undici fixes

c6451cf - Mar 14 - Working, no hanging logs
6c16055 - Mar 15 - Broken - Hanging logs for Xapi#getResource /rrd_update

[olivierlambert]

Ping @julien-f

[julien-f]

@14wkinnersley Commit 583c728 should have fixed the issue, please test with latest master.

[14wkinnersley]

@julien-f I updated to master, but still seeing hanging logs for Xapi#getResource /rrd_update

[olivierlambert]

Even after restarting the toolstack to get rid of the previous tasks?

[14wkinnersley]

@olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.

[14wkinnersley]

Just wanted to provide some additional details/screenshots to help in anyway possible.

My configuration has the master host at 192.168.1.4
I have 3 hosts in the cluster.
XO is on a Ubuntu VM at 192.168.1.15
Firewall at my router is allowing all traffic in that same VLAN.
Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.

Task List:

Host - Master:

Commit:

[olivierlambert]

Can you try to destroy your XO folder, re-pull, rebuid to be sure there's no leftovers?

[14wkinnersley]

@olivierlambert Will try it now and report back

[14wkinnersley]

@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building.

[olivierlambert]

Okay thanks, we'll try to see if we can reproduce it here.

[Mathieu]

@olivierlambert
I can also confirm that the issue is still there on my host with XO updated to master with latest commit a548a.

[uwood]

It is gone with a548a after I've restarted XO and toolstack.

[olivierlambert]

That's confusing. @Mathieu do you confirm?