Endless Xapi#getResource /rrd_updates in tasks list

[14wkinnersley]

@olivierlambert
I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build.

[14wkinnersley]

@olivierlambert
Appears the issue started on commit 6c16055 for fix(xen-api): various undici fixes

c6451cf - Mar 14 - Working, no hanging logs
6c16055 - Mar 15 - Broken - Hanging logs for Xapi#getResource /rrd_update

[olivierlambert]

Ping @julien-f

[julien-f]

@14wkinnersley Commit 583c728 should have fixed the issue, please test with latest master.

[14wkinnersley]

@julien-f I updated to master, but still seeing hanging logs for Xapi#getResource /rrd_update

[olivierlambert]

Even after restarting the toolstack to get rid of the previous tasks?

[14wkinnersley]

@olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.

[14wkinnersley]

Just wanted to provide some additional details/screenshots to help in anyway possible.

My configuration has the master host at 192.168.1.4
I have 3 hosts in the cluster.
XO is on a Ubuntu VM at 192.168.1.15
Firewall at my router is allowing all traffic in that same VLAN.
Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.

Task List:

Host - Master:

Commit:

[olivierlambert]

Can you try to destroy your XO folder, re-pull, rebuid to be sure there's no leftovers?

[14wkinnersley]

@olivierlambert Will try it now and report back

[14wkinnersley]

@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building.

[olivierlambert]

Okay thanks, we'll try to see if we can reproduce it here.

[Mathieu]

@olivierlambert
I can also confirm that the issue is still there on my host with XO updated to master with latest commit a548a.

[uwood]

It is gone with a548a after I've restarted XO and toolstack.

[olivierlambert]

That's confusing. @Mathieu do you confirm?

[Mathieu]

@olivierlambert
Yes, confusing, indeed
I just now restarted XO and the toolstack one more time to be sure. Yesterday, it was OK at the beginning but the issue reappeared after a few hours.
I'll let you know ASAP.

[olivierlambert]

Thanks!

[uwood]

@olivierlambert now, about 10 hours after restart of XO and Toolstack I have two task too.

[olivierlambert]

Thanks, might be useful for @julien-f

[Mathieu]

@olivierlambert
Same on my host, the first stuck task appeared 5 hours after toolstack and XO reboot.