Endless Xapi#getResource /rrd_updates in tasks list

[14wkinnersley]

@olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.

[14wkinnersley]

Just wanted to provide some additional details/screenshots to help in anyway possible.

My configuration has the master host at 192.168.1.4
I have 3 hosts in the cluster.
XO is on a Ubuntu VM at 192.168.1.15
Firewall at my router is allowing all traffic in that same VLAN.
Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.

Task List:

Host - Master:

Commit:

[olivierlambert]

Can you try to destroy your XO folder, re-pull, rebuid to be sure there's no leftovers?

[14wkinnersley]

@olivierlambert Will try it now and report back

[14wkinnersley]

@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building.

[olivierlambert]

Okay thanks, we'll try to see if we can reproduce it here.

[Mathieu]

@olivierlambert
I can also confirm that the issue is still there on my host with XO updated to master with latest commit a548a.

[uwood]

It is gone with a548a after I've restarted XO and toolstack.

[olivierlambert]

That's confusing. @Mathieu do you confirm?

[Mathieu]

@olivierlambert
Yes, confusing, indeed
I just now restarted XO and the toolstack one more time to be sure. Yesterday, it was OK at the beginning but the issue reappeared after a few hours.
I'll let you know ASAP.

[olivierlambert]

Thanks!

[uwood]

@olivierlambert now, about 10 hours after restart of XO and Toolstack I have two task too.

[olivierlambert]

Thanks, might be useful for @julien-f

[Mathieu]

@olivierlambert
Same on my host, the first stuck task appeared 5 hours after toolstack and XO reboot.

[uwood]

8b7e1 still got the tasks

[uwood]

[uwood]

@uwood 0794a still some tasks

[olivierlambert]

That's tricky, @julien-f couldn't reproduce it in our lab. So we'll need more details on your setup guys.

[14wkinnersley]

@olivierlambert
I have one pool, and three Nodes in my pool.
Node 1 – Dell R420, 2x 10gb NICs (Master) – 192.168.1.4
Node 2 – Dell R620, 2x 10gb NICs – 192.168.1.7
Node 3 – Dell R420, 2x 10gb NICs – 192.168.1.3
XO VM – Ubuntu 22.04.4 LTS – 192.168.1.15

Each Node has a Bond of NIC 2 + 3. (Node 2 with the R620 has the mac addresses re-assigned to work correctly)

Above each node, network wise, is 2x switches (2x Unifi USW-Agg) and I use the Unifi Dream Machine Pro as my router. I am able to ping the other Nodes from each Node.

Local DNS utilizes Technitium DNS (Primary & Secondary), as a recursive DNS.

My SR’s are two iSCSI datastores that run on a separate server running TrueNAS Scale on a Dell R320.

Within my pools, I run about 25x virtual machines. I run nightly backups for ~5 VM’s, and weekly backup’s for all VM’s. Backups have a remote NFS storage location hosted on a separate server. I have 3 VM’s that run on separate network vlans than the rest, and those networks are setup under the pool, and upstream on the router.

Plugins, I have the following enabled
• Backup-reports
• Load-balancer (performance mode)
• Perf-alert
• Transport-email
• Usage-report

From my testing, this was introduced with commit 6c16055 - Mar 15. I have since rolled back to c6451cf and have stayed on this commit for the past several days.

[olivierlambert]

Thanks for the details @14wkinnersley !