Endless Xapi#getResource /rrd_updates in tasks list
-
@olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.
-
Just wanted to provide some additional details/screenshots to help in anyway possible.
My configuration has the master host at 192.168.1.4
I have 3 hosts in the cluster.
XO is on a Ubuntu VM at 192.168.1.15
Firewall at my router is allowing all traffic in that same VLAN.
Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.Task List:
Host - Master:
Commit:
-
Can you try to destroy your XO folder, re-pull, rebuid to be sure there's no leftovers?
-
@olivierlambert Will try it now and report back
-
@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building. -
Okay thanks, we'll try to see if we can reproduce it here.
-
@olivierlambert
I can also confirm that the issue is still there on my host with XO updated to master with latest commit a548a. -
It is gone with a548a after I've restarted XO and toolstack.
-
That's confusing. @Mathieu do you confirm?
-
@olivierlambert
Yes, confusing, indeed
I just now restarted XO and the toolstack one more time to be sure. Yesterday, it was OK at the beginning but the issue reappeared after a few hours.
I'll let you know ASAP. -
Thanks!
-
@olivierlambert now, about 10 hours after restart of XO and Toolstack I have two task too.
-
Thanks, might be useful for @julien-f
-
@olivierlambert
Same on my host, the first stuck task appeared 5 hours after toolstack and XO reboot. -
8b7e1 still got the tasks
-
-
@uwood 0794a still some tasks
-
That's tricky, @julien-f couldn't reproduce it in our lab. So we'll need more details on your setup guys.
-
@olivierlambert
I have one pool, and three Nodes in my pool.
Node 1 – Dell R420, 2x 10gb NICs (Master) – 192.168.1.4
Node 2 – Dell R620, 2x 10gb NICs – 192.168.1.7
Node 3 – Dell R420, 2x 10gb NICs – 192.168.1.3
XO VM – Ubuntu 22.04.4 LTS – 192.168.1.15Each Node has a Bond of NIC 2 + 3. (Node 2 with the R620 has the mac addresses re-assigned to work correctly)
Above each node, network wise, is 2x switches (2x Unifi USW-Agg) and I use the Unifi Dream Machine Pro as my router. I am able to ping the other Nodes from each Node.
Local DNS utilizes Technitium DNS (Primary & Secondary), as a recursive DNS.
My SR’s are two iSCSI datastores that run on a separate server running TrueNAS Scale on a Dell R320.
Within my pools, I run about 25x virtual machines. I run nightly backups for ~5 VM’s, and weekly backup’s for all VM’s. Backups have a remote NFS storage location hosted on a separate server. I have 3 VM’s that run on separate network vlans than the rest, and those networks are setup under the pool, and upstream on the router.
Plugins, I have the following enabled
• Backup-reports
• Load-balancer (performance mode)
• Perf-alert
• Transport-email
• Usage-reportFrom my testing, this was introduced with commit 6c16055 - Mar 15. I have since rolled back to c6451cf and have stayed on this commit for the past several days.
-
Thanks for the details @14wkinnersley !
