Endless Xapi#getResource /rrd_updates in tasks list
-
Can you try to destroy your XO folder, re-pull, rebuid to be sure there's no leftovers?
-
@olivierlambert Will try it now and report back
-
@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building. -
Okay thanks, we'll try to see if we can reproduce it here.
-
@olivierlambert
I can also confirm that the issue is still there on my host with XO updated to master with latest commit a548a. -
It is gone with a548a after I've restarted XO and toolstack.
-
That's confusing. @Mathieu do you confirm?
-
@olivierlambert
Yes, confusing, indeed
I just now restarted XO and the toolstack one more time to be sure. Yesterday, it was OK at the beginning but the issue reappeared after a few hours.
I'll let you know ASAP. -
Thanks!
-
@olivierlambert now, about 10 hours after restart of XO and Toolstack I have two task too.
-
Thanks, might be useful for @julien-f
-
@olivierlambert
Same on my host, the first stuck task appeared 5 hours after toolstack and XO reboot. -
8b7e1 still got the tasks
-
-
@uwood 0794a still some tasks
-
That's tricky, @julien-f couldn't reproduce it in our lab. So we'll need more details on your setup guys.
-
@olivierlambert
I have one pool, and three Nodes in my pool.
Node 1 – Dell R420, 2x 10gb NICs (Master) – 192.168.1.4
Node 2 – Dell R620, 2x 10gb NICs – 192.168.1.7
Node 3 – Dell R420, 2x 10gb NICs – 192.168.1.3
XO VM – Ubuntu 22.04.4 LTS – 192.168.1.15Each Node has a Bond of NIC 2 + 3. (Node 2 with the R620 has the mac addresses re-assigned to work correctly)
Above each node, network wise, is 2x switches (2x Unifi USW-Agg) and I use the Unifi Dream Machine Pro as my router. I am able to ping the other Nodes from each Node.
Local DNS utilizes Technitium DNS (Primary & Secondary), as a recursive DNS.
My SR’s are two iSCSI datastores that run on a separate server running TrueNAS Scale on a Dell R320.
Within my pools, I run about 25x virtual machines. I run nightly backups for ~5 VM’s, and weekly backup’s for all VM’s. Backups have a remote NFS storage location hosted on a separate server. I have 3 VM’s that run on separate network vlans than the rest, and those networks are setup under the pool, and upstream on the router.
Plugins, I have the following enabled
• Backup-reports
• Load-balancer (performance mode)
• Perf-alert
• Transport-email
• Usage-reportFrom my testing, this was introduced with commit 6c16055 - Mar 15. I have since rolled back to c6451cf and have stayed on this commit for the past several days.
-
Thanks for the details @14wkinnersley !
-
@14wkinnersley can you disable perf alert and load balancer plugin and see if it still happening?
-
@olivierlambert
Simpler setup on my side:Pool 1 - 1 x ASRock Rack 1U4LW-X570/2L2T RPSU with about 10 VMs
Pool 2 - 1 x HP DL360 Gen9 with only the XO VM (Debian 11).Each host has 2 x 10 Gb NIC in use (one for VMs/MGMT, the other one for NFS storage).
The storage is a NFS share on a QNAP NAS (except for the XO VM which is on the local storage of the DL360 host).
The 2 hosts and the NFS storage are connected on the same10Gbit Ubiquiti EdgeSwitch.
Same plugins as @14wkinnersley + audit and sdn-controller.
The issue with stuck task is only appearing on the pool 1, not on the pool 2 with only the XO VM.
