Endless Xapi#getResource /rrd_updates in tasks list

[14wkinnersley]

@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building.

[olivierlambert]

Okay thanks, we'll try to see if we can reproduce it here.

[Mathieu]

@olivierlambert
I can also confirm that the issue is still there on my host with XO updated to master with latest commit a548a.

[uwood]

It is gone with a548a after I've restarted XO and toolstack.

[olivierlambert]

That's confusing. @Mathieu do you confirm?

[Mathieu]

@olivierlambert
Yes, confusing, indeed
I just now restarted XO and the toolstack one more time to be sure. Yesterday, it was OK at the beginning but the issue reappeared after a few hours.
I'll let you know ASAP.

[olivierlambert]

Thanks!

[uwood]

@olivierlambert now, about 10 hours after restart of XO and Toolstack I have two task too.

[olivierlambert]

Thanks, might be useful for @julien-f

[Mathieu]

@olivierlambert
Same on my host, the first stuck task appeared 5 hours after toolstack and XO reboot.

[uwood]

8b7e1 still got the tasks

[uwood]

[uwood]

@uwood 0794a still some tasks

[olivierlambert]

That's tricky, @julien-f couldn't reproduce it in our lab. So we'll need more details on your setup guys.

[14wkinnersley]

@olivierlambert
I have one pool, and three Nodes in my pool.
Node 1 – Dell R420, 2x 10gb NICs (Master) – 192.168.1.4
Node 2 – Dell R620, 2x 10gb NICs – 192.168.1.7
Node 3 – Dell R420, 2x 10gb NICs – 192.168.1.3
XO VM – Ubuntu 22.04.4 LTS – 192.168.1.15

Each Node has a Bond of NIC 2 + 3. (Node 2 with the R620 has the mac addresses re-assigned to work correctly)

Above each node, network wise, is 2x switches (2x Unifi USW-Agg) and I use the Unifi Dream Machine Pro as my router. I am able to ping the other Nodes from each Node.

Local DNS utilizes Technitium DNS (Primary & Secondary), as a recursive DNS.

My SR’s are two iSCSI datastores that run on a separate server running TrueNAS Scale on a Dell R320.

Within my pools, I run about 25x virtual machines. I run nightly backups for ~5 VM’s, and weekly backup’s for all VM’s. Backups have a remote NFS storage location hosted on a separate server. I have 3 VM’s that run on separate network vlans than the rest, and those networks are setup under the pool, and upstream on the router.

Plugins, I have the following enabled
• Backup-reports
• Load-balancer (performance mode)
• Perf-alert
• Transport-email
• Usage-report

From my testing, this was introduced with commit 6c16055 - Mar 15. I have since rolled back to c6451cf and have stayed on this commit for the past several days.

[olivierlambert]

Thanks for the details @14wkinnersley !

[olivierlambert]

@14wkinnersley can you disable perf alert and load balancer plugin and see if it still happening?

[Mathieu]

@olivierlambert
Simpler setup on my side:

Pool 1 - 1 x ASRock Rack 1U4LW-X570/2L2T RPSU with about 10 VMs
Pool 2 - 1 x HP DL360 Gen9 with only the XO VM (Debian 11).

Each host has 2 x 10 Gb NIC in use (one for VMs/MGMT, the other one for NFS storage).

The storage is a NFS share on a QNAP NAS (except for the XO VM which is on the local storage of the DL360 host).

The 2 hosts and the NFS storage are connected on the same10Gbit Ubiquiti EdgeSwitch.

Same plugins as @14wkinnersley + audit and sdn-controller.

The issue with stuck task is only appearing on the pool 1, not on the pool 2 with only the XO VM.

[14wkinnersley]

@olivierlambert Will do. Plugins are disabled and I'm updating back to master right now. Will report back.

[olivierlambert]

Thank you very much, both of you