@olivierlambert I disabled perf-alert and restarted toolstack and let it sit for a while, no hanging tasks.
Posts made by 14wkinnersley
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert I realized I was on the stable release... After updating to latest release I definately have hanging tasks
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert
I activated the trial and imported my configuration from xo, making sure the perf-alert plugin was enabled/configured. I have not received any hanging tasks in XOA. -
RE: Endless Xapi#getResource /rrd_updates in tasks list
@rmarion Im registered but dont have a license. Dont really want to use up the free trial, and it still only shows the two plugins. So maybe someone else can test.
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert Maybe I am doing something wrong but I went ahead and deployed xoa, and under plugins I only have two plugins and neither are for perf-alert. Ive never used XOA, only built XO, so its probably user error.
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert Unfortunately, I am still having this issue if I enable perf-alert. Im on the latest commit (cc049d3) and perf-alert (v0.3.7). I removed the plugin entirely from /packages/xo-server-perf-alert and rebuilt to test further (and restarted toolstack). Still having the tasks hang up
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert Updated to master (Commit: 79c9ef0) with perf alert and load balancer plugin's disabled. Cleared the toolstack for good measure, and I got no hanging tasks.
For some further testing, I flipped the load balancer plugin back on and waited about 5 hours and received no hanging tasks. I turned load balancer plugin back off (Just in case), cleared the toolstack, and turned the perf alert plugin on and received the first hanging task within about 10 minutes, and subsequently had more tasks hang every ~10-12 minutes.
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert Will do. Plugins are disabled and I'm updating back to master right now. Will report back.
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert
I have one pool, and three Nodes in my pool.
Node 1 – Dell R420, 2x 10gb NICs (Master) – 192.168.1.4
Node 2 – Dell R620, 2x 10gb NICs – 192.168.1.7
Node 3 – Dell R420, 2x 10gb NICs – 192.168.1.3
XO VM – Ubuntu 22.04.4 LTS – 192.168.1.15Each Node has a Bond of NIC 2 + 3. (Node 2 with the R620 has the mac addresses re-assigned to work correctly)
Above each node, network wise, is 2x switches (2x Unifi USW-Agg) and I use the Unifi Dream Machine Pro as my router. I am able to ping the other Nodes from each Node.
Local DNS utilizes Technitium DNS (Primary & Secondary), as a recursive DNS.
My SR’s are two iSCSI datastores that run on a separate server running TrueNAS Scale on a Dell R320.
Within my pools, I run about 25x virtual machines. I run nightly backups for ~5 VM’s, and weekly backup’s for all VM’s. Backups have a remote NFS storage location hosted on a separate server. I have 3 VM’s that run on separate network vlans than the rest, and those networks are setup under the pool, and upstream on the router.
Plugins, I have the following enabled
• Backup-reports
• Load-balancer (performance mode)
• Perf-alert
• Transport-email
• Usage-reportFrom my testing, this was introduced with commit 6c16055 - Mar 15. I have since rolled back to c6451cf and have stayed on this commit for the past several days.
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building. -
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert Will try it now and report back
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
Just wanted to provide some additional details/screenshots to help in anyway possible.
My configuration has the master host at 192.168.1.4
I have 3 hosts in the cluster.
XO is on a Ubuntu VM at 192.168.1.15
Firewall at my router is allowing all traffic in that same VLAN.
Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.Task List:
Host - Master:
Commit:
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@julien-f I updated to master, but still seeing hanging logs for Xapi#getResource /rrd_update
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert
Appears the issue started on commit 6c16055 for fix(xen-api): various undici fixesc6451cf - Mar 14 - Working, no hanging logs
6c16055 - Mar 15 - Broken - Hanging logs for Xapi#getResource /rrd_update -
RE: Endless Xapi#getResource /rrd_updates in tasks list
@olivierlambert
I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build. -
RE: Endless Xapi#getResource /rrd_updates in tasks list
I never installed any certificates on my instance. I can rollback unless anyone needs me to stay on the latest and for any testing. Up to about 1500 tasks in the past 20 hours since I rebooted all my hosts and xo last night.
-
RE: Endless Xapi#getResource /rrd_updates in tasks list
I am also having this same issue after updating all the patches and XO. XO from sources. Commit 214b8. Was there a solution to this? Logged in today, and had close to 2000 tasks.