@stormi Follow up on the steps I took to possibly help others that may have similar scenarios.

1. Migrated all VM's off of my Master (Node1) and rebooted, upgraded to 8.3 from ISO still on BIOS mode.
2. Once Node1 was back online, I went ahead and upgraded Node2, and Node3 one at a time. I would just move my current VM's around to the host I wasn't upgrading. Finally had all 3 hosts on 8.3.
3. Started the UEFI transition. I started with Node1 which was the master, but I changed the master to be on Node3. I went ahead and migrated all the VM's off the Node1, shutdown, pull the HDD (So I would have a backup) and put a new HDD in. Changed my boot mode to UEFI and did a fresh install. I removed Node1 from the cluster in XO. Once back online, I went ahead and just added Node1 back into my cluster and let XO do its magic of bringing everything back online.

Only issue I ran into was a POOL_JOINING_HOST_TLS_VERIFICATION_MISMATCH when I attempted to add the Node1 to the cluster. I followed the steps listed here: https://xcp-ng.org/forum/post/63470

Running xe pool-enable-tls-verification and xe host-emergency-reenable-tls-verification on Node3 which was still the master and was then able to get the new server added to the pool.

4. Moved the master back to Node1 and started the process over again on Node2 and 3.

Needing some assistance on where to start with the upgrade. I want to upgrade my 3-node cluster. As of right now all 3 nodes are on 8.2.1 but are BIOS boot, and it appears you need UEFI for 8.3 RC1. What is the best way to go about upgrading to where I wont lose data/vm’s? Understand I will most likely need to do a fresh install on each host.. What’s recommended? Should I start with the node that holds the master first on doing the fresh install?

Spoke too soon. Still getting them... Just much slower and father apart.. about 5-10 a day

Just wanted to provide an update, it seems the issue is resolved for me in the latest round of commits.

@olivierlambert I disabled perf-alert and restarted toolstack and let it sit for a while, no hanging tasks.

@olivierlambert I realized I was on the stable release... After updating to latest release I definately have hanging tasks

@olivierlambert I activated the trial and imported my configuration from xo, making sure the perf-alert plugin was enabled/configured. I have not received any hanging tasks in XOA.

@rmarion Im registered but dont have a license. Dont really want to use up the free trial, and it still only shows the two plugins. So maybe someone else can test.

@olivierlambert Maybe I am doing something wrong but I went ahead and deployed xoa, and under plugins I only have two plugins and neither are for perf-alert. Ive never used XOA, only built XO, so its probably user error.

@olivierlambert Unfortunately, I am still having this issue if I enable perf-alert. Im on the latest commit (cc049d3) and perf-alert (v0.3.7). I removed the plugin entirely from /packages/xo-server-perf-alert and rebuilt to test further (and restarted toolstack). Still having the tasks hang up

@olivierlambert Updated to master (Commit: 79c9ef0) with perf alert and load balancer plugin's disabled. Cleared the toolstack for good measure, and I got no hanging tasks.

For some further testing, I flipped the load balancer plugin back on and waited about 5 hours and received no hanging tasks. I turned load balancer plugin back off (Just in case), cleared the toolstack, and turned the perf alert plugin on and received the first hanging task within about 10 minutes, and subsequently had more tasks hang every ~10-12 minutes.

@olivierlambert Will do. Plugins are disabled and I'm updating back to master right now. Will report back.

@olivierlambert
I have one pool, and three Nodes in my pool.
Node 1 – Dell R420, 2x 10gb NICs (Master) – 192.168.1.4
Node 2 – Dell R620, 2x 10gb NICs – 192.168.1.7
Node 3 – Dell R420, 2x 10gb NICs – 192.168.1.3
XO VM – Ubuntu 22.04.4 LTS – 192.168.1.15

Each Node has a Bond of NIC 2 + 3. (Node 2 with the R620 has the mac addresses re-assigned to work correctly)

Above each node, network wise, is 2x switches (2x Unifi USW-Agg) and I use the Unifi Dream Machine Pro as my router. I am able to ping the other Nodes from each Node.

Local DNS utilizes Technitium DNS (Primary & Secondary), as a recursive DNS.

My SR’s are two iSCSI datastores that run on a separate server running TrueNAS Scale on a Dell R320.

Within my pools, I run about 25x virtual machines. I run nightly backups for ~5 VM’s, and weekly backup’s for all VM’s. Backups have a remote NFS storage location hosted on a separate server. I have 3 VM’s that run on separate network vlans than the rest, and those networks are setup under the pool, and upstream on the router.

Plugins, I have the following enabled
• Backup-reports
• Load-balancer (performance mode)
• Perf-alert
• Transport-email
• Usage-report

From my testing, this was introduced with commit 6c16055 - Mar 15. I have since rolled back to c6451cf and have stayed on this commit for the past several days.

@olivierlambert
No luck. Destroyed xo folder, rebooted, rebuilt xo from sources, restarted toolstack. Still having the logs building.

@olivierlambert Will try it now and report back

Just wanted to provide some additional details/screenshots to help in anyway possible.

My configuration has the master host at 192.168.1.4
I have 3 hosts in the cluster.
XO is on a Ubuntu VM at 192.168.1.15
Firewall at my router is allowing all traffic in that same VLAN.
Seems to gain 1-2 hanging logs every 1-4 minutes. I can ping the hosts from XO VM using either IP or DNS name.

Task List:

Host - Master:

Commit:

@olivierlambert Correct, toolstack has been restarted on the master. I have not tried any reboots yet of my hosts in the cluster, or of XO.

@julien-f I updated to master, but still seeing hanging logs for Xapi#getResource /rrd_update

@olivierlambert
Appears the issue started on commit 6c16055 for fix(xen-api): various undici fixes

c6451cf - Mar 14 - Working, no hanging logs
6c16055 - Mar 15 - Broken - Hanging logs for Xapi#getResource /rrd_update

@olivierlambert
I am still testing but so far have been able to determine it started between the 0ccfd4b - Mar 14 commit (@Mathieu stated was working) and the 8e5d94d - Mar 15 commit (Xapi#getResource /rrd_update errors). I am testing each commit (six commits) between those individually and will report back, it's just taking some time to load each one and then wait for the logs to build.