<![CDATA[huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks]]>Hi, i've this situation in Task
6bd4ea87-1a46-48b4-a92f-0edba988c385-immagine.png
What's this task?
Now i've ~50 of this call..

]]>https://xcp-ng.org/forum/topic/9259/huge-number-of-api-call-sr-getallunhealthyvdichainslength-in-tasksRSS for NodeTue, 10 Mar 2026 07:14:26 GMTWed, 26 Jun 2024 07:39:56 GMT60<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Wed, 17 Jul 2024 09:28:12 GMT]]>Hello, can you please try the buffered-task-events branch and let me know if that solves the issue?

]]>https://xcp-ng.org/forum/post/80311https://xcp-ng.org/forum/post/80311Wed, 17 Jul 2024 09:28:12 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 05 Jul 2024 22:54:17 GMT]]>With today's update (feat: release 5.96.0, master 96b7673) things are working well again. I have not seen any Orphan VDI/snapshots or stuck Control Domain VDIs (but it may take time to test). Thanks to the Vates team!

]]>https://xcp-ng.org/forum/post/79817https://xcp-ng.org/forum/post/79817Fri, 05 Jul 2024 22:54:17 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 05 Jul 2024 20:24:48 GMT]]>@Andrew @julien-f Ran a full backup with commit 96b76 and it is working properly now. Started thinking that I was the only one 🙂 Thanks a lot for everyone's help!

]]>https://xcp-ng.org/forum/post/79806https://xcp-ng.org/forum/post/79806Fri, 05 Jul 2024 20:24:48 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 05 Jul 2024 09:13:51 GMT]]>@julien-f It's happening to me too... my [NOBAK] disk is being backed up now (commit 0e4a3) using the normal Backup (running a full).

]]>https://xcp-ng.org/forum/post/79757https://xcp-ng.org/forum/post/79757Fri, 05 Jul 2024 09:13:51 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Thu, 04 Jul 2024 13:42:58 GMT]]>@ryan-gravel I cannot reproduce, both Full Backup and Incremental Backup correctly ignore VDIs with [NOBAK] in their name label.

]]>https://xcp-ng.org/forum/post/79724https://xcp-ng.org/forum/post/79724Thu, 04 Jul 2024 13:42:58 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Mon, 01 Jul 2024 20:27:42 GMT]]>Ping @julien-f

]]>https://xcp-ng.org/forum/post/79547https://xcp-ng.org/forum/post/79547Mon, 01 Jul 2024 20:27:42 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Mon, 01 Jul 2024 20:33:07 GMT]]>@olivierlambert I believe so. It is the Disks tab of a VM. [NOBAK] was working but now the NAS is complaining about storage space. This VM has a disk on the NAS so it is backing it up twice. Added [NOSNAP] just as a test with the same result.
24516021-d80c-4418-a539-a89b7ca17797-image.png

It is running a full backup if that makes any difference.

]]>https://xcp-ng.org/forum/post/79533https://xcp-ng.org/forum/post/79533Mon, 01 Jul 2024 20:33:07 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Mon, 01 Jul 2024 18:03:34 GMT]]>On the VDI name, right?

]]>https://xcp-ng.org/forum/post/79531https://xcp-ng.org/forum/post/79531Mon, 01 Jul 2024 18:03:34 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Mon, 01 Jul 2024 18:00:36 GMT]]>Doesn't seem to be respecting [NOBAK] anymore for storage repos. Tried '[NOSNAP][NOBAK] StorageName' and it still grabs it.

]]>https://xcp-ng.org/forum/post/79530https://xcp-ng.org/forum/post/79530Mon, 01 Jul 2024 18:00:36 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Mon, 01 Jul 2024 09:07:53 GMT]]>Same problem here 300+ sessions and counting
07564582-c58a-490e-8921-5f5de0e98fa6-image.png
XAPI restart did not solve the issue...

]]>https://xcp-ng.org/forum/post/79497https://xcp-ng.org/forum/post/79497Mon, 01 Jul 2024 09:07:53 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Sat, 29 Jun 2024 09:52:46 GMT]]>@manilx Was fine for a short while only: https://xcp-ng.org/forum/topic/9275/starting-getting-again-retry-the-vm-backup-due-to-an-error-error-vdi-must-be-free-or-attached-to-exactly-one-vm

]]>https://xcp-ng.org/forum/post/79460https://xcp-ng.org/forum/post/79460Sat, 29 Jun 2024 09:52:46 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 28 Jun 2024 17:07:13 GMT]]>@flakpyro Just updated, run my delta backups and all is fine 🎂

]]>https://xcp-ng.org/forum/post/79421https://xcp-ng.org/forum/post/79421Fri, 28 Jun 2024 17:07:13 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 28 Jun 2024 15:51:44 GMT]]>@olivierlambert

After i seen the CBT stuff was reverted on github, i updated to the latest commit on my home server, (253aa) i can report my backups are now working as they should and coalesce runs without issues leaving a clean health check dashboard again. :). Glad to see this has been held back on XOA as well as i was planning to stay on 5.95.1 otherwise! Looking forward to CBT eventually all the same!

]]>https://xcp-ng.org/forum/post/79416https://xcp-ng.org/forum/post/79416Fri, 28 Jun 2024 15:51:44 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 28 Jun 2024 09:05:20 GMT]]>This thread will be used to centralize all backup issues related to CBT: https://xcp-ng.org/forum/topic/9268/cbt-the-thread-to-centralize-your-feedback

]]>https://xcp-ng.org/forum/post/79393https://xcp-ng.org/forum/post/79393Fri, 28 Jun 2024 09:05:20 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 28 Jun 2024 07:07:38 GMT]]>@olivierlambert I applaud this decision. Running stable for many many months suddenly to have all these issues, without really knowing what caused it and how to resolve has been a nightmare for me 😞

I'm all in for those new features (!) but if updating to new build to get fixes results in those issues is not good.

I've spent many hours lately, rebooted the hosts more times than the last 5 years to get backups/coalesce working again and still am not sure if everything is back to normal....

So, yes, this is a good decision.

]]>https://xcp-ng.org/forum/post/79382https://xcp-ng.org/forum/post/79382Fri, 28 Jun 2024 07:07:38 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 28 Jun 2024 06:40:57 GMT]]>FYI, CBT was reverted, we'll continue to fix bugs on it in a dedicated branch for people to test (with a community and Vates joint test campaign).

]]>https://xcp-ng.org/forum/post/79380https://xcp-ng.org/forum/post/79380Fri, 28 Jun 2024 06:40:57 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 28 Jun 2024 06:40:03 GMT]]>Hi,

XO tasks aren't related to XAPI tasks, it's normal to see those. They were hidden before, now it's helpful to see when we have so much calls 🙂

]]>https://xcp-ng.org/forum/post/79379https://xcp-ng.org/forum/post/79379Fri, 28 Jun 2024 06:40:03 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Fri, 28 Jun 2024 06:14:19 GMT]]>@olivierlambert I'm not sure that I am doing it correctly. XOA is up to date (Master, commit 6ee7a) and I went through and re scanned the storage repositories to get snapshots to coalesce. Backups work properly now.

I have 50 or so instances of the following in XO tasks

  • API call: sr.getAllUnhealthyVdiChainsLength
  • API call: sr.getVdiChainsInfo
  • API call: sr.reclaimSpace
  • API call: proxy.getAll
  • API call: session.getUser
  • API call: pool.listMissingPatches

I've rebooted the host and ran 'xe-toolstack-restart' but I'm getting the same results. I must be missing something.

Thanks for your help.

]]>https://xcp-ng.org/forum/post/79377https://xcp-ng.org/forum/post/79377Fri, 28 Jun 2024 06:14:19 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Thu, 27 Jun 2024 23:34:19 GMT]]>@manilx the .cbt logs are located on your SR (/run/sr-mount/UUID) i don't totally know if removing them is what fixed things for me but once backups started failing my goal was to "revert" everything back to a known good working state. Doing this, reverting XOA and restarting the tool stack on the host got things back to how they were.

I'm very much looking forward to CBT backups though as i've been spoiled by Veeam / ESX CBT backups for a lot of years!

]]>https://xcp-ng.org/forum/post/79375https://xcp-ng.org/forum/post/79375Thu, 27 Jun 2024 23:34:19 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Thu, 27 Jun 2024 23:18:36 GMT]]>@flakpyro said in huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks:

removing all .cbtlog files from my SR got backups back up and running

What did you do exctly to remove the .cbtlog files? Where are they located?

I find that after backups with the "broken" versions (also the latest with the 2 fixes) I get my coalescing stopped. And checking the logs (cat /var/log/SMlog | grep -i coalesce) I have entries like

['/usr/sbin/cbt-util', 'coalesce', '-p', '/var/run/sr-mount/ea3c92b7-0e82-5726-502b-482b40a8b097/0cc54304-4961-482f-a1b7-a8222dd143a1.cbtlog', '-c', '/var/run/sr-mount/ea3c92b7-0e82-5726-502b-482b40a8b097/fe11c7f1-7331-427e-8fa9-76412a2cbb75.cbtlog

And those correspond to the hanged coalesces...

When I reboot the host they disappear but on next delta backup I have the issue again.

]]>https://xcp-ng.org/forum/post/79374https://xcp-ng.org/forum/post/79374Thu, 27 Jun 2024 23:18:36 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Thu, 27 Jun 2024 17:03:26 GMT]]>@olivierlambert Still broken backups
ScreenShot 2024-06-27 at 18.02.12.png

]]>https://xcp-ng.org/forum/post/79334https://xcp-ng.org/forum/post/79334Thu, 27 Jun 2024 17:03:26 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Thu, 27 Jun 2024 16:27:58 GMT]]>Hi,

We added 2 fixes in latest commit on master, please restart your XAPI (to clear all tasks), and try again with your updated XO 🙂

]]>https://xcp-ng.org/forum/post/79325https://xcp-ng.org/forum/post/79325Thu, 27 Jun 2024 16:27:58 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Wed, 26 Jun 2024 19:34:18 GMT]]>@flakpyro Same here!

All related:

https://xcp-ng.org/forum/topic/9242/retry-the-vm-backup-due-to-an-error-error-handle_invalid-vdi/9?_=1719430401473

https://xcp-ng.org/forum/topic/9215/backups-started-failing-error-vdi-must-be-free-or-attached-to-exactly-one-vm

]]>https://xcp-ng.org/forum/post/79303https://xcp-ng.org/forum/post/79303Wed, 26 Jun 2024 19:34:18 GMT<![CDATA[Reply to huge number of api call "sr.getAllUnhealthyVdiChainsLength" in tasks on Wed, 26 Jun 2024 19:26:09 GMT]]>Updating to the latest commit (93833) and its still happening. Backups are also no longer functioning for me as seen below:

cccb07e9-7210-46ee-8de1-2dfb5fc30121-image.png

It seems editing the backup job and disabling NBD/CBT in advanced settings gets backups back up and running again however the health check shows all my VDIs as unhealthy and needing to coalesce.

As a test I tried creating an entirely new job with CBT enabled and while the job finishes now the task lingers and never goes away under tasks until you restart the toolstack of the host.

Edit: Rolling back to commit 3c13e and removing all .cbtlog files from my SR got backups back up and running and made the sr.getAllUnhealthyVdiChainsLength loop end. Im guessing theres been a lot of change to enable CBT and something is misbehaving somewhere.

]]>https://xcp-ng.org/forum/post/79302https://xcp-ng.org/forum/post/79302Wed, 26 Jun 2024 19:26:09 GMT