CBT: the thread to centralize your feedback
-
Hi all,
We recently upgraded our production pools to the latest XCP-ng 8.3 release. After some struggles during the upgrade (mostly around the pool master), everything seems to be running fine now in general.
However, since upgrading, we’re seeing much longer durations for certain XAPI-related tasks, especially:
VDI.enable_cbt
VDI.destroy
VDI.list_changed_blocks (during backups)
In some cases, these tasks take up to 25 minutes to complete on specific VMs. Meanwhile, similar operations on other VMs are done in just a few minutes. The behavior is inconsistent but reproducible.
We’ve checked:
Storage performance is normal (LVM over local SSD)
No I/O bottlenecks on the hosts
No VM performance impact during these tasks
It seems to affect CBT-enabled VMs more strongly, but we’re only seeing this behavior since the upgrade to 8.3 — especially after upgrading the pool master.
Has anyone else seen this since upgrading?
Is there a known issue with CBT or coalesce interaction in 8.3?
Would love to hear if others experience this or have suggestions for tuning. -
I'm not aware of such issues (at least now)
(doesn't mean it doesn't exist more broadly but it's the first report). We'll upgrade our own production to 8.3 relatively soon, so it will be the opportunity to also check internally. -
@olivierlambert ok, maybe u experience some differences as well, we were on 8.3 since February and did patch last friday, since the patching is see a decrease.
What i believe is happening is that GC processes are blocking other storage operations. So only if a coalece is done on an sr i see multiple actions like destroy, enable cbt and change block calculations being processed. As far as i know this was not the case before, they also could take some longer but it was not related to (or i have never noticed it).
Maybe we can confirm if this behavior is by design or not?
-
I'm not sure there's a change related to that but I can ask @Team-Storage
-
Anyone else experiencing this issue?
https://github.com/vatesfr/xen-orchestra/issues/8713it's a long time bug that i believe is pretty easy to fix and get CBT backups to get more robust. Would be great if this can be implemented!
-
Pinging @florent about this
-
@olivierlambert yes the fix is in the pipeline (on xapi side )
it won't migrate a snapshot with cbt enabled, and won't allow to disable cbt on a snapshot -
Thanks!
-
@florent so not an option solving this inside xoa? Could be usefull for the short term.
-
Support found that automatic refresh of our sr every 30 seconds delayed this. It seems we had this for a longer time but now it’s more aggressive. Disabled this as this is not required. This resolves our issue here.
-
Wow, rescan every 30sec? I thought the default value was 10 min or something
Do you remember setting it manually at some point? -
@olivierlambert No never changed it. From what i understand from Jon it's default this where his words "and all these SRs have auto-scan enabled. This means every 30 seconds, the pool master will scan the entirety of every SR". We changed this and the problem is gone.
-
Okay, I thought the autoscan was only for like 10 minutes or so, but hey I'm not deep down in the stack anymore
