@olivierlambert No never changed it. From what i understand from Jon it's default this where his words "and all these SRs have auto-scan enabled. This means every 30 seconds, the pool master will scan the entirety of every SR". We changed this and the problem is gone.

Support found that automatic refresh of our sr every 30 seconds delayed this. It seems we had this for a longer time but now it’s more aggressive. Disabled this as this is not required. This resolves our issue here.

Hi All,

are there other users running 8.3 combined with CBT backups? we keep running into vdi's that stay connected to the control domain, we are investigating these with Vates as it looks like the problem is within xapi of tapdisk it would be helpfull to understand if there are others running into the same issue.

As far as i understand from vates it looks like the problem we face is somewhere within Xapi but we need more logging to understand it better. If anyone has this running with the same issues or without please let us know so we can compare.

Cheers!

Hi All,

First of all best wished to you all for 2025! I have just deployed the latest build to do some testing on the one remaining issue we have with cbt backups, we were still facing full backups on some vms, this is expected to happen because cbt is not activated fast enough on some vdi’s, i will update this post once it completed some test runs to let u know if this build resolves it (there is a fix inside this build that should fix it).

Robin

@andyh no cbt should be disabled, u can’t migrate an cbt enabled vdi.

You need to remove all snapshots before migration and disable cbt. Storage migration is not supported when cbt is invalid. I believe xoa should do this automatically however.

@florent ok thanks, but how does this work for a job with multiple retentions? if i have a job with 3 schedules and retentions set for this job, how does the sequence handle this, in other words as the retention is set on the schedule level and i disable 1 of 3 schedules, how does the sequence know what retention it should keep?

@Andrew we see the same behavior here, no strange backup issues so far!

Hi all, i can confirm the vdi_in_use error is resolved by https://github.com/vatesfr/xen-orchestra/pull/7960 we no longer see issues there.

Only remaining issue we see is the “ can’t create a stream from a metadata vdi fall back to base”

fbeauchamp opened this pull request in vatesfr/xen-orchestra

closed feat(backups/CBT): retry data_destroy when error is VDI_IN USE #7960

@olivierlambert i have changed it to V3 and it feels like it is performing normally now. i need to do some more testing but it seems the synology does handle the read/writes better on V3 then V4.

@olivierlambert No never changed it. From what i understand from Jon it's default this where his words "and all these SRs have auto-scan enabled. This means every 30 seconds, the pool master will scan the entirety of every SR". We changed this and the problem is gone.

Support found that automatic refresh of our sr every 30 seconds delayed this. It seems we had this for a longer time but now it’s more aggressive. Disabled this as this is not required. This resolves our issue here.

@florent so not an option solving this inside xoa? Could be usefull for the short term.

Anyone else experiencing this issue?
https://github.com/vatesfr/xen-orchestra/issues/8713

it's a long time bug that i believe is pretty easy to fix and get CBT backups to get more robust. Would be great if this can be implemented!

rtjdamen created this issue in vatesfr/xen-orchestra

open CBT backup fails repeatedly due to leftover invalid CBT snapshots #8713

@olivierlambert ok, maybe u experience some differences as well, we were on 8.3 since February and did patch last friday, since the patching is see a decrease.

What i believe is happening is that GC processes are blocking other storage operations. So only if a coalece is done on an sr i see multiple actions like destroy, enable cbt and change block calculations being processed. As far as i know this was not the case before, they also could take some longer but it was not related to (or i have never noticed it).

Maybe we can confirm if this behavior is by design or not?

Hi all,

We recently upgraded our production pools to the latest XCP-ng 8.3 release. After some struggles during the upgrade (mostly around the pool master), everything seems to be running fine now in general.

However, since upgrading, we’re seeing much longer durations for certain XAPI-related tasks, especially:

VDI.enable_cbt

VDI.destroy

VDI.list_changed_blocks (during backups)

In some cases, these tasks take up to 25 minutes to complete on specific VMs. Meanwhile, similar operations on other VMs are done in just a few minutes. The behavior is inconsistent but reproducible.

We’ve checked:

Storage performance is normal (LVM over local SSD)

No I/O bottlenecks on the hosts

No VM performance impact during these tasks

It seems to affect CBT-enabled VMs more strongly, but we’re only seeing this behavior since the upgrade to 8.3 — especially after upgrading the pool master.

Has anyone else seen this since upgrading?
Is there a known issue with CBT or coalesce interaction in 8.3?
Would love to hear if others experience this or have suggestions for tuning.

@ThasianXi thx mate!

We patched all our hosts with the latest xcp-ng version, looking into live cpu usage of multiple pools we see this

we see this on multiple pools with different setups. not shure if it's a glitch in the software showing incorrect information or that we are facing a real problem in the latest version.

Anyone else seeing this issue?

@flakpyro i understand they did, but not shure if it is allready fixed or not. it had to do with the dedicated migration network being selected. maybe @olivierlambert is aware of the current status?

On vmware u would need als vcenter for this kind of features. And as u can easy deploy an empty xoa, why would this be an issue?