our solution to the problem
we want to share a post-incident analysis of a data-loss event on XCP-ng 8.3 LTS (shared block storage over FC, Veeam B&R 13.1 with CBT enabled on the pool) — consolidated from our own incident, topic 12402, the CBT feedback thread 9268, and the still-open blktap PR #17. The pieces fit together into one coherent failure chain, and we believe it may be worth a sticky/KB article.
A note on the trigger, to be fair and complete: in our case the interrupted jobs (step 2) were caused by expired/invalid worker tokens on the XCP-ng hosts, so the backups died mid-run with the locks in place. We consider that our availability problem. However, an interrupted backup must never be able to damage production VM data — cleaning up VDI locks on job abort is the hypervisor's job, and that is the part that turned a backup hiccup into guest data loss.
Failure chain (as we understand it)
Veeam backup job with CBT runs and sets locks on VDIs — paused: true + host_OpaqueRef:xxx: RW entries in the VDI sm_config (XAPI state.db).
The job is interrupted (timeout / crash / restart). The locks are never cleaned up → stale paused: true remains in sm_config. These entries are MRO, so xe vdi-param-remove can't clear them.
A later leaf coalesce / commit runs into a cbtlog disk: per PR #17, tapdisk_vbd_first_image returns the cbtlog disk on td_commit, and the cbtlog driver has no commit action → commit fails early. Result: broken VDI chains, CBT metadata VDIs without a vhd parent, hundreds of orphaned VDIs, and .cbtlog files hanging coalesces (as reported in thread 9268).
SR rescan believes a GC is already running and aborts; a host reboot was the only way to force the coalesce through (also reported in 9268).
Storage cleaning freezes the VM disk briefly, but the un-freeze fails on the stale lock → failed to unpause tapdisk ... VMs using this tapdisk have lost access to the corresponding disk(s).
The guest keeps writing on a frozen/lost disk → NTFS corruption inside the guest and, in our case, actual SQL Server data loss.
Step 5 matches exactly the theory Veeam R&D is currently investigating ("storage cleaning freezes VM disks briefly during backup and sometimes fails to un-freeze them"). The stale paused:true lock appears to be the missing "why" behind the failed unpause.
What helped us recover
Patching the stale lock out of XAPI state.db (stop xapi, backup state.db, remove paused + host_OpaqueRef entries from the affected VDI's sm_config, start xapi).
Then: reset CBT on the affected VDIs and trigger a full backup so CBT re-initializes cleanly — otherwise the next interrupted job re-creates the same situation.
For the coalesce backlog: with the affected VMs powered off and CBT disabled, snapshot-create-then-delete to kick the GC, watch SMlog, iterate. (Same recipe a user documented in 9268.)
