Veeam CBT on XCP-ng: Broken VHD chain causes tapdisk failure and filesystem corruption
Hello everyone,
We are experiencing a serious issue with Veeam CBT (Changed Block Tracking) on XCP-ng that has led to filesystem corruption on our production VMs. I want to share our analysis here in the hope that others have had similar experiences or that Veeam developers can provide some insights.
WHAT HAPPENED?
Two of our production VMs (a SQL Server and a File Server) suffered corrupted filesystems (NTFS) after a regular backup job. The VMs had to be stopped and restored from backup. Both VMs had CBT (Changed Block Tracking) enabled.
THE SYMPTOMS
When you look at the VM configuration on XCP-ng, you find so-called "CBT metadata" VDIs for each disk with CBT enabled. These are thin snapshot VDIs that Veeam creates to track changed blocks for incremental backups. So far, so normal.
The problem: These CBT metadata VDIs have a VHD parent reference that points to a VDI UUID which NO LONGER EXISTS in the XCP-ng database. The VHD chain is broken:
Base-VDI -> CBT metadata VDI (Snapshot) -> Parent = DOES NOT EXIST
When XCP-ng tries to unpause the tapdisk after the backup snapshot operation, it cannot resolve the VHD chain. This results in the error:
".SR-Name: failed to unpause tapdisk"
"Failed to unpause tapdisk for VDI <UUID>, VMs using this tapdisk have lost access to the corresponding disk(s)"
The VM then loses disk access -> I/O errors -> NTFS becomes corrupt.
Additionally, both VMs have a VBD (Virtual Block Device) that points to a VDI which shows "<not in database>" — also orphaned.
THE TRIGGER
We were able to trace the trigger back:
-
On August 2nd, Veeam installed a package update on our Windows backup repository. This triggered an automatic server restart.
-
During the restart, the Veeam vPower NFS Service (VeeamNFSSvc) failed to start — Error 1920. This is a known MSI bug: the installer tries to start the service before the installation is fully complete. After a manual restart, the service ran again.
-
But during the ~40 minutes without vPower NFS, Veeam's "Configuration Resynchronize" failed. It ran for 8 hours (19:59 to 04:03) and ended with "Failed".
-
As a result, the Updater tokens of all 3 Veeam Worker VMs expired. The workers could no longer authenticate with the VBR server — every refresh attempt was rejected with 401 "Your session has expired". This persisted for 3+ days.
-
Despite the expired worker tokens, Veeam executed the backup job on August 5th and reported it as "Success". The job created CBT snapshots, but the consolidation (Transform) ran with broken worker infrastructure.
THE RESULT
The CBT metadata VDIs remain as snapshots, but their VHD parent references point to nothing. The tapdisk crashes, the VM loses disk access, the filesystem becomes corrupt.
For VM-A (SQL Server), both disks (60GB + 80GB) have broken CBT snapshots. virtual-size=0, physical-utilisation=0.
For VM-B (File Server), the 80GB disk has a broken CBT snapshot. Interestingly: virtual-size=85899345920 (80GB, same as the base disk) and physical-utilisation=8388608 (8MB). This is different behavior from VM-A — possibly a different code path in the CBT lifecycle.
THE BACKUP REPORTS "SUCCESS"
The frustrating part: The Veeam session log shows NO errors. The backup is reported as "Success" with:
- isTaskSuccess=true
- isJobSuccess=true
- isTaskWarning=false
The DeletePointTransformSpec (OperationType=3, EnableIntegrityChecks=false) cleanly progresses through states 6 -> 5 -> 0 (Complete). Only afterwards, when XCP-ng tries to unpause the tapdisk, it fails — but Veeam doesn't notice.
RESTORE ALSO FAILS
The attempt to restore VM-B from backup also failed:
- "Unable to write data to the transport connection: Broken pipe"
- "unable to restore the disk in the HotAdd mode. Restoring disk using the Import functionality"
SCALE
We found over 200 CBT metadata VDIs across the entire XCP-ng pool, from dozens of different backup jobs. We don't yet know how many of them have broken parent references, but it could affect significantly more VMs than just the 2 that are already corrupt.
OUR QUESTIONS
-
Has anyone else used CBT on XCP-ng with Veeam v13.1 and observed similar issues?
-
Is it known that the DeletePointTransformSpec can destroy VHD parent references?
-
Why does Veeam report "Success" when the worker tokens are expired? Shouldn't the job at least give a warning?
-
Is the Error 1920 (vPower NFS during MSI install) a known issue? Is there a fix for it?
-
Has anyone found a way to safely verify whether CBT metadata VDIs have valid parent references?
-
Is there a best practice for periodically resetting CBT (Active Full) without the risk of a broken VHD chain?
OUR WORKAROUND
- Deleted affected CBT metadata VDIs (xe vdi-destroy)
- Restored VMs from backup
- Disabled CBT on all VMs until the issue is understood
- Restarted Veeam Backup Service (worker tokens reset)
- Scheduled Active Full instead of incremental
Thank you for any feedback and insights.
WARNING!
This issue has already completely destroyed 4 VMs.
Additional VMs are currently being checked.
