Veeam 13.1 Rocky9 Linux Appliance: Potential Data Loss with CBT and Workers with Expired Tokens

msupport

Veeam CBT on XCP-ng: Broken VHD chain causes tapdisk failure and filesystem corruption

Hello everyone,

We are experiencing a serious issue with Veeam CBT (Changed Block Tracking) on XCP-ng that has led to filesystem corruption on our production VMs. I want to share our analysis here in the hope that others have had similar experiences or that Veeam developers can provide some insights.

WHAT HAPPENED?

Two of our production VMs (a SQL Server and a File Server) suffered corrupted filesystems (NTFS) after a regular backup job. The VMs had to be stopped and restored from backup. Both VMs had CBT (Changed Block Tracking) enabled.

THE SYMPTOMS

When you look at the VM configuration on XCP-ng, you find so-called "CBT metadata" VDIs for each disk with CBT enabled. These are thin snapshot VDIs that Veeam creates to track changed blocks for incremental backups. So far, so normal.

The problem: These CBT metadata VDIs have a VHD parent reference that points to a VDI UUID which NO LONGER EXISTS in the XCP-ng database. The VHD chain is broken:

Base-VDI -> CBT metadata VDI (Snapshot) -> Parent = DOES NOT EXIST

When XCP-ng tries to unpause the tapdisk after the backup snapshot operation, it cannot resolve the VHD chain. This results in the error:

".SR-Name: failed to unpause tapdisk"
"Failed to unpause tapdisk for VDI <UUID>, VMs using this tapdisk have lost access to the corresponding disk(s)"

The VM then loses disk access -> I/O errors -> NTFS becomes corrupt.

Additionally, both VMs have a VBD (Virtual Block Device) that points to a VDI which shows "<not in database>" — also orphaned.

THE TRIGGER

We were able to trace the trigger back:

On August 2nd, Veeam installed a package update on our Windows backup repository. This triggered an automatic server restart.
During the restart, the Veeam vPower NFS Service (VeeamNFSSvc) failed to start — Error 1920. This is a known MSI bug: the installer tries to start the service before the installation is fully complete. After a manual restart, the service ran again.
But during the ~40 minutes without vPower NFS, Veeam's "Configuration Resynchronize" failed. It ran for 8 hours (19:59 to 04:03) and ended with "Failed".
As a result, the Updater tokens of all 3 Veeam Worker VMs expired. The workers could no longer authenticate with the VBR server — every refresh attempt was rejected with 401 "Your session has expired". This persisted for 3+ days.
Despite the expired worker tokens, Veeam executed the backup job on August 5th and reported it as "Success". The job created CBT snapshots, but the consolidation (Transform) ran with broken worker infrastructure.

THE RESULT

The CBT metadata VDIs remain as snapshots, but their VHD parent references point to nothing. The tapdisk crashes, the VM loses disk access, the filesystem becomes corrupt.

For VM-A (SQL Server), both disks (60GB + 80GB) have broken CBT snapshots. virtual-size=0, physical-utilisation=0.

For VM-B (File Server), the 80GB disk has a broken CBT snapshot. Interestingly: virtual-size=85899345920 (80GB, same as the base disk) and physical-utilisation=8388608 (8MB). This is different behavior from VM-A — possibly a different code path in the CBT lifecycle.

THE BACKUP REPORTS "SUCCESS"

The frustrating part: The Veeam session log shows NO errors. The backup is reported as "Success" with:

isTaskSuccess=true
isJobSuccess=true
isTaskWarning=false

The DeletePointTransformSpec (OperationType=3, EnableIntegrityChecks=false) cleanly progresses through states 6 -> 5 -> 0 (Complete). Only afterwards, when XCP-ng tries to unpause the tapdisk, it fails — but Veeam doesn't notice.

RESTORE ALSO FAILS

The attempt to restore VM-B from backup also failed:

"Unable to write data to the transport connection: Broken pipe"
"unable to restore the disk in the HotAdd mode. Restoring disk using the Import functionality"

SCALE

We found over 200 CBT metadata VDIs across the entire XCP-ng pool, from dozens of different backup jobs. We don't yet know how many of them have broken parent references, but it could affect significantly more VMs than just the 2 that are already corrupt.

OUR QUESTIONS

Has anyone else used CBT on XCP-ng with Veeam v13.1 and observed similar issues?
Is it known that the DeletePointTransformSpec can destroy VHD parent references?
Why does Veeam report "Success" when the worker tokens are expired? Shouldn't the job at least give a warning?
Is the Error 1920 (vPower NFS during MSI install) a known issue? Is there a fix for it?
Has anyone found a way to safely verify whether CBT metadata VDIs have valid parent references?
Is there a best practice for periodically resetting CBT (Active Full) without the risk of a broken VHD chain?

OUR WORKAROUND

Deleted affected CBT metadata VDIs (xe vdi-destroy)
Restored VMs from backup
Disabled CBT on all VMs until the issue is understood
Restarted Veeam Backup Service (worker tokens reset)
Scheduled Active Full instead of incremental

Thank you for any feedback and insights.

WARNING!
This issue has already completely destroyed 4 VMs.
Additional VMs are currently being checked.

acebmxer

@msupport said:

The CBT metadata VDIs remain as snapshots, but their VHD parent references point to nothing. The tapdisk crashes, the VM loses disk access, the filesystem becomes corrupt.

For VM-A (SQL Server), both disks (60GB + 80GB) have broken CBT snapshots. virtual-size=0, physical-utilisation=0.

For VM-B (File Server), the 80GB disk has a broken CBT snapshot. Interestingly: virtual-size=85899345920 (80GB, same as the base disk) and physical-utilisation=8388608 (8MB). This is different behavior from VM-A — possibly a different code path in the CBT lifecycle.

In my testing i did see similar issue

Edit - updated with actual error message from what I am getting. The first full backup were all successful. This first delta backup have 4 vms with this error. The first time I saw similar was only 2 vms.

8/5/2026 5:34:50 PM Warning : Failed to use CBT: [Task e75c4247-2ee1-7b51-088c-b970ace60f3d (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]

For me i took it as because i original had backups from veeam 12.x the first xcp-ng beta. Pointed new backups from 13.1 to older backups. I then purged all older backups and started with a fresh backup chain.

I ended up reverting to the windows base as i was not able to get the windows mount server setup. This was setting up from a windows client using the console to setup the repo.

Danp

@msupport Thanks for the detailed report. Please answer these questions to help us to better understand the source of the issue:

What OS were the affected VMs running?
Which guest tools were installed?
Were the affected VDIs in VHD or QCOW2 format?

msupport

@Danp

I have sent detailed log files from XCP-NG, Veeam, and Windows related to the incident to Veeam; the issue has already been escalated, and it appears to be a major problem.
We're already in touch with Olga from the Veeam team

Regards
Micha

Hello,
Thank you for contacting Veeam Technical Support! My name is Olga and I will be assisting you with the case from now on.
Thank you for sharing the logging and the analysis! I will now review the data provided and I will keep you posted on the findings and the next steps.
Have a nice day,
Olga Demidova,
Customer Support Engineer,
Veeam Technical Support.

Previous Communication

2026-08-06 14:11:22 UTC - system Generated Message
servicenow_email_separator_e8dd76512b2e031caf56f5f7f891bfa7_c79268ad93ea47508575fcffb903d6e9
---Do not reply below this line---

Hello,
thank you for uploading!
I provided the files to our R&D team. They will need some time to review everything and I will revert to you as soon as I have any updates.
Best regards,
Viktoria Nesmiyanova
Technical Customer Support - EMEA
Veeam Software

Previous Communication

Hello team,
hope my email finds you well!
I would like to inform you that we are actively researching the logs and details with R&D team. I will update you on the investigation status on a regular basis and will let you know as soon as I have any news.
Thank you!
Best regards,
Viktoria Nesmiyanova
Technical Customer Support - EMEA
Veeam Software

acebmxer

I have also opened a support ticket with Veeam. Well trying to... Issue with my account preventing me to and working to fix that issue.

Veeam forum did make this statment about CBT and snapshots -

There is one known snapshot issue that I'm guessing could possibly have this side-effect (noted in the release notes BTW). We take standard Xen snapshots for CBT but then immediately rename them with our own naming convention. We've noticed that sometimes this rename operation returns success status even if it fails. In this case I can see how this might throw this sort of error but again best to confirm w/support

Veeam support ticket - 08188639

Update from Veeam -

Thanks for the update

I'm currently checking with my team and I will update as soon as possible, what I noticed in the debug logs is the following API failing

\Backup\Plugins\XEN\Backup\Backup Job 1\Delilah ArcDC02 with debug enable

2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]:  <== Request localhost:19000, IsTaskFinished, body: {"taskId":"491371d9-4699-4087-b549-9862d2188177"}
2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]:  ==> Response localhost:19000, IsTaskFinished, success, duration: 0.1404 msec, body: false
2026-08-07 20:05:04.843 00019 INFO  | [XenRpcClient]: Start ListChangedBlocks
2026-08-07 20:05:04.843 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/Async.VDI.list_changed_blocks, body: ["OpaqueRef:b77ca409-d438-d97e-d8c3-f212647ad469","OpaqueRef:8c83f698-8836-3b4c-1506-8e97315aae1f"]
2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/Async.VDI.list_changed_blocks code: "OK", duration: 2 msec, body:{"opaque_ref":"OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"}
2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:04.847 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 2 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":["cancel"],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"1970-01-01T00:00:00Z","status":"pending","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":0.0,"type":"<none/>","result":"","error_info":[],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"()","opaque_ref":null}
2026-08-07 20:05:04.848 00019 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "pending" Progress: 0
2026-08-07 20:05:06.848 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 1 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":[],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"2026-08-08T00:05:05Z","status":"failure","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":1.0,"type":"<none/>","result":"","error_info":["SR_BACKEND_FAILURE_460","","Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]",""],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 150))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))","opaque_ref":null}
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "failure" Progress: 1
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.destroy, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:06.851 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.destroy code: "OK", duration: 0.735 msec, body:""
2026-08-07 20:05:06.851 00018 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. 
Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. 
 ---> Failed to calculate changed blocks for given VDIs.
   --- End of inner exception stack trace ---
   at Veeam.XenBackup.RestClient.XenRpcClient.GetTaskResult(XenRef`1 taskRef, CancellationToken cancellationToken)
   at Veeam.XenBackup.RestClient.XenRpcClient.<>c__DisplayClass94_0.<<ListChangedBlocksAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Veeam.Vbf.Common.Helper.Retry.RetryHelper.ExecuteActionAsync[T](Func`2 asyncAction, String description, ILogger logger, LogLevel logLevel, CancellationToken cancellationToken)
2026-08-07 20:05:06.851 00018 INFO  | [XenRpcClient]: Retry after 10 sec. Retry 6/10 for ListChangedBlocks

There is a XEN forum and guide from him, but I need to check internally as first

https://docs.xenserver.com/en-us/xenserver/developer/changed-block-tracking-guide/troubleshoot.html#you-cant-list-changed-blocks-between-two-vdi-snapshots

CBT: the thread to centralize your feedback | XCP-ng and XO forum
https://xcp-ng.org/forum/topic/9268/cbt-the-thread-to-centralize-your-feedback/364
Regards

Update -

Veeam came back and suggested I power off the vms with the warrning and power back on. This cleared the error for 2 out of the 4 vms. 1 vm still showed the error the other vm i was not able to power off at that time.

@olivierlambert - From veeam...

After checking with my team, is it possible if you can engage XEN support on this, I checked some of the XEN forum including the KB article about how to troubleshoot CBT errors, seems to be a clean metadata that can be done from their side Troubleshoot Changed Block Tracking | Develop for XenServer

CBT: the thread to centralize your feedback | XCP-ng and XO forum

Regards

Looks like our issue may be different at the end even with involving CBT... Let me know if i should continue in a new post. Support ticket created - Ticket#7762393

MajorP93

@msupport @acebmxer
Hello guys.
Thanks for sharing your experience with veeam and reporting these issues.
My team is also planning on evaluating veeam and I was wondering: did you get a response from veeam?
Did they give you a time line on when they plan to release a fix?

msupport

@MajorP93

Last Message from Veeam:
Veeam Support - Case # 08187386

thank you for your email!
We are currently waiting for R&D team conclusion, and a bit more time is required for the investigation. I am sorry for the possible inconveniences here!
Best regards,
Viktoria Nesmiyanova
Technical Customer Support - EMEA
Veeam Software