Veeam 13.1 Rocky9 Linux Appliance: Potential Data Loss with CBT and Workers with Expired Tokens

[acebmxer]

I have also opened a support ticket with Veeam. Well trying to... Issue with my account preventing me to and working to fix that issue.

Veeam forum did make this statment about CBT and snapshots -

• There is one known snapshot issue that I'm guessing could possibly have this side-effect (noted in the release notes BTW). We take standard Xen snapshots for CBT but then immediately rename them with our own naming convention. We've noticed that sometimes this rename operation returns success status even if it fails. In this case I can see how this might throw this sort of error but again best to confirm w/support

Veeam support ticket - 08188639

Update from Veeam -

Thanks for the update

I'm currently checking with my team and I will update as soon as possible, what I noticed in the debug logs is the following API failing

\Backup\Plugins\XEN\Backup\Backup Job 1\Delilah ArcDC02 with debug enable

2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]:  <== Request localhost:19000, IsTaskFinished, body: {"taskId":"491371d9-4699-4087-b549-9862d2188177"}
2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]:  ==> Response localhost:19000, IsTaskFinished, success, duration: 0.1404 msec, body: false
2026-08-07 20:05:04.843 00019 INFO  | [XenRpcClient]: Start ListChangedBlocks
2026-08-07 20:05:04.843 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/Async.VDI.list_changed_blocks, body: ["OpaqueRef:b77ca409-d438-d97e-d8c3-f212647ad469","OpaqueRef:8c83f698-8836-3b4c-1506-8e97315aae1f"]
2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/Async.VDI.list_changed_blocks code: "OK", duration: 2 msec, body:{"opaque_ref":"OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"}
2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:04.847 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 2 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":["cancel"],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"1970-01-01T00:00:00Z","status":"pending","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":0.0,"type":"<none/>","result":"","error_info":[],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"()","opaque_ref":null}
2026-08-07 20:05:04.848 00019 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "pending" Progress: 0
2026-08-07 20:05:06.848 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 1 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":[],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"2026-08-08T00:05:05Z","status":"failure","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":1.0,"type":"<none/>","result":"","error_info":["SR_BACKEND_FAILURE_460","","Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]",""],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 150))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))","opaque_ref":null}
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "failure" Progress: 1
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.destroy, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:06.851 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.destroy code: "OK", duration: 0.735 msec, body:""
2026-08-07 20:05:06.851 00018 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. 
Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. 
 ---> Failed to calculate changed blocks for given VDIs.
   --- End of inner exception stack trace ---
   at Veeam.XenBackup.RestClient.XenRpcClient.GetTaskResult(XenRef`1 taskRef, CancellationToken cancellationToken)
   at Veeam.XenBackup.RestClient.XenRpcClient.<>c__DisplayClass94_0.<<ListChangedBlocksAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Veeam.Vbf.Common.Helper.Retry.RetryHelper.ExecuteActionAsync[T](Func`2 asyncAction, String description, ILogger logger, LogLevel logLevel, CancellationToken cancellationToken)
2026-08-07 20:05:06.851 00018 INFO  | [XenRpcClient]: Retry after 10 sec. Retry 6/10 for ListChangedBlocks 

There is a XEN forum and guide from him, but I need to check internally as first

https://docs.xenserver.com/en-us/xenserver/developer/changed-block-tracking-guide/troubleshoot.html#you-cant-list-changed-blocks-between-two-vdi-snapshots

CBT: the thread to centralize your feedback | XCP-ng and XO forum
https://xcp-ng.org/forum/topic/9268/cbt-the-thread-to-centralize-your-feedback/364
Regards

Update -

Veeam came back and suggested I power off the vms with the warrning and power back on. This cleared the error for 2 out of the 4 vms. 1 vm still showed the error the other vm i was not able to power off at that time.

@olivierlambert - From veeam...

After checking with my team, is it possible if you can engage XEN support on this, I checked some of the XEN forum including the KB article about how to troubleshoot CBT errors, seems to be a clean metadata that can be done from their side Troubleshoot Changed Block Tracking | Develop for XenServer

CBT: the thread to centralize your feedback | XCP-ng and XO forum

Regards

Looks like our issue may be different at the end even with involving CBT... Let me know if i should continue in a new post. Support ticket created - Ticket#7762393

[MajorP93]

@msupport @acebmxer
Hello guys.
Thanks for sharing your experience with veeam and reporting these issues.
My team is also planning on evaluating veeam and I was wondering: did you get a response from veeam?
Did they give you a time line on when they plan to release a fix?

[msupport]

@MajorP93

Last Message from Veeam:
Veeam Support - Case # 08187386

thank you for your email!
We are currently waiting for R&D team conclusion, and a bit more time is required for the investigation. I am sorry for the possible inconveniences here!
Best regards,
Viktoria Nesmiyanova
Technical Customer Support - EMEA
Veeam Software

[dsauce]

@msupport We're testing Veeam now. From what I read, I thought CBT was supposed to be disabled in XCP as Veeam uses it's own CBT engine, is that not correct?

Also, is it normal for the SR to show a bunch of veeamsnap files for all the VDI's it backed up? I was under the impression Veeam was supposed to remove those when the backup was complete, but perhaps one of them needs to stay for tracking?

[acebmxer]

@dsauce

To test your theory out on my issue i just tried to disable CBT on XOA side and one vm gave me this error...

vdi.set
{
  "id": "11286a97-b773-4ec4-a0b7-464ab87254ca",
  "cbt": false
}
{
  "code": "UUID_INVALID",
  "params": [
    "VDI",
    "a52fe4ab-edc2-4975-a118-f6db84435379"
  ],
  "call": {
    "duration": 1,
    "method": "VDI.get_by_uuid",
    "params": [
      "* session id *",
      "a52fe4ab-edc2-4975-a118-f6db84435379"
    ]
  },
  "message": "UUID_INVALID(VDI, a52fe4ab-edc2-4975-a118-f6db84435379)",
  "name": "XapiError",
  "stack": "XapiError: UUID_INVALID(VDI, a52fe4ab-edc2-4975-a118-f6db84435379)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Another vm..

vdi.set
{
  "id": "966fe072-4865-48a2-867a-0eb59b23dcc7",
  "cbt": false
}
{
  "code": "UUID_INVALID",
  "params": [
    "VDI",
    "d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5"
  ],
  "call": {
    "duration": 1,
    "method": "VDI.get_by_uuid",
    "params": [
      "* session id *",
      "d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5"
    ]
  },
  "message": "UUID_INVALID(VDI, d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5)",
  "name": "XapiError",
  "stack": "XapiError: UUID_INVALID(VDI, d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Update - after backup completed with warnings i looked back and CBT was re-enabled in XOA. Did not help with my Veeam issues.

[acebmxer]

@msupport - Looks like veeam is still working with your on your issues. While veeam has pushed me off to vates / xen.

@poddingue - Any updates from Vates about these issues? Is it possible the least patches just pushed might help with either mine or @msupport's issue?

Update - Just got a reply back from veeam ...

As per internal testing, I’m escalation this to the next tier

Regards

[poddingue]

Nothing from our side that I can pass on, sorry.
Two public things I can point at, neither of which I've tested against your case: the updates that went live yesterday list tapdisk crash fixes among the storage changes (https://xcp-ng.org/blog/2026/08/18/august-2026-updates-1-for-xcp-ng-8-3-lts/), and there's an open PR on tapdisk picking up a cbtlog disk during commit and failing early because that driver has no commit action (https://github.com/xcp-ng/blktap/pull/17).
I'm reading a changelog and a PR body rather than reproducing anything, so treat both as leads.
@msupport, Danp's three questions from 6 August are still open (guest OS, guest tools, VHD or QCOW2), and filling those in is probably worth more than anything I can add here.
@dsauce, I don't know the answer to yours about CBT and the leftover veeamsnap VDIs, and I'd rather say so than guess on a thread about data loss.

[msupport]

@Danp
We encountered these issues with 4 VMs running different versions of Windows (2019–2022); one VM had the Rust Guest Agent, while the others had the Xen drivers. All VMs used VHD disks.

[msupport]

our solution to the problem

we want to share a post-incident analysis of a data-loss event on XCP-ng 8.3 LTS (shared block storage over FC, Veeam B&R 13.1 with CBT enabled on the pool) — consolidated from our own incident, topic 12402, the CBT feedback thread 9268, and the still-open blktap PR #17. The pieces fit together into one coherent failure chain, and we believe it may be worth a sticky/KB article.
A note on the trigger, to be fair and complete: in our case the interrupted jobs (step 2) were caused by expired/invalid worker tokens on the XCP-ng hosts, so the backups died mid-run with the locks in place. We consider that our availability problem. However, an interrupted backup must never be able to damage production VM data — cleaning up VDI locks on job abort is the hypervisor's job, and that is the part that turned a backup hiccup into guest data loss.

Failure chain (as we understand it)

Veeam backup job with CBT runs and sets locks on VDIs — paused: true + host_OpaqueRef:xxx: RW entries in the VDI sm_config (XAPI state.db).
The job is interrupted (timeout / crash / restart). The locks are never cleaned up → stale paused: true remains in sm_config. These entries are MRO, so xe vdi-param-remove can't clear them.
A later leaf coalesce / commit runs into a cbtlog disk: per PR #17, tapdisk_vbd_first_image returns the cbtlog disk on td_commit, and the cbtlog driver has no commit action → commit fails early. Result: broken VDI chains, CBT metadata VDIs without a vhd parent, hundreds of orphaned VDIs, and .cbtlog files hanging coalesces (as reported in thread 9268).
SR rescan believes a GC is already running and aborts; a host reboot was the only way to force the coalesce through (also reported in 9268).
Storage cleaning freezes the VM disk briefly, but the un-freeze fails on the stale lock → failed to unpause tapdisk ... VMs using this tapdisk have lost access to the corresponding disk(s).
The guest keeps writing on a frozen/lost disk → NTFS corruption inside the guest and, in our case, actual SQL Server data loss.
Step 5 matches exactly the theory Veeam R&D is currently investigating ("storage cleaning freezes VM disks briefly during backup and sometimes fails to un-freeze them"). The stale paused:true lock appears to be the missing "why" behind the failed unpause.

What helped us recover

Patching the stale lock out of XAPI state.db (stop xapi, backup state.db, remove paused + host_OpaqueRef entries from the affected VDI's sm_config, start xapi).
Then: reset CBT on the affected VDIs and trigger a full backup so CBT re-initializes cleanly — otherwise the next interrupted job re-creates the same situation.
For the coalesce backlog: with the affected VMs powered off and CBT disabled, snapshot-create-then-delete to kick the GC, watch SMlog, iterate. (Same recipe a user documented in 9268.)

[acebmxer]

Veeam scheduled a remote call with me and pulled more log files. Of coarse when we ran the backup job twice in a row both times al vms were successful. Veeam needs to baby sit our backups :). The call was cut short do to internet going down. I have uploaded the logs and waiting to hear back.