Veeam 13.1 Rocky9 Linux Appliance: Potential Data Loss with CBT and Workers with Expired Tokens

MajorP93

@msupport @acebmxer
Hello guys.
Thanks for sharing your experience with veeam and reporting these issues.
My team is also planning on evaluating veeam and I was wondering: did you get a response from veeam?
Did they give you a time line on when they plan to release a fix?

msupport

@MajorP93

Last Message from Veeam:
Veeam Support - Case # 08187386

thank you for your email!
We are currently waiting for R&D team conclusion, and a bit more time is required for the investigation. I am sorry for the possible inconveniences here!
Best regards,
Viktoria Nesmiyanova
Technical Customer Support - EMEA
Veeam Software

dsauce

@msupport We're testing Veeam now. From what I read, I thought CBT was supposed to be disabled in XCP as Veeam uses it's own CBT engine, is that not correct?

Also, is it normal for the SR to show a bunch of veeamsnap files for all the VDI's it backed up? I was under the impression Veeam was supposed to remove those when the backup was complete, but perhaps one of them needs to stay for tracking?

acebmxer

@dsauce

To test your theory out on my issue i just tried to disable CBT on XOA side and one vm gave me this error...

vdi.set
{
  "id": "11286a97-b773-4ec4-a0b7-464ab87254ca",
  "cbt": false
}
{
  "code": "UUID_INVALID",
  "params": [
    "VDI",
    "a52fe4ab-edc2-4975-a118-f6db84435379"
  ],
  "call": {
    "duration": 1,
    "method": "VDI.get_by_uuid",
    "params": [
      "* session id *",
      "a52fe4ab-edc2-4975-a118-f6db84435379"
    ]
  },
  "message": "UUID_INVALID(VDI, a52fe4ab-edc2-4975-a118-f6db84435379)",
  "name": "XapiError",
  "stack": "XapiError: UUID_INVALID(VDI, a52fe4ab-edc2-4975-a118-f6db84435379)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Another vm..

vdi.set
{
  "id": "966fe072-4865-48a2-867a-0eb59b23dcc7",
  "cbt": false
}
{
  "code": "UUID_INVALID",
  "params": [
    "VDI",
    "d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5"
  ],
  "call": {
    "duration": 1,
    "method": "VDI.get_by_uuid",
    "params": [
      "* session id *",
      "d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5"
    ]
  },
  "message": "UUID_INVALID(VDI, d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5)",
  "name": "XapiError",
  "stack": "XapiError: UUID_INVALID(VDI, d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Update - after backup completed with warnings i looked back and CBT was re-enabled in XOA. Did not help with my Veeam issues.

acebmxer

@msupport - Looks like veeam is still working with your on your issues. While veeam has pushed me off to vates / xen.

@poddingue - Any updates from Vates about these issues? Is it possible the least patches just pushed might help with either mine or @msupport's issue?

Update - Just got a reply back from veeam ...

As per internal testing, I’m escalation this to the next tier

Regards

poddingue

Nothing from our side that I can pass on, sorry.
Two public things I can point at, neither of which I've tested against your case: the updates that went live yesterday list tapdisk crash fixes among the storage changes (https://xcp-ng.org/blog/2026/08/18/august-2026-updates-1-for-xcp-ng-8-3-lts/), and there's an open PR on tapdisk picking up a cbtlog disk during commit and failing early because that driver has no commit action (https://github.com/xcp-ng/blktap/pull/17).
I'm reading a changelog and a PR body rather than reproducing anything, so treat both as leads.
@msupport, Danp's three questions from 6 August are still open (guest OS, guest tools, VHD or QCOW2), and filling those in is probably worth more than anything I can add here.
@dsauce, I don't know the answer to yours about CBT and the leftover veeamsnap VDIs, and I'd rather say so than guess on a thread about data loss.

msupport

@Danp
We encountered these issues with 4 VMs running different versions of Windows (2019–2022); one VM had the Rust Guest Agent, while the others had the Xen drivers. All VMs used VHD disks.

msupport

our solution to the problem

we want to share a post-incident analysis of a data-loss event on XCP-ng 8.3 LTS (shared block storage over FC, Veeam B&R 13.1 with CBT enabled on the pool) — consolidated from our own incident, topic 12402, the CBT feedback thread 9268, and the still-open blktap PR #17. The pieces fit together into one coherent failure chain, and we believe it may be worth a sticky/KB article.
A note on the trigger, to be fair and complete: in our case the interrupted jobs (step 2) were caused by expired/invalid worker tokens on the XCP-ng hosts, so the backups died mid-run with the locks in place. We consider that our availability problem. However, an interrupted backup must never be able to damage production VM data — cleaning up VDI locks on job abort is the hypervisor's job, and that is the part that turned a backup hiccup into guest data loss.

Failure chain (as we understand it)

Veeam backup job with CBT runs and sets locks on VDIs — paused: true + host_OpaqueRef:xxx: RW entries in the VDI sm_config (XAPI state.db).
The job is interrupted (timeout / crash / restart). The locks are never cleaned up → stale paused: true remains in sm_config. These entries are MRO, so xe vdi-param-remove can't clear them.
A later leaf coalesce / commit runs into a cbtlog disk: per PR #17, tapdisk_vbd_first_image returns the cbtlog disk on td_commit, and the cbtlog driver has no commit action → commit fails early. Result: broken VDI chains, CBT metadata VDIs without a vhd parent, hundreds of orphaned VDIs, and .cbtlog files hanging coalesces (as reported in thread 9268).
SR rescan believes a GC is already running and aborts; a host reboot was the only way to force the coalesce through (also reported in 9268).
Storage cleaning freezes the VM disk briefly, but the un-freeze fails on the stale lock → failed to unpause tapdisk ... VMs using this tapdisk have lost access to the corresponding disk(s).
The guest keeps writing on a frozen/lost disk → NTFS corruption inside the guest and, in our case, actual SQL Server data loss.
Step 5 matches exactly the theory Veeam R&D is currently investigating ("storage cleaning freezes VM disks briefly during backup and sometimes fails to un-freeze them"). The stale paused:true lock appears to be the missing "why" behind the failed unpause.

What helped us recover

Patching the stale lock out of XAPI state.db (stop xapi, backup state.db, remove paused + host_OpaqueRef entries from the affected VDI's sm_config, start xapi).
Then: reset CBT on the affected VDIs and trigger a full backup so CBT re-initializes cleanly — otherwise the next interrupted job re-creates the same situation.
For the coalesce backlog: with the affected VMs powered off and CBT disabled, snapshot-create-then-delete to kick the GC, watch SMlog, iterate. (Same recipe a user documented in 9268.)

27.08.2026 | Veeam identified this issue:
For all three: A disk refresh operation failed because a disk was in the “paused” state; for SQL03, there was also a failed unpause. This is a strong indication that the XCP-ng Storage API (VDI pause/unpause during the snapshot) is the crux of the problem here.

acebmxer

Veeam scheduled a remote call with me and pulled more log files. Of coarse when we ran the backup job twice in a row both times al vms were successful. Veeam needs to baby sit our backups :). The call was cut short do to internet going down. I have uploaded the logs and waiting to hear back.

Update - Veeam took alot more logs from Veeam and from xcp-ng pool.

Their response back -

I've got someone else getting similiar results, so I'm providing both of your logs to get some insights. Basically when you see the error, it's because something happened to the bitmap we left behind on the previous run and so next run, we re-read the entire disk. I've not found anything super clear to what's going wrong with the bitmap and why its gone, even from the Xen server logs, so I'm hoping from QA's eyes might see what I might be missing. I will keep you posted if they have any details.

Update 8.26.26 -

I just wanted to provide an update, the QA team is still checking stuff, but they did advised the following.
They noticed for the disks, they show there are configured XO native backups:
ie.
xo:backup:deltaChainLength: 4; xo:backup:contentKey: 4bce47c7-04bb-44ae-a687-c35c9cfbb2b2;
xo:backup:job: e3616a64-6b83-4bd0-80b1-00523678e909; xo:backup:includeNonNbdQcow2Fix: true; xo:backup:schedule: 3488baee-fe48-4acb-a5d0-e728fe28efa1;
xo:backup:vm: 5703adef-d804-6b15-ba2f-7b3357a711bb; xo:backup:datetime: 20260819T01:00:33Z

I believe you said the native XO Backup was disabled, can you re-confirm if that is accurate, and provide a screenshot for XO's Job and backup list to verify.
QA did confirm that native XO Backups can cause issues as it treats the objects as two different chains and so that's why a CBT comparison can fail. QA says they are looking into ways to change how the comparison works in a future update, and expects it to help if in situations if there are native backups, but asked if the Native backup can be verified as disabled for now.

I have respnded stating that Veeam backup is only used for Windows based vms. Backup in Xen Orchestra is only used for linux base vms.

acebmxer

I have received a very long update back from Veeam on my backup issues....

There appears to be another user with similar setup / issue not sure if that user is the OP this post specifically...

Hello,

Thank you for your patience. The QA team has finished the analysis, and I am going to outline the details as below:

1. CBT Inconsistency Issue

Whenever you see the Warning about CBT showing:

2026-08-16 17:37:27.459 00079 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 291af3d7-28c1-15a9-7f13-c6a1a12283e9 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated].

We can go to the previous run and see that we get reports that the CBT of the previous snapshot is inconsistent and thus removed by XCP:

2026-08-16 09:54:34.120 00004 ERROR | [XenBackupManager]: Failed to retain the data for the snapshot 5a17fb31-2616-4e72-a85c-e235883c8c91
Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 39c0f25f-e811-ee45-1b0b-e4a9ca9839ae (Async.VDI.data_destroy) failed: . VDI_NO_CBT_METADATA. OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed

The team has been reviewing and testing, and this is what they see. We send the request for the snapshot to be created, and it is sent to the coordinator: xcp-ng-vyadytkn

Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] vdi_clone: introduced VDI: OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed (5a17fb31-2616-4e72-a85c-e235883c8c91)

CBT shows as open and reading fine:

Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] ['/usr/sbin/cbt-util', 'set', '-n', '/var/run/sr-mount/7911c9c5-5f20-01e1-8b8d-39c6a98a2704/5a17fb31-2616-4e72-a85c-e235883c8c91.cbtlog', '-f', '1']
Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread]   pread SUCCESS

But look at how on xcp-ng-host2 the CBT gets marked as inconsistent by XCP, even though it's open and reading:

Aug 16 09:51:05 xcp-ng-host2 SM: [1683689][MainThread] Changed Block Tracking metadata is inconsistent for disk 5a17fb31-2616-4e72-a85c-e235883c8c91.

The Breakdown:

xcp-ng-vyadytkn was the coordinator—the one we talk to, who then passes everything around to the hosts.
xcp-ng-host2 was the host that the VM resided on at that time.
xcp-ng-host2 is marking the CBT as inconsistent and deleting the snapshot CBT log, which means we cannot reference it on the next run. We do not see anything else interacting with the CBT besides that host.

This matches exactly what we see with another client running the same setup. The team successfully replicated the environment, which is configured as follows:

VM storage is NFS.
The VM is running on a host that is not the coordinator.

They have been able to reproduce this behavior occasionally, and the working theory is that the VM host keeps its own tracking separate from the coordinator. Part of the backup process requires the VM host to issue a pause/resume via a process called tapdisk. When it resumes, it pushes a data cache (likely inside the NFS cache), overwriting the CBT reference held by the coordinator server. It acts as a race condition—whoever pushes the CBT data last wins. If the VM host pushes last, it breaks what the coordinator is sending.

Next Steps for CBT:
The team is working to raise this issue directly with Vates so they can address the race condition. We ask that you also open a Vates ticket if possible to help draw more attention to the bug. We are trying to find ways to code around this race condition in the future, but there are currently no ETAs or guarantees.

2. Synthetic Full Failures (`Delilah_ArcFS01`)

In addition to the CBT bug, the team discovered a separate issue. Recently, the synthetic fulls for Delilah_ArcFS01 have been failing. This appears to be related to the NFS repository:

[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)    Failed to execute full transform task
[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)    Agent: Failed to process method {Transform.CompileFIB}: NfsFileEx was already stopped. File: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]] (Veeam.Backup.Common.CCppComponentException)
[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)       in c++: Failed to execute command Command: READ, Offset: 2523136, Data size: 659456, Chunk size: 131072
[30.08.2026 00:43:22.856]    <24> [0007]   Error (1)       in c++: Failed to read file: Offset: 2523136, Block size: 659456, File: Path: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]], Handle: [01000702080061030000000007fffd5f65840fb20000000000000000150061038a2f7c0b0400610379207c0b], Read chunk size: 131072, Write chunk size: 131072, Read only: true

Because it continuously fails during the synthetic full, it eventually causes the snapshot to be lost when retries fail. The team will change this logic in a future update.

Action Items for the NFS Issue:
To help prevent these snapshot loss failures, could you please provide the logs from the repository NFS (192.168.20.91)?

Export the logs from the Veeam server and select the repository host.
Provide the results of running this command directly on the Repository host:

journalctl --since "30 days ago" > journal_repo.log

Temporary Workaround:
If you can, please temporarily switch to active fulls instead of synthetic fulls to help stabilize the job.

Please let me know if you have any questions, and if you are able to raise that ticket with Vates.

They also just responded back with this statment...

Regarding the second part of the last email with the noticed Synthetic full issue, I actually would like you to also make this registry entry on the Veeam server and keep synthetic fulls enabled to see if it helps with that issue:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
Name: Nfs3CommandWaitTimeoutSec
Type: DWORD
Value (In Decimal): 86400

Veeam 13.1 Rocky9 Linux Appliance: Potential Data Loss with CBT and Workers with Expired Tokens

1. CBT Inconsistency Issue

2. Synthetic Full Failures (Delilah_ArcFS01)

2. Synthetic Full Failures (`Delilah_ArcFS01`)