Categories

  • All news regarding Xen and XCP-ng ecosystem

    145 Topics
    5k Posts
    B
    @gduperrey Installed on all pools. No issues so far.
  • Everything related to the virtualization platform

    1k Topics
    15k Posts
    D
    @dthenot Thanks - I'll be interested to hear about your findings. I'll hold off on submitting a report until I hear back.
  • 3k Topics
    29k Posts
    acebmxerA
    I have received a very long update back from Veeam on my backup issues.... There appears to be another user with similar setup / issue not sure if that user is the OP this post specifically... Hello, Thank you for your patience. The QA team has finished the analysis, and I am going to outline the details as below: 1. CBT Inconsistency Issue Whenever you see the Warning about CBT showing: 2026-08-16 17:37:27.459 00079 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 291af3d7-28c1-15a9-7f13-c6a1a12283e9 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. We can go to the previous run and see that we get reports that the CBT of the previous snapshot is inconsistent and thus removed by XCP: 2026-08-16 09:54:34.120 00004 ERROR | [XenBackupManager]: Failed to retain the data for the snapshot 5a17fb31-2616-4e72-a85c-e235883c8c91 Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 39c0f25f-e811-ee45-1b0b-e4a9ca9839ae (Async.VDI.data_destroy) failed: . VDI_NO_CBT_METADATA. OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed The team has been reviewing and testing, and this is what they see. We send the request for the snapshot to be created, and it is sent to the coordinator: xcp-ng-vyadytkn Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] vdi_clone: introduced VDI: OpaqueRef:a005d328-6da0-b8b7-34ef-0607ed44c2ed (5a17fb31-2616-4e72-a85c-e235883c8c91) CBT shows as open and reading fine: Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] ['/usr/sbin/cbt-util', 'set', '-n', '/var/run/sr-mount/7911c9c5-5f20-01e1-8b8d-39c6a98a2704/5a17fb31-2616-4e72-a85c-e235883c8c91.cbtlog', '-f', '1'] Aug 16 09:50:08 xcp-ng-vyadytkn SM: [1555017][MainThread] pread SUCCESS But look at how on xcp-ng-host2 the CBT gets marked as inconsistent by XCP, even though it's open and reading: Aug 16 09:51:05 xcp-ng-host2 SM: [1683689][MainThread] Changed Block Tracking metadata is inconsistent for disk 5a17fb31-2616-4e72-a85c-e235883c8c91. The Breakdown: xcp-ng-vyadytkn was the coordinator—the one we talk to, who then passes everything around to the hosts. xcp-ng-host2 was the host that the VM resided on at that time. xcp-ng-host2 is marking the CBT as inconsistent and deleting the snapshot CBT log, which means we cannot reference it on the next run. We do not see anything else interacting with the CBT besides that host. This matches exactly what we see with another client running the same setup. The team successfully replicated the environment, which is configured as follows: VM storage is NFS. The VM is running on a host that is not the coordinator. They have been able to reproduce this behavior occasionally, and the working theory is that the VM host keeps its own tracking separate from the coordinator. Part of the backup process requires the VM host to issue a pause/resume via a process called tapdisk. When it resumes, it pushes a data cache (likely inside the NFS cache), overwriting the CBT reference held by the coordinator server. It acts as a race condition—whoever pushes the CBT data last wins. If the VM host pushes last, it breaks what the coordinator is sending. Next Steps for CBT: The team is working to raise this issue directly with Vates so they can address the race condition. We ask that you also open a Vates ticket if possible to help draw more attention to the bug. We are trying to find ways to code around this race condition in the future, but there are currently no ETAs or guarantees. 2. Synthetic Full Failures (Delilah_ArcFS01) In addition to the CBT bug, the team discovered a separate issue. Recently, the synthetic fulls for Delilah_ArcFS01 have been failing. This appears to be related to the NFS repository: [30.08.2026 00:43:22.856] <24> [0007] Error (1) Failed to execute full transform task [30.08.2026 00:43:22.856] <24> [0007] Error (1) Agent: Failed to process method {Transform.CompileFIB}: NfsFileEx was already stopped. File: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]] (Veeam.Backup.Common.CCppComponentException) [30.08.2026 00:43:22.856] <24> [0007] Error (1) in c++: Failed to execute command Command: READ, Offset: 2523136, Data size: 659456, Chunk size: 131072 [30.08.2026 00:43:22.856] <24> [0007] Error (1) in c++: Failed to read file: Offset: 2523136, Block size: 659456, File: Path: [Host:, Mount: [/volume1/veeam], Disk: [Delilah ArcFS01 Backup/Delilah ArcFS01 Backup_2026-08-29T232336.vib], Type: [nfs3 (1)]], Handle: [01000702080061030000000007fffd5f65840fb20000000000000000150061038a2f7c0b0400610379207c0b], Read chunk size: 131072, Write chunk size: 131072, Read only: true Because it continuously fails during the synthetic full, it eventually causes the snapshot to be lost when retries fail. The team will change this logic in a future update. Action Items for the NFS Issue: To help prevent these snapshot loss failures, could you please provide the logs from the repository NFS (192.168.20.91)? Export the logs from the Veeam server and select the repository host. Provide the results of running this command directly on the Repository host: journalctl --since "30 days ago" > journal_repo.log Temporary Workaround: If you can, please temporarily switch to active fulls instead of synthetic fulls to help stabilize the job. Please let me know if you have any questions, and if you are able to raise that ticket with Vates. They also just responded back with this statment... Regarding the second part of the last email with the noticed Synthetic full issue, I actually would like you to also make this registry entry on the Veeam server and keep synthetic fulls enabled to see if it helps with that issue: Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication Name: Nfs3CommandWaitTimeoutSec Type: DWORD Value (In Decimal): 86400
  • Our hyperconverged storage solution

    51 Topics
    808 Posts
    J
    @poddingue Okay, we're back here. I was told I have a hung CIFS mount that no longer exists, so a host needs to reboot to clear it, and faulty XOSTOR resources. But I can't reboot the host until I can move VMs off. Is there any indication of how to repair the faulty XOSTOR resources so I can move the VMs and reboot the host? Thanks for taking a look.
  • 37 Topics
    136 Posts
    J
    @AtaxyaNetwork Merci pour tes recherches ! Oui "cd_label" serait cool comme ajout au plugin ce qui permet sur les distro type Fedora/Redhat de ne pas avoir de boot_command à gérer