@poddingue My colleage managed to workaround the issue by re-installing the lab on other hardware, I think it is now Cisco UCS-servers. It was HP ProLiant-servers before. Maybe it was an issue with some of the NIC or firmware, im not really sure, but it works now.

@skipthompson81 said: this CONFLICT is not OK! You will need to provide more details about this specific backup job. It's possible that the backup succeeded, but then the subsequent Health Check failed or timed out. Have you tried downloading and examining the full log from the failed backup? [image: 1781371917690-65a4470c-0b71-474b-b506-1512e1fbbf45-image.jpeg]

Problem still present in commit 04deb. Here is the log from the XO6 interface: "XapiError: VDI_IN_USE(OpaqueRef:ce1e2c1b-9b52-d813-88fc-18513a51306f, destroy)\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/_getTaskResult.mjs:13:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1078:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1112:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1102:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202606131310/packages/xen-api/index.mjs:1275:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)" Thanks for your assistance ! ! ! Regards, Marc

@poddingue Since setting NBD=1, I have not seen the problem. SR is NFS on dual 40G ethernet with a TrueNAS scale 25.10 server using all NVMe SSD, so storage performance is as good as I can make it. I'll have to enable NBD=2 again to see if it still happens and if I can find the relevant part of the logs. As this is a random problem I can't recreate it on a normal test environment.

Pilow's right that moving a VM to another SR forces one full pass while the CBT bitmap is rebuilt; that part is expected. But your screenshot actually shows the likely culprit for the all-VMs-fall-back-to-full pattern: you have Purge snapshot data when using CBT enabled, and XO's incremental backup docs flag exactly that combination as a known issue where you can occasionally get unexpected fulls: https://docs.xen-orchestra.com/xo5/incremental_backups#known-issues. It might be worth running a few jobs with that toggle off to see if the deltas hold. It is a known rough edge on the CBT side, so following the central CBT feedback thread and maybe a nudge to @Team-XO-Backend wouldn't hurt.

@henri9813 said: Hello, I see also this behavior which is "new" since few weeks. Previously, when a backup start: it stake a snapshot ( if there another one before, it delete it ). it upload the snapshot as a backup it coalesce the backup on the remote. end of the game. Now, the old snapshots are not deleted anymore which can lead easily to some disk full. Even with a retention of 1, the problem is present. I observe this only in Backup job, not DR/CR job. I just updated my XO to latest version, i will see if the issue is fixed. Hi @henri9813 , Issue should be resolved in 6.5.x, can you confirm on your side ? Thanks!

@tsukraw multiple NBD connection will open multiple reading connection, but the writing one is always one stream per disk in incremental replication with full replication, it's one stream for read and one for write

@pierrebrunet Thanks.

@pierrebrunet I have updated my XOA and Proxies... It seems i did not see the warning on the next round of backups. Will continue to monitor now patches are installed.

I'm waiting for veeam 13.1 release as well. Will move a few things over and test out the native backup in the meantime.

@pierrebrunet I inadvertently found more information yesterday. I installed an old e492b commit on a new machine, configured identically to the original, and got a different error but the same effect. The new error is: xcp xo backup Error: read ETIMEDOUT That lead me to this page: https://xcp-ng.org/forum/topic/10799/backup-timeout-error I moved the XO machine onto the same subnet as XCP, leaving the storage behind in a different network. This fixed the issue, at least on commit e492b ... I have not yet tried it on the new release... however I am pretty confident it will be successful. If I'm honest, I think this is a tuning issue between XO and XCP. XO is now mounting an NFS share across this subnet boundary and that mount is not having any sort of performance issue, but for some reason XO times out transferring the same data from dom0? That seems off to me. It's not a probem moving XO to the other subnet, so I've done that, but maybe this deserves a look in the future.

@Rod-G These have 9.4.2-xxxx and I need to go through and update everything now that I see how far behind I am. the second VM completed fine after its reboot, something was just stuck in a dirty state and got in the way of the import or health check.

@tonyp90 great, thanks !

@Danp fully patched 8.3 (through the most recent May patches). XO commit d810e (master commit e3a58). I usually update XO around the first of each month; so it's a little bit behind master. There are crash logs in /var/crash. I can provide a log bundle if needed, or copy/paste some info here if I know what to provide.

the last rewrite of the stream processing ( spring 2025 ) focused on stability and memory footprint, and , on a standard cpu, it tops at around 300MB/s per backup job. Your benchmarks are very interesting, and they confirm most of it. this limit was not really an issue since, in most case the xapi was limiting around 100MB/s per disk , but it will be more a more visible limit Note that master have some fixes on the memory usage (not related to backups) That's why we have started an internal workforce focused on performance, with all the teams from the kernel to the backups, including storage, network and xapi. If I can brag a little : [image: 1779106650898-afd7b59b-a4f0-4a92-88ee-2c7ba52d18bf-image.jpeg] i9 , nvme disk , backup to a nvme disk in passthrough, xoa and vm are on the same host, so it's quite far from real world data, but it shows where the limit is

@JL457 Hi, For now it looks like Wasabi is not sending us the correct date format, which is strange because we support this provider and don't usually have issues. In order to allow us to investigate further, could you send us the full backup job logs ? You can find them by clicking on the failed backup status and then on the download logs button: [image: 1779106635704-export-logs.png] Relevant XO logs would also help. If you are a client, also don't hesitate to open a ticket with an open support tunnel. Thanks.

@pierrebrunet @poddingue the size is really 2.17Tb, but showing last incremental size on the key I'll wait for the patch, this is really a visual bug, backup is working okay.