Thanks for the feedback @norpan , worth it! Ping @florent

Got this issue since 2026-07-12.

@pierrebrunet said: Hi, just to be sure, are you saying it is working ok apart one task 8 days ago? Yes, after I adjusted the time it's working

@simonp , thank you very much and all of your team ! ! ! Keep up the outstanding work. Regards, Marc

Indeed, it should be detected/reported correctly to avoid losing time finding the problem. Let me ping @julienXOVates

Ping @julienXOVates

Ping @julienXOVates

@Pilow pretty much

@pierrebrunet ticket created https://help.vates.tech/#ticket/zoom/59723

@poddingue Since setting NBD=1, I have not seen the problem. SR is NFS on dual 40G ethernet with a TrueNAS scale 25.10 server using all NVMe SSD, so storage performance is as good as I can make it. I'll have to enable NBD=2 again to see if it still happens and if I can find the relevant part of the logs. As this is a random problem I can't recreate it on a normal test environment.

Pilow's right that moving a VM to another SR forces one full pass while the CBT bitmap is rebuilt; that part is expected. But your screenshot actually shows the likely culprit for the all-VMs-fall-back-to-full pattern: you have Purge snapshot data when using CBT enabled, and XO's incremental backup docs flag exactly that combination as a known issue where you can occasionally get unexpected fulls: https://docs.xen-orchestra.com/xo5/incremental_backups#known-issues. It might be worth running a few jobs with that toggle off to see if the deltas hold. It is a known rough edge on the CBT side, so following the central CBT feedback thread and maybe a nudge to @Team-XO-Backend wouldn't hurt.

@henri9813 said: Hello, I see also this behavior which is "new" since few weeks. Previously, when a backup start: it stake a snapshot ( if there another one before, it delete it ). it upload the snapshot as a backup it coalesce the backup on the remote. end of the game. Now, the old snapshots are not deleted anymore which can lead easily to some disk full. Even with a retention of 1, the problem is present. I observe this only in Backup job, not DR/CR job. I just updated my XO to latest version, i will see if the issue is fixed. Hi @henri9813 , Issue should be resolved in 6.5.x, can you confirm on your side ? Thanks!

@tsukraw multiple NBD connection will open multiple reading connection, but the writing one is always one stream per disk in incremental replication with full replication, it's one stream for read and one for write

@pierrebrunet Thanks.

@pierrebrunet I have updated my XOA and Proxies... It seems i did not see the warning on the next round of backups. Will continue to monitor now patches are installed.

I'm waiting for veeam 13.1 release as well. Will move a few things over and test out the native backup in the meantime.

@pierrebrunet I inadvertently found more information yesterday. I installed an old e492b commit on a new machine, configured identically to the original, and got a different error but the same effect. The new error is: xcp xo backup Error: read ETIMEDOUT That lead me to this page: https://xcp-ng.org/forum/topic/10799/backup-timeout-error I moved the XO machine onto the same subnet as XCP, leaving the storage behind in a different network. This fixed the issue, at least on commit e492b ... I have not yet tried it on the new release... however I am pretty confident it will be successful. If I'm honest, I think this is a tuning issue between XO and XCP. XO is now mounting an NFS share across this subnet boundary and that mount is not having any sort of performance issue, but for some reason XO times out transferring the same data from dom0? That seems off to me. It's not a probem moving XO to the other subnet, so I've done that, but maybe this deserves a look in the future.

@Rod-G These have 9.4.2-xxxx and I need to go through and update everything now that I see how far behind I am. the second VM completed fine after its reboot, something was just stuck in a dirty state and got in the way of the import or health check.

@tonyp90 great, thanks !

the last rewrite of the stream processing ( spring 2025 ) focused on stability and memory footprint, and , on a standard cpu, it tops at around 300MB/s per backup job. Your benchmarks are very interesting, and they confirm most of it. this limit was not really an issue since, in most case the xapi was limiting around 100MB/s per disk , but it will be more a more visible limit Note that master have some fixes on the memory usage (not related to backups) That's why we have started an internal workforce focused on performance, with all the teams from the kernel to the backups, including storage, network and xapi. If I can brag a little : [image: 1779106650898-afd7b59b-a4f0-4a92-88ee-2c7ba52d18bf-image.jpeg] i9 , nvme disk , backup to a nvme disk in passthrough, xoa and vm are on the same host, so it's quite far from real world data, but it shows where the limit is