@bogikornel We had a very old XO built from sources, which worked fine. One day we decided to deploy a new VM and migrate to a new XO built from sources, and immediately ran into problems. All tho it was resolved, after switching machines from HP to Cisco, which makes us believe it has something to do with the NIC, driver or firmware.

Hello again! [image: 1783187754229-c5189e53-c2d1-4de5-a26d-243ddd41e5db-image.jpeg] Here are the logs: b2_mirror_error.txt Hope it helps!

@olivierlambert said: On your backup remote, look at the file left behind by the failed run It's empty I increased the schedule to every 30 min. All 3 of them are OK [image: 1783076106533-ddfcf1d5-bae9-4a7a-b45d-64ba131fe80c-image.jpeg]

Ping @julienXOVates

Ping @julienXOVates

Evening @pierrebrunet My bad. I am now on commit b440f and the issue persist. I have written you a message with the link to the file. Regards, Marc

@Pilow pretty much

@pierrebrunet ticket created https://help.vates.tech/#ticket/zoom/59723

@poddingue Since setting NBD=1, I have not seen the problem. SR is NFS on dual 40G ethernet with a TrueNAS scale 25.10 server using all NVMe SSD, so storage performance is as good as I can make it. I'll have to enable NBD=2 again to see if it still happens and if I can find the relevant part of the logs. As this is a random problem I can't recreate it on a normal test environment.

Pilow's right that moving a VM to another SR forces one full pass while the CBT bitmap is rebuilt; that part is expected. But your screenshot actually shows the likely culprit for the all-VMs-fall-back-to-full pattern: you have Purge snapshot data when using CBT enabled, and XO's incremental backup docs flag exactly that combination as a known issue where you can occasionally get unexpected fulls: https://docs.xen-orchestra.com/xo5/incremental_backups#known-issues. It might be worth running a few jobs with that toggle off to see if the deltas hold. It is a known rough edge on the CBT side, so following the central CBT feedback thread and maybe a nudge to @Team-XO-Backend wouldn't hurt.

@henri9813 said: Hello, I see also this behavior which is "new" since few weeks. Previously, when a backup start: it stake a snapshot ( if there another one before, it delete it ). it upload the snapshot as a backup it coalesce the backup on the remote. end of the game. Now, the old snapshots are not deleted anymore which can lead easily to some disk full. Even with a retention of 1, the problem is present. I observe this only in Backup job, not DR/CR job. I just updated my XO to latest version, i will see if the issue is fixed. Hi @henri9813 , Issue should be resolved in 6.5.x, can you confirm on your side ? Thanks!

@tsukraw multiple NBD connection will open multiple reading connection, but the writing one is always one stream per disk in incremental replication with full replication, it's one stream for read and one for write

@pierrebrunet Thanks.

@pierrebrunet I have updated my XOA and Proxies... It seems i did not see the warning on the next round of backups. Will continue to monitor now patches are installed.

I'm waiting for veeam 13.1 release as well. Will move a few things over and test out the native backup in the meantime.

@pierrebrunet I inadvertently found more information yesterday. I installed an old e492b commit on a new machine, configured identically to the original, and got a different error but the same effect. The new error is: xcp xo backup Error: read ETIMEDOUT That lead me to this page: https://xcp-ng.org/forum/topic/10799/backup-timeout-error I moved the XO machine onto the same subnet as XCP, leaving the storage behind in a different network. This fixed the issue, at least on commit e492b ... I have not yet tried it on the new release... however I am pretty confident it will be successful. If I'm honest, I think this is a tuning issue between XO and XCP. XO is now mounting an NFS share across this subnet boundary and that mount is not having any sort of performance issue, but for some reason XO times out transferring the same data from dom0? That seems off to me. It's not a probem moving XO to the other subnet, so I've done that, but maybe this deserves a look in the future.

@Rod-G These have 9.4.2-xxxx and I need to go through and update everything now that I see how far behind I am. the second VM completed fine after its reboot, something was just stuck in a dirty state and got in the way of the import or health check.