@olivierlambert it has happened again

"XXXXX-nfs-02 (FAST all Flash Storage ZFS/NFS): failed to unpause tapdisk","Failed to unpause tapdisk for VDI ab867371-7a74-4a9f-88a9-9fb89e7dd3de, VMs using this tapdisk have lost access to the corresponding disk(s)","XXXXX XXXX-xcp Pool","Oct 23, 2025 8:18 PM",""

VM for this VDI had been running on host XXX-xcp-03 but there were attachments on the vdi from host XXX-xcp-07 .

we did not manually move this VM but it has moved.

I really think this problem is getting caused by load balancer moving things around while delta backups are running.

can we look into having XOA LB plugin NOT move VM's getting touched by the XOA backup processes?

@DustinB thats not the point of the post.

the point of the post is on HTTP connection has timed out, the TASK never actually times out on the proxy.

I am aware it says space issues on the SR and its unfortunate that error is in the screen shot but its a distraction, not the actual problem.

XOA 5.110.1
XOA Proxy involved is version 0.29.30

we have a backup task that is refusing to time out. the only way i can get this to end is by restarting the proxy

this has been an issue sense the new backup process was introduced.

timeout for the task is set to 8 hours but XOA and the XOA proxy seem to not care what this value is.. it just sits there and is keeping the VDI's locked.

dont get hyper focused on this next screen shot, it has nothing todo with the actual issue, just showing that even though the process is timed out but running.. the VID's are still locked by the host

Thanks

I have, what is hopefully a final update to this issue(s).

we upgraded to xoa version .99 a few weeks ago and the problem has now gone away. we suspect that some changes were made for timeouts in xoa that have resolved this , and a few other related problems.

@olivierlambert it has happened again

"XXXXX-nfs-02 (FAST all Flash Storage ZFS/NFS): failed to unpause tapdisk","Failed to unpause tapdisk for VDI ab867371-7a74-4a9f-88a9-9fb89e7dd3de, VMs using this tapdisk have lost access to the corresponding disk(s)","XXXXX XXXX-xcp Pool","Oct 23, 2025 8:18 PM",""

VM for this VDI had been running on host XXX-xcp-03 but there were attachments on the vdi from host XXX-xcp-07 .

we did not manually move this VM but it has moved.

I really think this problem is getting caused by load balancer moving things around while delta backups are running.

can we look into having XOA LB plugin NOT move VM's getting touched by the XOA backup processes?

@olivierlambert minor update... its now been 48+ hours and task is still in the same state, not timed out yet.

@olivierlambert the answer seems to be "no timeout" as of now the task has been stuck like this for 30+ hours. only way i can make it go away is to restart the XO Proxy. something i can't do right now due to another task running a huge delta mirror sync.

@DustinB thats not the point of the post.

the point of the post is on HTTP connection has timed out, the TASK never actually times out on the proxy.

I am aware it says space issues on the SR and its unfortunate that error is in the screen shot but its a distraction, not the actual problem.

XOA 5.110.1
XOA Proxy involved is version 0.29.30

we have a backup task that is refusing to time out. the only way i can get this to end is by restarting the proxy

this has been an issue sense the new backup process was introduced.

timeout for the task is set to 8 hours but XOA and the XOA proxy seem to not care what this value is.. it just sits there and is keeping the VDI's locked.

dont get hyper focused on this next screen shot, it has nothing todo with the actual issue, just showing that even though the process is timed out but running.. the VID's are still locked by the host

Thanks

@Pilow not easily. as an Telco/ISP we run massive multiple 40 to 100gb/s links across the country so data transfers from site to site is not an issue allowing us to have priv routed IP networks spanning anywhere. Much harder todo for SMB deployments.
may also involve adding more vnics and IP's to XOA and XOA-PROXY

SiteA:
XOA
proxy-A
Storage-A-SR
Backup-A-SR

priv network routing in both directions to:

SiteB:
proxy-B
Storage-B-SR
Backup-B-SR

so XOA, Proxy-A, Proxy-B ALL have access to:
Storage-A-SR
Backup-A-SR
Storage-B-SR
Backup-B-SR

sorry.. this is probably not helpfull for most people

@Pilow thats what we had todo..

we had to allow XOA AND ALL proxies access to ALL backup remotes.

@Pilow I think what you are seeing is the result of tapdisk single threaded nature (among other things master Dom0 related). I would suggest changing concurrency to 4 or 8 and see if the speed AT THE PORT is higher during backups. You may still only see <80MiB/s PER vm getting backed up, maybe less, but you may end with 4 or 8 backing up at 40 to 60MiB/s == high total bandwidth.

also note.. you 61.63MiB (note capital B for Mebibyte/s) is == 517.267 mb/s network speed megabite/second

@olivierlambert the 3 truenas servers are very bored we be lucking to be hitting 10% of there abilities even during the backups.

i think i have stumbled across something that maybe a bug or the cause:

lets say we have 5 servers, VM getting this error during backups is running on server3, When this happens we power it down on server3 and clear the stuck tapdisk connections a log entry shows up saying a snapshot link on server5 was also cleared....

example: "Cleared RW for 7d213264-1f89-488f-b273-eb91c186eaab on host 14c98969-ae12-4d7f-9c42-d7656aae01e5 on server5"
... but the VM was running on server3.....

so HERE is what i think is happening... just a theory... I think the load balance plugin is moving VMs around during the backup process. could that be possible?? i feel based on what i am seeing this could be the cause.

This problem is back with vengeance. after almost a year of no issues, we have upgraded to version 109.2 of XOA and boom... probably is back

XCP-NG servers are on 8.2.1 May 6th update

we get this following error in XOA

Error: SR_BACKEND_FAILURE_82(, Failed to snapshot VDI [opterr=failed to pause VDI 7d213264-1f89-488f-b273-eb91c186eaab], )

cant make snapshots of the effected VM anymore...
manual snapshots we get same error:
SR_BACKEND_FAILURE_82(, Failed to snapshot VDI [opterr=failed to pause VDI c51f22e2-ba5c-413b-8afc-9208a2b944ef],
but there is NO issues with the SR.

failed 17 backups last night,
failed 3 backups tonight.
across 3 different storage arrays

all those VM's we had to shut down and fix.
tapdisk connections to the disk have to be manually killed.

we will double check all our interfaces again but i suspect it will be the same as last time:, no errors on networking or storage side.

something has changed, reverted.