Samba Timeout - How to configure timeout limit / possible issues ?

HolgiB

Hey there,
we are using XO to backup VMs from both XCP-ng and Citrix Xen-Server. We use a DIY FreeNAS instance on a beefy server with external SAS HDD enclosure via SMB as remote. Our Xen Orchestra is build from source via install script on a Ubuntu 22.04 server. In the recent time we have seen backup jobs fail with what seems to be some sort of Samba timeout issue. Typically the kernel log of the XO host show something like this:

[1508788.516400] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[1509688.124079] CIFS: Attempting to mount //172.25.20.29\xenexports/xohn
[1509688.124115] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[1510588.329807] CIFS: Attempting to mount //172.25.20.29\xenexports/xohn
[1510588.329848] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[1516245.125756] CIFS VFS: \\172.25.20.29 sends on sock 000000009e27dde1 stuck for 15 seconds
[1516245.125805] CIFS VFS: \\172.25.20.29 Error -11 sending data on socket to server
[1526634.554814] CIFS VFS: \\172.25.20.29 sends on sock 00000000ada8aee7 stuck for 15 seconds
[1526634.554870] CIFS VFS: \\172.25.20.29 Error -11 sending data on socket to server
[1526698.938970] CIFS VFS: \\172.25.20.29 Send error in SessSetup = -11
[1529188.866336] CIFS VFS: \\172.25.20.29 has not responded in 180 seconds. Reconnecting...
[1548386.401279] CIFS: Attempting to mount //172.25.20.29\xenexports/xohn
[1548386.401306] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[1569985.548675] CIFS: Attempting to mount //172.25.20.29\xenexports/xohn
[1569985.548702] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[2027171.988655] CIFS: Attempting to mount //172.25.20.29\xenexports/xohn
[2027171.988682] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[2028071.838874] CIFS: Attempting to mount //172.25.20.29\xenexports/xohn
[2028071.838898] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[2028971.872043] CIFS: Attempting to mount //172.25.20.29\xenexports/xohn
[2028971.872077] No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3 (or SMB2.1) specify vers=1.0 on mount.
[2041130.424668] CIFS VFS: \\172.25.20.29 sends on sock 000000002ff9cdef stuck for 15 seconds
[2041130.424727] CIFS VFS: \\172.25.20.29 Error -11 sending data on socket to server

The strange thing here though: The FreeNAS used for storage is connected via 2x10 GBit SFP bond to our main switch. All Xen servers which are accessed via XO also run a 10 GBit NIC as Management interface.

I tried (as experiment) to set a timeout for the SMB remote in XO via -o option.
This is specified in te samba docs:
-t|--timeout <timeout-seconds>
This allows the user to tune the default timeout used for each SMB request. The default setting is 20 seconds. Increase it if requests to the server sometimes time out. This can happen when SMB3 encryption is selected and smbclient is overwhelming the server with requests. This can also be set using the timeout command inside smbclient.

But unfortunately XO simply replies with an error message "CIFS: Unknown mount option "--timeout 60""
I guess this is because we are talking about CIFS and not a plain mount command with Samba options ?

My main questions here:

Has anyone doing full VM backups from snapshots to a SMB share experienced similar error messages ?
Is there an approach to modify the default SMB timeout which seems to be 15 seconds ?

TIA,
Holger

olivierlambert

Hi,

Since I have no idea on which 3rd party install script you are using, it's hard to assist. I would try with XOA first, and see if the mount works correctly with your options

Also, I think it's a bit weird that you have those stuck transfer Have you also tried in NFS?

HolgiB

@olivierlambert Thanks for your quick reply.
Fair enough...I didn´t try XO as appliance in the first place.

There is this one XO install script from this guy called Dustin B: (https://github.com/Jarli01)
https://github.com/Jarli01/xenorchestra_updater

Has worked flawlessly so far and from what I tell so far XO is pretty similar to what you get when following the XO from the sources guide.
As for the storage protocol used: Would you suggest NFS over SMB ?
We do no fancy incremental backups. Only full backups without RAM from snapshots.

Thanks,

olivierlambert

I can't follow what's done in 3rd party scripts, and it might be not trivial to find problems with them (if it's a problem due to the script or not). But before pointing any finger (I don't like that ) I would prefer to check with a fully up to date XOA on latest release channel and see if you have the same issue. At least, we'll be able to compare the behavior