Backing up the backup

mike

Hey peeps,

We've been backing up our VM estate via XO to various TrueNAS based NFS targets for quite some time now to great success.

However we're currently bolstering our backup resiliance by using Arcserve to backup as much as we can to a custom appliance and ultimately to tape, and we'd like to backup the XO backup locations this way too. Unfortunately while we can point the Arcserve agent at the NFS share and it can see everything inside, it's unable to open any files due to file locks, regardless of whether there's an active XO backup running at the time or not.

We've tried applying the nolock option to the mount but it hasn't made any difference (I may have mis-understood what that option actually does).

So what would be the recommended way for Arcserve to access these files, in a strictly read-only fashion of course?

Options we've considered but not yet tried:

A separate NFS share pointing to the same place. Can't see it making any difference, as I believe the locks are at the server filesystem level?
Using an SMB share for Arcserve. Just feels risky somehow and not sure it would work for the same reason as above.
Backing up from a TrueNAS snapshot instead of the main share itself. But snapshots must have the time and date in their name and there's no "latest" symlink so I don't know how we could make that work.

Any thoughts?

olivierlambert

Hello there,

Hmm question for @fohdeesha maybe

fohdeesha

@mike hmm, NFS locking is handled by statd so make sure that is running properly on your NFS server.

Also, this may be related to the .nfsXXXX files that are generated/used by the system to keep track of open files that should later be deleted when closed. These things:

root@zfs[/mnt/chunker/xen-backup/XH-01]# ls -lh
total 18
-rw-r--r--   1 xen  wheel     0B Oct 19 05:24 .nfs000000000000020300000002
drwx------   3 xen  wheel     3B Jun 29 05:59 xo-config-backups
drwx------   3 xen  wheel     3B Jun 29 05:59 xo-pool-metadata-backups
drwxr-xr-x  15 xen  wheel    15B Sep 26 09:39 xo-vm-backups

In your backup of the backup job, can you have it exclude any .nfs* files so it doesn't try to access or copy them, and see if the locking errors go away?

mike

@fohdeesha Thanks for your reply and apologies for the delay in getting back to you.

You've provided some good avenues to check, most notable right now is that we have lock files in that share going back to July! I assume it's safe to delete these and/or restart the statd and lockd services to see if we can clear things up...

mike

Doing a bit more hunting it seems like if we could get Arcserve to use the nolock option then that would likely solve the problem, but there's nowhere to add any options and the Arcserve Proxy itself uses the Windows NFS client behind the scenes.

Right now we've worked around the problem by creating a separate, read-only SMB share pointing to the same location and using that for Arcserve instead. Our only concern now is what could happen if XO starts a job while Arcserve is still doing its own backup.

Edit: I did a bit more Googling after posting the above and came across this thread: https://www.truenas.com/community/threads/nfs-the-process-cannot-access-the-file-because-another-process-has-locked.12644/ ... which suggests that it's actually a long-standing TrueNAS bug.

Buried within the comments is this suggestion:

If you're having trouble with the NFS locking mechanism, and you would like to use the "NET USE" command, or the [WNetAddConnection] function to mount a NFS share without locking, there's an undocumented registry entry hidden in the NFSNP.DLL:

"HKLM\Software\Microsoft\Client for NFS\CurrentVersion\Users\Default\Mount" REG_DWORD "Locking"
If the value is "0x0" then locking will be disabled globally.
If the value is "0x1" then locking will ben enabled globally.

We'll test it and report back.