[solved] NFS on XO failing, on Center has no issues

mavoff

Hi there,

I am having an issue where I can't add NFS remotes to XCP-ng (in order to perform backups).

I created a new share exactly alike the others, and proceeded to add it to XO. However, it fails with the following error:

Command failed with exit code 32: mount -o vers=3 -t nfs 10.49.10.1:/export/backups /run/xo-server/mounts/ae51465b-b3ad-4f79-91e7-8705061aa000 mount: /run/xo-server/mounts/ae51465b-b3ad-4f79-91e7-8705061aa000: bad option; for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount.<type> helper program.

However if I go to XCP-ng Center I just select the pool, "New SR" and its straightforward with no errors.

Running xe sr-probe type=nfs device-config:server=10.49.10.1

# xe sr-probe type=nfs device-config:server=10.49.10.1
Error code: SR_BACKEND_FAILURE_101
Error parameters: , The request is missing the serverpath parameter, <?xml version="1.0" ?>
<nfs-exports>
        ...
        <Export>
                <Target>10.49.10.1</Target>
                <Path>/export/vdisks/.issc-vdisks-backup_202010280342</Path>
                <Accesslist>10.49.10.0/24</Accesslist>
        </Export>
        ...
        <Export>
                <Target>10.49.10.1</Target>
                <Path>/export/iso</Path>
                <Accesslist>10.49.10.0/24</Accesslist>
        </Export>
        <Export>
                <Target>10.49.10.1</Target>
                <Path>/export/backups</Path>
                <Accesslist>10.49.10.0/24</Accesslist>
        </Export>
        <Export>
                <Target>10.49.10.1</Target>
                <Path>/export/vdisks</Path>
                <Accesslist>10.49.10.0/24</Accesslist>
        </Export>
        <Export>
                <Target>10.49.10.1</Target>
                <Path>/export/vdisks/.issc-vdisks-backup_202010290342</Path>
                <Accesslist>10.49.10.0/24</Accesslist>
        </Export>
</nfs-exports>

Can someone point me how to fix this? And could it have something to do that the XO does not have an IP address on the 10.49.10.0/24 sublet? because, XCP-ng Hosts do have an IP in this network, no other VM's do. But I assume as long as the XCP-ng hypervisor has a connection, that's what matters.
Thank you.

Danp

There seem to be a bunch of NFS issues posted recently.

I assume that you are running XO from the sources. Do you have the nfs-common package installed on your XO VM?

Have you tried with XOA?

mavoff

Hi @Danp thank you very much for your reply.

Its XO from sources yes, I used XOA for a while but its a bit limited and I can't afford its license, so... took this option. Also is always good to learn and get to know the software better a bit I think. So, I didn't have the nfs-common installed, I installed it, afterwards rebooted the XO vm and upon it coming back up tried to add the remote again, it worked fine.

EDIT:

It did not. It seemed so, but after a while (60 secs maybe) I got an error notice and on the log it shows:

Command failed with exit code 32: mount -o vers=3 -t nfs 10.49.10.1:/export/backups /run/xo-server/mounts/f2e5b3d3-60c6-4baf-94b7-36104beb77bc
mount.nfs: Connection timed out

olivierlambert

Remote are connected to XO, not XCP-ng. Your XO must have access to the NFS remote.

mavoff

@olivierlambert thank you for your reply.

I have a question about this tho. I remember when I changed the storage from local to NFS iirc I added the shares via the same method using remotes (XOA not XO back then) for the storage pools. XOA also didn't have an interface directly facing the NFS server but it worked without this issues.

I have now added another network interface directly to the XO VM.
However, some issues I see from the start:

I can't choose MTU manually. Our network has a requirement of 1400 MTU. When I add the interface, I can't select the MTU value. The corresponding field is not editable.
The XO does not reflect the correct MTU for a given interface. See example below.

Screenshot 2020-11-03 at 14.55.55.png

xo@xo:~$ ip ad
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc mq state UP group default qlen 1000
    link/ether 36:8b:c6:73:e2:4a brd ff:ff:ff:ff:ff:ff
    inet 10.49.13.1/24 brd 10.49.13.255 scope global dynamic eth0
       valid_lft 7131sec preferred_lft 7131sec
    inet6 fe80::348b:c6ff:fe73:e24a/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc mq state UP group default qlen 1000
    link/ether ae:13:4c:1d:ca:66 brd ff:ff:ff:ff:ff:ff
    inet 10.49.10.2/24 brd 10.49.10.255 scope global dynamic eth1
       valid_lft 7133sec preferred_lft 7133sec
    inet6 fe80::ac13:4cff:fe1d:ca66/64 scope link 
       valid_lft forever preferred_lft forever

olivierlambert

Don't mix a SR and a remote.

A SR is connected directly to your host, a remote is NOT. If XOA can't connect to the remote, you can't make a backup there.

About the MTU: this must be set on the network, and any VIF on this network with use this MTU.

mavoff

Lol sometimes I feel dumb and really appreciate your effort in answering to all.
I changed the vlan recently and when added the network it got the wrong MTU indeed. As I have a dhcp option to force MTU of 1400 on the router, all works and I didn't notice it.

mavoff

Ok... back again! lol

So here's where we're at:

I added a new interface to XO within the storage network to have direct access to the nfs share. As you can see its up.

I proceeded going to Remotes and trying to add this new Remote. I get an immediate error:

remote.test
{
  "id": "8039a311-60d3-4fba-a9b5-46cbedc60edd"
}
{
  "errno": -13,
  "code": "EACCES",
  "syscall": "open",
  "path": "/run/xo-server/mounts/8039a311-60d3-4fba-a9b5-46cbedc60edd/.keeper_3p3xb34zzxq",
  "message": "EACCES: permission denied, open '/run/xo-server/mounts/8039a311-60d3-4fba-a9b5-46cbedc60edd/.keeper_3p3xb34zzxq'",
  "name": "Error",
  "stack": "Error: EACCES: permission denied, open '/run/xo-server/mounts/8039a311-60d3-4fba-a9b5-46cbedc60edd/.keeper_3p3xb34zzxq'"
}

Other outputs:

root@xo:~# rpcinfo -p 10.49.10.1
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  20048  mountd
    100024    1   udp  47049  status
    100005    1   tcp  20048  mountd
    100024    1   tcp  38617  status
    100005    2   udp  20048  mountd
    100005    2   tcp  20048  mountd
    100005    3   udp  20048  mountd
    100005    3   tcp  20048  mountd
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049
    100003    3   udp   2049  nfs
    100227    3   udp   2049
    100021    1   udp  51923  nlockmgr
    100021    3   udp  51923  nlockmgr
    100021    4   udp  51923  nlockmgr
    100021    1   tcp  33019  nlockmgr
    100021    3   tcp  33019  nlockmgr
    100021    4   tcp  33019  nlockmgr

root@xo:~# showmount -e 10.49.10.1
Export list for 10.49.10.1:
/export/vdisks/.issc-vdisks-backup_202010250342 10.49.10.0/24
/export/vdisks/.issc-vdisks-backup_202010260342 10.49.10.0/24
/export/vdisks/.issc-vdisks-backup_202010270342 10.49.10.0/24
/export/vdisks/.issc-vdisks-backup_202010280342 10.49.10.0/24
/export/vdisks/.issc-vdisks-backup_202011020342 10.49.10.0/24
/export/vdisks/.issc-vdisks-backup_202011010342 10.49.10.0/24
/export/iso                                     10.49.10.0/24
/export/backups                                 10.49.10.0/24
/export/vdisks                                  10.49.10.0/24
/export/vdisks/.issc-vdisks-backup_202010290342 10.49.10.0/24

olivierlambert

I have no idea how you installed XO (from the sources right?), and it looks like permissions issues, not a problem directly related to XO itself

mavoff

This XO has been built from sources on a debian VM according the documentation it worked flawlessly. (you know I spent several hours trying to build on FreeBSD, didn't work, debian worked fine).

This Remote / NFS share is configured alike the others. Inclusive the same user and permissions. I can mount the share on XCP-ng host without issues as well as using the XCP-ng Center. If it works everywhere else except on XO, it isn't an XO problem? How come?

olivierlambert

As I already said, that's because a SR is NOT a remote, it's not the same thing. It's not used the same way, it's just 2 different things.

So that's why it's not the same behavior. You can have a permission issue on the same NFS share used as a remote, that you don't have if you use it as a SR. Not the same thing.

mavoff

So exactly what do I have to consider to add the remote to XO? Is there a specific user I should configure access to the NFS share?

I don't find these processes minimally straightforward. Not looking to put 777 permissions on the share.

olivierlambert

You need to check that your current XO user for xo-server has the NFS permissions to write on the targeted folder, even for . files.

Check your NFS permissions.

mavoff

hmm... well, so xo-server is running under root as it was the option that would allow to use the standard ports.

Does this mean I have to assign access to user root to this share?

btw I have another question... is there a way of getting plugins to the XO from source? thank you

olivierlambert

This is NFS permission question, here is a good read about it: http://www.troubleshooters.com/linux/nfs.htm

mavoff

I am going to read that now.

Anyway, please allow me to share this thought:

I tried with two users: the root user and the user I use to login to XO.
None worked both returned the same error.

Many times I have expressed some unease with XO, I am requested examples and to share.
Well, this is one of those situations where I'm definitely hating XO and craving for Center.
I never had this sort of issues with Center. You add storage from various sources and it works. I am wasting hours to "fix" something that isn't broken (I can easily mount this share on CLI:

root@xo:~# mount -t nfs -o rw 10.49.10.1:/export/backups /mnt
root@xo:~# mount -l | grep backups
10.49.10.1:/export/backups on /mnt type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.49.10.2,local_lock=none,addr=10.49.10.1)

EDIT: wrong NFS version above.

root@xo:~# umount /mnt
root@xo:~# mount -t nfs -o rw -o vers=3 10.49.10.1:/export/backups /mnt
root@xo:~# mount -l | grep backups
10.49.10.1:/export/backups on /mnt type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.49.10.1,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.49.10.1)

So I'll look to the link you provided, and try to figure out why xo-server can't write to a folder that itself its supposed to create. This should be handled by the orchestra software itself... imo... thanks.

olivierlambert

Okay so I'm patient and explaining this again. You are comparing 2 different things. You can't make backups with XCP-ng Center, so there's no remote concept or whatever in there. If you crave for it, use it if you like, but you won't have any backup possible. Different things.

If you create an SR with Xen Orchestra or XCP-ng Center, it's exactly the same XAPI call, you won't have any issue. So if you compare apples to apples, it works the same way

Here, you are working with something else: being able to mount NFS share from XO (something you don't do at all with XCP-ng Center). If you have high expectation on something that you can have working out of the box, you have 2 choices:

Learning how thing are working with the version from the sources. In your case, it means understand how NFS permissions are working. I have to emphasis on the fact XO is using mount command, so it's relatively standard. However, if you are disappointed because it doesn't work out-of-the-box, be advised we can't do the NFS server permission configuration for you, because it's not Xen Orchestra related, but NFS related on your server. XO can't configure your NFS server, this is something beyond our scope. It's like you are complaining about the road quality to your car dealer.
If you don't want to spend time to figure it out (and I can fully understand that), we also offer pro support and configuration with remote assistance.

mavoff

@olivierlambert first of all please let me appreciate your time and effort of trying to explain all so we understand it correctly.

I got the first part. XO is different from Center and the Remote concept doesn't apply there.

Following point 1: that's where I'm at, trying to understand how to work with XO from sources... However, notice this: my issue isn't with mount. My issue is with XO not being able to read on a location itself created, after trying with all possible users that would fit for purpose:

EACCES: permission denied, open '/run/xo-server/mounts/cb8db284-3d88-42fc-9f43-181979914b20/.keeper_i109kg76ya'

So my question is, the folder /run/xo-server/mounts/etc should be read by who? The issue isn't the mount command itself it seems to me, but reading permissions on a folder created by XO/mount.

The xo-server config has useSudo=true - tried with and without - same exact error. I have configured the NFS share for the user root and for the admin user of XO. none work.
Sorry if I'm being a bit hard on getting this, I configure the users correctly for NFS for a bunch of different stuff and I don't come across these errors. I would like to learn and understand how NFS permissions are working HERE.
I'm sure if I'd reduce the security standards this would work. And I am also sure many of those adept of 777 permissions also would have marked this solved. But that isn't "understanding how NFS permissions work (here)". I truly would love to understand what user is that that would have permission, if root doesn't work, the admin user of XO doesn't work, useSudo=true or false is the same.

Biggen

Have you actually tried to mount the NFS share from another machine and see if you can read/write to it? This sounds like an NFS permissions issue.

With NFS, you don't usually have a username/password. You can restrict access, if you wish, via IP or other measures. I have a Synology NAS unit configured as a NFS Remote for XO to send backups to. I don't specify a user/password for the share. I just feed XO the path to the share and let it rip.

mavoff

I'm not expecting any user/passwd, NFS works with User ID mapping ...

From redhat knowledge base:

nfsd bases its access control to files on the server machine on the uid and gid provided in each NFS RPC request. The normal behavior a user would expect is that she can access her files on the server just as she would on a normal file system. This requires that the same uids and gids are used on the client and the server machine. This is not always true, nor is it always desirable.

When a uid or gid on the nfs client does not exist on the nfs server, nfsd by default "squashes" the non-existent uid/gid. This means that for uids and gids that do not exist on the nfs server, exportfs chooses a uid and gid of 65534 for squashed access, which corresponds to user and group nfsnobody.

In summary, client uids/gids are used if they exist on the server. If they do not exist, they get squashed to nfsnobody. This is the default behavior.