Skip to main content
Delphix

KBA1202 VDB provision failures on Linux due to rpc.statd error

Applicable Delphix Versions

All Delphix Engine versions.

Issue

Enabling a virtual database (VDB) on an affected target will fail with an error suggestive of incorrect sudo privileges:

Error
Failed to mount "Vdaisy_56B" because the mount command executed as user "sybase" on host "nstackrh69.delphix.com" failed for mount point "/home/sybase/toolkit/564d21b7-5756-10a6-694b-5bdb5f806c53-vdb-28/datafile".

Error Code
exception.db.generictarget.mount.failed

Suggested Action
Make sure the user has permission to mount file systems on the host and that NFS traffic from the host can reach the Delphix Engine.

The detailed portion of the error message showing the "Command Output" shows:

mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

mount: only root can do that
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
mount.sh: MOUNT_FAILED

Similarly, the debug log file from the Delphix Engine shows:

[2017-10-13 13:14:12,963][INFO][jcm.impl.JobInstance#execute:352][Worker-4161|JOB-2468|DB_PROVISION(ASE_DB_CONTAINER-153)][] job failed with user error
com.delphix.appliance.server.dco.events.enums.DbGenerictargetExceptions$MountFailed: exception.db.generictarget.mount.failed {Vdaisy_56B|sybase|nstackrh69.dcenter.delphix.com|/home/sybase/toolkit/564d21b7-5756-10a6-694b-5bdb5f806c53-vdb-28/datafile|mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

mount: only root can do that
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
mount.sh: MOUNT_FAILED
}

Troubleshooting

Four disparate scenarios have been identified as causing this error.

Scenario 1

Usage of the hosts access control files "/etc/hosts.allow" and "/etc/hosts.deny" can prevent the mount command from succeeding because rpcbind is disallowed from making connections to the "localhost" address. Errors similar to the following can be seen in "/var/log/messages":

10:34:04 nstackrh69 kernel: RPC: server rpc.statd requires stronger authentication.
10:34:04 nstackrh69 rpcbind[9783]: connect from 127.0.0.1 to getport/addr(status): request from unauthorized host

Checking the two files, strict access is configured:

$ cat hosts.deny
#
# hosts.deny    This file contains access rules which are used to
#        deny connections to network services that either use
#        the tcp_wrappers library or that have been
#        started through a tcp_wrappers-enabled xinetd.
#
ALL:ALL

$ cat hosts.allow
#
# hosts.allow This file contains access rules which are used to
# allow or deny connections to network services that
# either use the tcp_wrappers library or that have been
# started through a tcp_wrappers-enabled xinetd.
#
#
ALL: 10.0.0.0/255.0.0.0
Scenario 2

This is caused by a bug in the statd startup script included within a range of Oracle Enterprise Linux software releases (Oracle Enterprise Linux 6.0 - 6.8 inclusive). The PATH environment variable in /usr/sbin/start-statd does not include the path for command cat. The detailed error message/logging from Delphix should include an additional error about being unable to find the "cat" command:

/usr/sbin/start-statd: line 10: cat: command not found;mount.nfs: 
rpc.statd is not running but is required for remote locking.

Review the "/usr/sbin/start-statd" script to confirm "/bin" is included in the PATH.

Scenario 3

Incorrect permissions on /etc/hosts.allow and /etc/hosts.deny can cause this error.

Run "ls -la /etc/hosts.*" to make sure that the user performing the mount operation has read access to these files.

Scenario 4

The rpcbind service is not running. Run the Linux "service" command as follows to see if the rpcbind service is running:

# service rpcbind status
rpcbind is stopped

Resolution

Scenario 1

Grant rpcbind permission to the localhost address by adding the following line to "/etc/hosts":

# cat hosts.allow
#
# hosts.allow    This file contains access rules which are used to
#        allow or deny connections to network services that
#        either use the tcp_wrappers library or that have been
#        started through a tcp_wrappers-enabled xinetd.
#
ALL: 10.0.0.0/255.0.0.0
rpcbind: 127.0.0.1
Scenario 2

Add /bin as a path to script /usr/sbin/start-statd

e.g.

PATH=/bin:/sbin:/usr/sbin

# cp /usr/sbin/start-statd /usr/sbin/start-statd.orig
# diff /usr/sbin/start-statd /usr/sbin/start-statd.orig
7c7
< PATH=/bin:/sbin:/usr/sbin
---
> PATH=/sbin:/usr/sbin
Scenario 3

Grant read permission to the appropriate user and groups. Normally these files have permission 644:

# chmod 644 /etc/hosts.allow
# chmod 644 /etc/hosts.deny
Scenario 4

Start the rpcbind service from an account with root permission:

# service rpcbind status
rpcbind is stopped
# service rpcbind start
Starting rpcbind:                                          [  OK  ]