Troubleshooting "Exhausted available authentication methods" (KBA6009)

Issue

There are many different scenarios in which Delphix must establish an SSH connection to a host (adding a new environment, refreshing an existing environment, provisioning a VDB, taking a SnapSync for a dSource, et cetera). Sometimes these SSH connections will fail because they are rejected by the host Delphix attempts to connect to.

The error received by Delphix in response to the connection rejection is very generic. It simply states that it Exhausted available authentication methods and does not provide any additional information:

Error: Could not log into "snoopy.acme.com" with the provided username and password.

Error Code: exception.ccc.authenticate.failed

Suggested Action: Make sure that the host user and password are correct, and that the Delphix Connector is properly registered with the Delphix Engine.

Command Output: net.schmizz.sshj.userauth.UserAuthException: Exhausted available authentication methods

From the Delphix side, you can only see that the host ("snoopy.acme.com" for example) rejected the login attempt because it has Exhausted available authentication methods. The host will not give more information about why it rejected the login attempt because it does not want to give any helpful hints on what to try next in the event that someone is trying to hack into the host.

The Exhausted available authentication methods message can trigger for numerous reasons, such as:

• User account is locked
• Password is expired
• Incorrect permissions on the home directory of the user or public keys
• Home directory of the user has been moved

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
ALL	ALL

Resolution

1. Most often this is due to the proper permissions not being applied to the public key directory.  If you are using public-key encryption for logging into your Unix-based environment:

1. Select the Delphix operating system Environment user and choose Public Key for the Login Type.
2. Click View Public Key.
3. Copy the public key that is displayed, and append it to the end of your ~/.ssh/authorized_keys file. If this file does not exist, you will need to create it.   
   Note:  If  ~/.ssh/authorized_keys, ~/.ssh or ~ (the user's home directory) are writable by other users other than the Delphix Enviroment User, sshd will not allow public key authentication.  The following are examples of permissions which conform to the sshd allowable permission convention:  
   • Run chmod 600 ~/.ssh/authorized_keys         - assign read/write privileges for your user.
   • Run   chmod  700 ~/.ssh                                                           - assign read/write/execute privileges for your user.
   • Run chmod 755 ~                                                               - assign home directory writable only by your user.

The public key needs to be added only once per user and per environment.

The specific references to the location of .ssh and authorized_keys files assume a default Unix/Linux SSH configuration. It is possible that the SSH server configuration is customized and will source authorized_keys from another location.

It is also possible to restrict SSH public key authentication by IP address, if a from="<ip address/netmask>" is prepended to the public key using CIDR address/masklen notation. This will cause SSH authentication to fail if the Engine IP address changes.

If you are not using public-key encryption (or the above is confirmed to be correct), it will be necessary to review the security files of the target host to determine the reason the host rejected the login attempt. Please review the troubleshooting section for further information.

 

---

Troubleshooting

 

1. Review the security logs of the host that is rejecting the login attempt:

Assuming a default SSHD configuration, Linux logs the errors in /var/log/secure.  Examples of various actionable 

authentication errors that may be found include, but are not limited to:

Oct 14 12:06:53 rh610-base sshd[15773]: Authentication refused: bad ownership or modes for directory /home/oracle/.ssh

Oct 14 12:07:40 rh610-base sshd[15801]: Authentication refused: bad ownership or modes for file /home/oracle/.ssh/authorized_keys

Oct 14 12:05:24 rh610-base sshd[5063]: Authentication refused: bad ownership or modes for directory /home/oracle

• On HP-UX 11.23 or higher with SSH 5.0, SSHD daemon logging of failed logins logs successfully to two files:
  • /var/adm/syslog/syslog.log
  • /var/adm/btmps
• AIX - Please refer to IBM's documentation
• Solaris - Please refer to Oracle's documentation

2. In the instance that /var/log/secure output includes boks_sshd rather than sshd, such as:

Nov 20 01:09:10 exampleHost boks_sshd[21250]: pam_unix(boks_sshd:session): session closed for user delphix     
Nov 20 01:11:29 exampleHost boks_sshd[26021]: Accepted publickey for delphix from <ip address> port 57146 ssh2     
Nov 20 01:11:29 exampleHost boks_sshd[26021]: pam_unix(boks_sshd:session): session opened for user delphix by (uid=0)     
Nov 20 01:11:29 exampleHost boks_sshd[26027]: Received disconnect from <ip address>: 11:

This indicates that Core Privilege Access Manager (BoKS), a third-party security layer, is applied in the Environment. This may affect the location of the applicable authorized_keys file, or frequently is used to restrict access by IP address. BoKS is also used in sudo privilege elevation replacement, so if a non-standard sudo configuration exists, it is more likely that BoKS may be involved in SSH access restriction.

BoKS can be configured to restrict by hostname or IP address. Restrictions configured by IP address will naturally break an existing configuration, if the Engine IP address changes for any reason, so it is strongly recommended to configure these restrictions by Engine hostname instead.  These rules are referred to as "access routes" in BoKS terminology.

3. It is also possible for a user with administrative privileges to increase the level of debug for sshd, either by restarting the existing daemon with additional debug enabled, or by starting a new instance of sshd listening on another open port (it is not possible to run two distinct sshd services listening on the same default port (22)).

4. Failure to login or refresh an Environment with the error message discussed here will confirm that the Engine is reaching a remote host; there is not likely to be any network routing or firewall concern. However, it is possible that some configuration change affects hostname resolution, so in the instance there is no reference to the Engine's attempted logins in any system log, the IP address of the host should be verified, and if hostname is being used in Delphix, it may be prudent to attempt changing the Host Address in the Delphix Environment configuration page to use IP address temporarily for testing.

 

 

---