Skip to main content
Delphix

Testing FTP and SFTP Masking Connectors (KBA1803)

 

 

At a Glance  

Applicable to: Applicable to File Masking using (S)FTP Connection in all Delphix Masking versions.
This document is based on 5.3.2.0 and updated with details in 5.3.5.0.
Errors: This KBA covers: 
  • Connection refused.
  • Invalid Hostname.
  • Invalid Port.
  • Incorrect Credentials.
  • Invalid Folder.
  • File Access Permission.
  • No Matching Host Key Type Found.
Tools: Issue investigation tools: 
  • Masking Engine (investigation of logs).
  • Command-Line FTP and SFTP Clients.
Delphix Supported Protocols: 
Protocol ME Comment
FTP  Yes * Supported by all platforms (Windows/UNIX/etc)
* Security issues.
SFTP Yes * SSH FTP (supported by Unix-Like OSs).
* Windows OpenSSH.
* Windows 3rd party SFTP server.
FTPS (FTP over SSL) No * Supported as of Version 16.0.0.0
FTP over SSH No * Not supported by Masking Engine.
Simple FTP No * Not supported by Masking Engine.

Connection Error

When testing the connection in the Connector popup in the GUI there are two responses: 

  • Success - Connection Succeeded!
    If you still have issues though test is successful, please look at File Access
  • Error - Connection Failed. Please check Connection Parameters. 
    This page is for you!

Masking UI - File Connector Test.png

Troubleshooting

This KBA goes through two methods for troubleshooting: 

  1. Using Masking GUI and Logs.
  2. Using (S)FTP Client. 

Troubleshooting using Masking GUI and Logs

Troubleshooting using the GUI means that we need to access the logs to investigate the issue. 

To access the logs go to Admin > Logs.
The most recent logs are at the bottom of the screen, scroll down to access.

Alternatively, use Export to export the logs to a text file.

 

tip

Tip:

Help reading the logs
The logs will have what is called stack trace with 'at...such.and.such' - ignore these, they are for code debugging.


Masking UI - Admin Logs.png

Troubleshooting using (S)FTP Client 

Rather than testing using the Masking UI and looking at the logs, it might be better to test the (S)FTP connection outside the masking engine. One benefit with this method is that it will enable you to look at the connection details on the screen and test them at the same time.

Recommended tests: 

The recommended tests and examples below.

  1. Check if the host is reachable:
    • Ping server with the FTP service.
    • Detail needed: [hostname]
  2. Check connection and authentication:
    • Connect to the FTP service using FTP client
    • Details needed: [hostname], [user], and [password]
  3. Check that the folder with the files to mask is accessible:
    • CD to the folder with the files:
    • Detail needed: [folder].
  4. Check that files can be read:
    • Get a file from the FTP server.
    • Detail needed: [file] (a small test file):
  5. Check that files can be written. 
    • Put a file to the FTP server.
    • Detail needed: [file] (a small test file).
  6. Quit.

Connection Examples 

The two examples use the command-line programs called SFTP on macOS and FTP on Windows.

Command-line: SFTP (Mac)
Ref Example command
1

2



3

4


5


6
$ ping [hostname]

$ sftp [user]@[hostname] 22
Password:[password]
Connected to [hostname].

sftp> cd [folder]

sftp> get [file]
Fetching [foler]/[file] to [file]

sftp> put [file]​
Uploading [file] to [folder]

sftp> quit
Command-line: FTP (Windows)
Ref Example command
1

2







3


4





5





6
C:\> ping [hostname]

C:\> ftp [hostname]
Connected to [hostname].
220 (vsFTPd 3.0.3)
User ([hostname]:(none)): [user]
331 Please specify the password.
Password: [password]
230 Login successful.

ftp> cd [folder]
250 Directory successfully changed.

ftp> get [file]
200 PORT command successful. Consider using PASV.
150 Opening BINARY mode data connection for [file] (8 bytes).
226 Transfer complete.
ftp: 8 bytes received in ...

ftp> put [file]
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 Transfer complete.
ftp: 8 bytes sent in ...

ftp> quit

Errors in Logs and Tests

Detailed below are errors from failed connections. They are used as examples to assist with troubleshooting, check the error message in the logs or the FTP client. 

 

Note

Note:

These errors have been generated by creating as similar connections as possible. There are differences in the return code/message due to type of protocol and type of connection. 

 

Successful Connector Test

A successful connector test looks like this (note - in later 5.3 versions it is very verbose): 

2019... INFO  com.dmsuite.common.utils.VFSUtility - User delphix, Mode sftp: Testing on dc3.delphix.com:22//tmp/
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Connecting to dc3.delphix.com port 22
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Connection established
...
### SFTP LOG ENTRIES
...
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: publickey,keyboard-interactive,password
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: publickey
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: password
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: password
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Authentication succeeded (password).
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Disconnecting from dc3.delphix.com port 22

Issue: Connection Refused/Closed

The service is not accepting the connection. 

Cause and resolution 

The FTP/SFTP service is not correctly configured, available or enabled. This error is seen when a connection is made to a server that does not have this service (for example connecting SFTP to a Windows box without 3rd party software installed). 

Troubleshooting using Masking GUI and Logs

Trying to access incorrectly configured (disabled) service. 

2018... ERROR c.u.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://[hostname]/".
...
Caused by: ...FileSystemException: Could not connect to SFTP server at "[hostname]".
...
Caused by: ...JSchException: Auth fail

Trying to access a host which does not have ssh installed. 

2018... ERROR c.u.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://[hostname]/".
Caused by: ...FileSystemException: Could not connect to SFTP server at "[hostname]".
...
Caused by: ...JSchException: java.net.ConnectException: Connection refused (Connection refused)
...
Caused by: ...ConnectException: Connection refused (Connection refused)

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

Trying to access incorrectly configured (disabled) service. This is likely producing the same result if the connection is blocked by a Firewall. 

$ sftp user@hostname
subsystem request failed on channel 0
Connection closed

Trying to access a host which does not have ssh installed. 

$ sftp user@hostname
ssh: connect to host [hostname] port 22: Connection refused
Connection closed
Command-line: FTP (Windows)

Trying to access an incorrectly configured FTP service. 

C:\>ftp hostname
> ftp: connect :Connection refused

Issue: Invalid Hostname

When the server name (hostname) is incorrect the following error messages will be returned. 

Troubleshooting using Masking GUI and Logs

Logs when the hostname is invalid.

2018... INFO, .sftp.SftpClientFactory - Connecting to inval_host port 22
2018... ERROR c.u.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://inval_host:22/".
...
Caused by: ...FileSystemException: Could not connect to SFTP server at "inval_host".
...
Caused by: ...JSchException: java.net.UnknownHostException: inval_host

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

Trying to access an invalid host (inval_host).

$ sftp user@inval_host
ssh: Could not resolve hostname inval_host: nodename nor servname provided, or not known
Connection closed
Command-line: FTP (Windows)

Trying to access an invalid host (inval_host).

C:\>ftp inval_host
Unknown host inval_host.
ftp>

Issue: No Matching Host Key Type

This error will happen if there are no matching host keys between the server-client (SFTP Server - Masking Engine).

Troubleshooting using Masking GUI and Logs

Logs are not showing a lot here. The stack trace is providing some hints.

2019... ERROR - XML Input - Unexpected error
2019... ERROR - XML Input - j.i.UncheckedIOException: o.a.c.vfs.FileSystemException: Could not connect to SFTP server at "sftp://***@hostname/".
...
2019... Caused by: org.apache.commons.vfs.FileSystemException: Could not connect to SFTP server at "sftp://***@hostname/".
2019...   at org.apache.commons.vfs.provider.sftp.SftpFileSystem.getChannel(Unknown Source)
2019...   at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(Unknown Source)
...
2019... Caused by: com.jcraft.jsch.JSchException: java.io.IOException: inputstream is closed
2019...   at com.jcraft.jsch.ChannelSftp.start(ChannelSftp.java:315)
2019...   at com.jcraft.jsch.Channel.connect(Channel.java:152)
2019...   at com.jcraft.jsch.Channel.connect(Channel.java:145)
2019...   ... 12 more
2019... Caused by: java.io.IOException: inputstream is closed
2019...   at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2911)
2019...   at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935)
2019...   at com.jcraft.jsch.ChannelSftp.start(ChannelSftp.java:262)
2019...   ... 14 more

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

Trying to access an invalid host (inval_host).

$ sftp -vvv user@hostname
Unable to negotiate with 192.168.5.145 port 22: no matching host key type found. Their offer: ssh-dss
Connection closed

To turn on verbose logging '-vvv' (debug level 1,2,3).

$ sftp -vvv user@hostname

This issue is hard to resolve. If needed, please seek assistance from Delphix Support. 

Issue: Invalid Port

If the port number is incorrect the following error messages are seen.

Troubleshooting using Masking GUI and Logs

Connection using an incorrect port number (here 222).

2018... INFO, .sftp.SftpClientFactory - Connecting to hostname port 222
2018... ERROR common.utils.VFSUtility - Error ::
...FileSystemException: Could not connect to SFTP server at "sftp://hostname:222/".
...
Caused by: ...FileSystemException: Could not connect to SFTP server at "hostname".
...
Caused by: ...JSchException: java.net.ConnectException: Connection refused (Connection refused)

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

Connection using an incorrect port number (here 222).

$ sftp -P 222 user@hostname
ssh: connect to host hostname port 222: Connection refused
Connection closed

Issue: Incorrect Credentials

If the FTP username or password is incorrect the following error messages are seen.

Troubleshooting using Masking GUI and Logs

Connection established but username/password failed. 

File Connector Test Button 

The logs from testing the connector.

...
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: publickey,keyboard-interactive,password
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: publickey
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: password
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: password
2019... INFO  o.a.c.v.p.sftp.SftpClientFactory - Disconnecting from dc3.delphix.com port 22
...
2019... ERROR common.utils.VFSUtility - Error ::
...FileSystemException: Could not connect to SFTP server at "sftp://hostname/".
...
Caused by: ...FileSystemException: Could not connect to SFTP server at "hostname".
...
Caused by: ...JSchException: Auth fail
Job Execution Error

If the credentials are incorrect when the masking job run, then this is the error: 

2019.. [JOB_ID_xx_yy] ERROR - XML Input - org.pentaho.di.core.exception.KettleFileException:
2019.. [JOB_ID_xx_yy] Unable to get VFS File object for filename 'sftp://delphix:...@hostname:22/folder/file' : Could not connect to SFTP server at "sftp://delphix:***@hostname/".
2019.. [JOB_ID_xx_yy]      at d.S.C.getFileObject(CustomKettleVFS.java:146)
...

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

Connection established but failed due to incorrect username/password. 

$ sftp wrong_user@hostname:22
wrong_user@hostname's password: wrong_password
Permission denied, please try again.
Command-line: FTP (Windows)

Connection established. The authentication failed due to incorrect username/password. 

...
User (hostname:(none)): wrong_user
331 Please specify the password.
Password: wrong_password
530 Login incorrect.
Login failed.

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

Issue: Invalid Folder

If the folder where the files to be masked are stored is incorrect then the following error messages are shown.

Troubleshooting using Masking GUI and Logs

The logs from testing the connector.

File Connector Test Button - version 5.3 and later

A very verbose log - and not a detailed error message. A test on the invalid folder can be seen.

2019.. INFO  c.dd.c.utils.VFSUtility - User delphix, Mode sftp: Testing on hostname:22//wrong_folder/
...
### SFTP LOG ENTRIES
...
2019.. INFO  o.a.c.v.p.sftp.SftpClientFactory - Authentication succeeded (password).
2019.. INFO  o.a.c.v.p.sftp.SftpClientFactory - Disconnecting from hostname port 22
2019.. INFO  o.a.c.v.p.sftp.SftpClientFactory - Caught an exception, leaving main loop due to Socket closed
File Connector Test Button - early version 5.3 and earlier

There is just a log entry that the folder was tested. 

2018... INFO, c.u.VFSUtility - User user, Mode sftp: Testing on hostname:22//wrong_folder
...
2018... INFO, .sftp.SftpClientFactory - Disconnecting from hostname port 22
Job Execution Error

If the folder or files can't be accessed during job execution.

2019.. [JOB_ID_xx_yy] ERROR - XML Input - Unexpected error
2019.. [JOB_ID_xx_yy] ERROR 11-03 11:13:00,775 - XML Input - org.pentaho.di.core.exception.KettleException:
2019.. [JOB_ID_xx_yy] o.a.c.vfs.FileSystemException: Could not read from "sftp://delphix:***@hostname/wrongfolder/file" because it is a not a file.
2019.. [JOB_ID_xx_yy] Could not read from "sftp://delphix:***@hostname/wrong_folder/file" because it is a not a file.

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

The cd to the 'wrong_folder' is returning an error. 

sftp> cd /wrong_folder
Couldn't stat remote file: No such file or directory
Command-line: FTP (Windows)

The cd to the 'wrong_folder' is returning an error. 

ftp> cd /wrong_folder
550 Failed to change directory.

Extended Tests: File Access

These tests are additional tests that can be done using the ftp client. The two tests are: 

  • get file
  • put file
Note

Note:

There are only error details from FTP and SFTP connection below since these two tests are not tested when testing the connection in the GUI. In saying that, File Access permissions are needed in order to mask files.

Issue: Get File

This tests if file permissions has been granted to read a file from the specified folder. Please check the file permissions on the ftp server.

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

No permission to open file. 

sftp> get [file]
Fetching /[folder]/[file] to [file]
remote open("/[folder[/[file]"): Permission denied
Command-line: FTP (Windows)

No permission to open file. 

ftp> get [file]
200 PORT command successful. Consider using PASV.
550 Failed to open file.

Issue: Put File

This tests if file permissions has been granted to write a file in the specified folder. It could also be that write is not granted in (s)ftp service configuration. Please check both.  

Troubleshooting using (S)FTP Client

Command-line: SFTP (Mac)

No permission to write file. 

sftp> put [file]
Uploading [file] to /[folder]/[file]
remote open("/[folder]/[file]"): Permission denied
Command-line: FTP (Windows)

No permission to write file. 

ftp> put [file]
200 PORT command successful. Consider using PASV.
550 Permission denied.