Testing FTP and SFTP Masking Connectors (KBA1803)
At a Glance
Applicable to: | Applicable to File Masking using (S)FTP Connection in all Delphix Masking versions. This document is based on 5.3.2.0 and updated with details in 5.3.5.0. |
||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Errors: | This KBA covers:
|
||||||||||||||||||
Tools: | Issue investigation tools:
|
||||||||||||||||||
Delphix Supported Protocols: |
|
Connection Error
When testing the connection in the Connector popup in the GUI there are two responses:
- Success - Connection Succeeded!
If you still have issues though test is successful, please look at File Access. - Error - Connection Failed. Please check Connection Parameters.
This page is for you!
Troubleshooting
This KBA goes through two methods for troubleshooting:
- Using Masking GUI and Logs.
- Using (S)FTP Client.
Troubleshooting using Masking GUI and Logs
Troubleshooting using the GUI means that we need to access the logs to investigate the issue.
To access the logs go to Admin > Logs.
The most recent logs are at the bottom of the screen, scroll down to access.
Alternatively, use Export to export the logs to a text file.
Troubleshooting using (S)FTP Client
Rather than testing using the Masking UI and looking at the logs, it might be better to test the (S)FTP connection outside the masking engine. One benefit with this method is that it will enable you to look at the connection details on the screen and test them at the same time.
Recommended tests:
The recommended tests and examples below.
- Check if the host is reachable:
- Ping server with the FTP service.
- Detail needed: [hostname]
- Check connection and authentication:
- Connect to the FTP service using FTP client
- Details needed: [hostname], [user], and [password]
- Check that the folder with the files to mask is accessible:
- CD to the folder with the files:
- Detail needed: [folder].
- Check that files can be read:
- Get a file from the FTP server.
- Detail needed: [file] (a small test file):
- Check that files can be written.
- Put a file to the FTP server.
- Detail needed: [file] (a small test file).
- Quit.
Connection Examples
The two examples use the command-line programs called SFTP on macOS and FTP on Windows.
Command-line: SFTP (Mac)
Ref | Example command |
1 2 3 4 5 6 |
$ ping [hostname] $ sftp [user]@[hostname] 22 Password:[password] Connected to [hostname]. sftp> cd [folder] sftp> get [file] Fetching [foler]/[file] to [file] sftp> put [file] Uploading [file] to [folder] sftp> quit |
Command-line: FTP (Windows)
Ref | Example command |
1 2 3 4 5 6 |
C:\> ping [hostname] C:\> ftp [hostname] Connected to [hostname]. 220 (vsFTPd 3.0.3) User ([hostname]:(none)): [user] 331 Please specify the password. Password: [password] 230 Login successful. ftp> cd [folder] 250 Directory successfully changed. ftp> get [file] 200 PORT command successful. Consider using PASV. 150 Opening BINARY mode data connection for [file] (8 bytes). 226 Transfer complete. ftp: 8 bytes received in ... ftp> put [file] 200 PORT command successful. Consider using PASV. 150 Ok to send data. 226 Transfer complete. ftp: 8 bytes sent in ... ftp> quit |
Errors in Logs and Tests
Detailed below are errors from failed connections. They are used as examples to assist with troubleshooting, check the error message in the logs or the FTP client.
Successful Connector Test
A successful connector test looks like this (note - in later 5.3 versions it is very verbose):
2019... INFO com.dmsuite.common.utils.VFSUtility - User delphix, Mode sftp: Testing on dc3.delphix.com:22//tmp/ 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Connecting to dc3.delphix.com port 22 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Connection established ... ### SFTP LOG ENTRIES ... 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: publickey,keyboard-interactive,password 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: publickey 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: password 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: password 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Authentication succeeded (password). 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Disconnecting from dc3.delphix.com port 22
Issue: Connection Refused/Closed
The service is not accepting the connection.
Cause and resolution
The FTP/SFTP service is not correctly configured, available or enabled. This error is seen when a connection is made to a server that does not have this service (for example connecting SFTP to a Windows box without 3rd party software installed).
Troubleshooting using Masking GUI and Logs
Trying to access incorrectly configured (disabled) service.
2018... ERROR c.u.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://[hostname]/". ... Caused by: ...FileSystemException: Could not connect to SFTP server at "[hostname]". ... Caused by: ...JSchException: Auth fail
Trying to access a host which does not have ssh installed.
2018... ERROR c.u.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://[hostname]/". Caused by: ...FileSystemException: Could not connect to SFTP server at "[hostname]". ... Caused by: ...JSchException: java.net.ConnectException: Connection refused (Connection refused) ... Caused by: ...ConnectException: Connection refused (Connection refused)
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
Trying to access incorrectly configured (disabled) service. This is likely producing the same result if the connection is blocked by a Firewall.
$ sftp user@hostname subsystem request failed on channel 0 Connection closed
Trying to access a host which does not have ssh installed.
$ sftp user@hostname ssh: connect to host [hostname] port 22: Connection refused Connection closed
Command-line: FTP (Windows)
Trying to access an incorrectly configured FTP service.
C:\>ftp hostname > ftp: connect :Connection refused
Issue: Invalid Hostname
When the server name (hostname) is incorrect the following error messages will be returned.
Troubleshooting using Masking GUI and Logs
Logs when the hostname is invalid.
2018... INFO, .sftp.SftpClientFactory - Connecting to inval_host port 22 2018... ERROR c.u.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://inval_host:22/". ... Caused by: ...FileSystemException: Could not connect to SFTP server at "inval_host". ... Caused by: ...JSchException: java.net.UnknownHostException: inval_host
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
Trying to access an invalid host (inval_host).
$ sftp user@inval_host ssh: Could not resolve hostname inval_host: nodename nor servname provided, or not known Connection closed
Command-line: FTP (Windows)
Trying to access an invalid host (inval_host).
C:\>ftp inval_host Unknown host inval_host. ftp>
Issue: No Matching Host Key Type
This error will happen if there are no matching host keys between the server-client (SFTP Server - Masking Engine).
Troubleshooting using Masking GUI and Logs
Logs are not showing a lot here. The stack trace is providing some hints.
2019... ERROR - XML Input - Unexpected error 2019... ERROR - XML Input - j.i.UncheckedIOException: o.a.c.vfs.FileSystemException: Could not connect to SFTP server at "sftp://***@hostname/". ... 2019... Caused by: org.apache.commons.vfs.FileSystemException: Could not connect to SFTP server at "sftp://***@hostname/". 2019... at org.apache.commons.vfs.provider.sftp.SftpFileSystem.getChannel(Unknown Source) 2019... at org.apache.commons.vfs.provider.DefaultFileContent.getInputStream(Unknown Source) ... 2019... Caused by: com.jcraft.jsch.JSchException: java.io.IOException: inputstream is closed 2019... at com.jcraft.jsch.ChannelSftp.start(ChannelSftp.java:315) 2019... at com.jcraft.jsch.Channel.connect(Channel.java:152) 2019... at com.jcraft.jsch.Channel.connect(Channel.java:145) 2019... ... 12 more 2019... Caused by: java.io.IOException: inputstream is closed 2019... at com.jcraft.jsch.ChannelSftp.fill(ChannelSftp.java:2911) 2019... at com.jcraft.jsch.ChannelSftp.header(ChannelSftp.java:2935) 2019... at com.jcraft.jsch.ChannelSftp.start(ChannelSftp.java:262) 2019... ... 14 more
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
Trying to access an invalid host (inval_host).
$ sftp -vvv user@hostname Unable to negotiate with 192.168.5.145 port 22: no matching host key type found. Their offer: ssh-dss Connection closed
To turn on verbose logging '-vvv' (debug level 1,2,3).
$ sftp -vvv user@hostname
This issue is hard to resolve. If needed, please seek assistance from Delphix Support.
Issue: Invalid Port
If the port number is incorrect the following error messages are seen.
Troubleshooting using Masking GUI and Logs
Connection using an incorrect port number (here 222).
2018... INFO, .sftp.SftpClientFactory - Connecting to hostname port 222 2018... ERROR common.utils.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://hostname:222/". ... Caused by: ...FileSystemException: Could not connect to SFTP server at "hostname". ... Caused by: ...JSchException: java.net.ConnectException: Connection refused (Connection refused)
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
Connection using an incorrect port number (here 222).
$ sftp -P 222 user@hostname ssh: connect to host hostname port 222: Connection refused Connection closed
Issue: Incorrect Credentials
If the FTP username or password is incorrect the following error messages are seen.
Troubleshooting using Masking GUI and Logs
Connection established but username/password failed.
File Connector Test Button
The logs from testing the connector.
... 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: publickey,keyboard-interactive,password 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: publickey 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Authentications that can continue: password 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Next authentication method: password 2019... INFO o.a.c.v.p.sftp.SftpClientFactory - Disconnecting from dc3.delphix.com port 22 ... 2019... ERROR common.utils.VFSUtility - Error :: ...FileSystemException: Could not connect to SFTP server at "sftp://hostname/". ... Caused by: ...FileSystemException: Could not connect to SFTP server at "hostname". ... Caused by: ...JSchException: Auth fail
Job Execution Error
If the credentials are incorrect when the masking job run, then this is the error:
2019.. [JOB_ID_xx_yy] ERROR - XML Input - org.pentaho.di.core.exception.KettleFileException: 2019.. [JOB_ID_xx_yy] Unable to get VFS File object for filename 'sftp://delphix:...@hostname:22/folder/file' : Could not connect to SFTP server at "sftp://delphix:***@hostname/". 2019.. [JOB_ID_xx_yy] at d.S.C.getFileObject(CustomKettleVFS.java:146) ...
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
Connection established but failed due to incorrect username/password.
$ sftp wrong_user@hostname:22 wrong_user@hostname's password: wrong_password Permission denied, please try again.
Command-line: FTP (Windows)
Connection established. The authentication failed due to incorrect username/password.
... User (hostname:(none)): wrong_user 331 Please specify the password. Password: wrong_password 530 Login incorrect. Login failed.
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
Issue: Invalid Folder
If the folder where the files to be masked are stored is incorrect then the following error messages are shown.
Troubleshooting using Masking GUI and Logs
The logs from testing the connector.
File Connector Test Button - version 5.3 and later
A very verbose log - and not a detailed error message. A test on the invalid folder can be seen.
2019.. INFO c.dd.c.utils.VFSUtility - User delphix, Mode sftp: Testing on hostname:22//wrong_folder/ ... ### SFTP LOG ENTRIES ... 2019.. INFO o.a.c.v.p.sftp.SftpClientFactory - Authentication succeeded (password). 2019.. INFO o.a.c.v.p.sftp.SftpClientFactory - Disconnecting from hostname port 22 2019.. INFO o.a.c.v.p.sftp.SftpClientFactory - Caught an exception, leaving main loop due to Socket closed
File Connector Test Button - early version 5.3 and earlier
There is just a log entry that the folder was tested.
2018... INFO, c.u.VFSUtility - User user, Mode sftp: Testing on hostname:22//wrong_folder ... 2018... INFO, .sftp.SftpClientFactory - Disconnecting from hostname port 22
Job Execution Error
If the folder or files can't be accessed during job execution.
2019.. [JOB_ID_xx_yy] ERROR - XML Input - Unexpected error 2019.. [JOB_ID_xx_yy] ERROR 11-03 11:13:00,775 - XML Input - org.pentaho.di.core.exception.KettleException: 2019.. [JOB_ID_xx_yy] o.a.c.vfs.FileSystemException: Could not read from "sftp://delphix:***@hostname/wrongfolder/file" because it is a not a file. 2019.. [JOB_ID_xx_yy] Could not read from "sftp://delphix:***@hostname/wrong_folder/file" because it is a not a file.
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
The cd to the 'wrong_folder' is returning an error.
sftp> cd /wrong_folder Couldn't stat remote file: No such file or directory
Command-line: FTP (Windows)
The cd to the 'wrong_folder' is returning an error.
ftp> cd /wrong_folder 550 Failed to change directory.
Extended Tests: File Access
These tests are additional tests that can be done using the ftp client. The two tests are:
- get file
- put file
Issue: Get File
This tests if file permissions has been granted to read a file from the specified folder. Please check the file permissions on the ftp server.
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
No permission to open file.
sftp> get [file] Fetching /[folder]/[file] to [file] remote open("/[folder[/[file]"): Permission denied
Command-line: FTP (Windows)
No permission to open file.
ftp> get [file] 200 PORT command successful. Consider using PASV. 550 Failed to open file.
Issue: Put File
This tests if file permissions has been granted to write a file in the specified folder. It could also be that write is not granted in (s)ftp service configuration. Please check both.
Troubleshooting using (S)FTP Client
Command-line: SFTP (Mac)
No permission to write file.
sftp> put [file] Uploading [file] to /[folder]/[file] remote open("/[folder]/[file]"): Permission denied
Command-line: FTP (Windows)
No permission to write file.
ftp> put [file] 200 PORT command successful. Consider using PASV. 550 Permission denied.