Skip to main content
Delphix

Masking Error "Connection reset" (KBA10239)

  1. Last updated
  2. Save as PDF
 

 

KBA

KBA# 10239

 

At a Glance   

Description This KBA details a troubleshooting guide for the JDBC error 'Connection reset' and 'Connection reset by peer'. 
Root cause The error is thrown when the existing connection to the database has been closed abruptly by the server, and the Java application was not expecting this. Consequently, any further attempts to use the closed connection will result in an error.
 

"Connection reset" errors are often symptoms of underlying issues and if you encounter frequent connection reset errors, it's essential to collaborate with your network or database administrators to identify and resolve the underlying problem.
Troubleshooting
Steps:		 The main goal here is to find what terminated the connection and why? This will then define what actions are needed (resource change, network change, or other) to resolve the issue.

The steps below are only suggestions on where to start investigating:
1. Jobs The main step on the Masking Engine is to find the time of the event. It might also be that multiple connections at the same time are causing resource saturation on the database or network which terminates the connection.
  • Check timestamps - when the connection was up and when there were connection errors. 
  • Are there multiple jobs running starting at the same time? Multiple Streams? Update Threads?
2. Scripts  Verify scripts that start and check status are not timing out (and as a result cause the database or listener to drop). Scripts should be written to keep the (i.e. API) connection alive. 
3. Network Network latency can cause JDBC connections to be terminated (or cause network errors resulting in connections issues):
  • Talk to your network administrators and check relevant logs. 
4. Router or Firewall The firewall can have rules defined that will kill long-lived network connections (such as masking engine JDBC connections).
  • Talk to your network administrators and check the firewall logs.
5. Database The database can hit resource constraints and the database might kill connections:
  • Talk to your DBA and check database logs for errors or any signs of resource saturation. 
    • For example, check memory, CPU, and storage.
6. Host It is possible that this error is caused by changes on the hosting platform (such as ESX) or another external host or service (aka 'Noisy Neighbor').
More Info: More info on how to access logs: 

Other Connection errors;

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Date Release
Sep 21, 2023 15.0.0.0
Aug 24, 2023 14.0.0.0
Jul 24, 2023 13.0.0.0
Jun 21, 2023 12.0.0.0
May 25, 2023 11.0.0.0
Apr 13, 2023 10.0.0.0 | 10.0.0.1
Mar 13, 2023 | Mar 20, 2023 9.0.0.0 | 9.0.0.1
Feb 13, 2023 8.0.0.0
Jan 12, 2023 7.0.0.0
Releases Prior to 2023
Major Release All Sub Releases
6.0

6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2

5.3

 5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Troubleshooting Connection reset

'Connection reset' or 'Connection reset by peer' errors are issues caused on the other side of the connection. Therefore, the investigation should start at the different systems connected to the Masking Engine that can be the point of failure. These systems are outside the scope of this KBA. 

The details here will focus on getting the relevant details from the Masking Engine.

Analyzing Logs

This section shows how to use grep (Linux) (or using Select-String for Windows) to get more details from the Masking Engine logs.

The logs can be downloaded from the Job Monitor, from Admin > Logs, or using the API. For more details on how to access these, see KBA7988 listed above.

Search terms

In order to find the details in the logs, you need to look at all errors in the job. Search for (some modifications might be required based on the connector type and version):

  • E=1
  • SocketException
  • Connection reset
Linux example

Below is an example using grep:

grep -E "E=1|SocketException|Connection reset" logfile.log
Window example 

Below is an example using Select-String:

Select-String "E=1|SocketException|Connection reset" logfile.log

Example Output

The commands above should return an output similar to:

[JOB_ID_n_m] 2023/07/15 14:14:33 - DelphixTableUpdate.0 - Connection reset
[JOB_ID_n_m] 2023/07/15 14:14:33 - DelphixTableUpdate.0 - Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
[JOB_ID_n_m] 2023/07/15 14:14:33 - DelphixTableUpdate.0 - Caused by: java.net.SocketException: Connection reset
[JOB_ID_n_m] 2023/07/15 14:14:33 - DelphixTableUpdate.0 - Finished processing (I=18370000, O=0, R=18270000, W=18269999, U=18269999, E=1)
[JOB_ID_n_m] Connection reset
[JOB_ID_n_m] Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
[JOB_ID_n_m] 2023/07/15 14:14:33 - Table input.0 - Connection reset
[JOB_ID_n_m] 2023/07/15 14:14:33 - Table input.0 - Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
[JOB_ID_n_m] 2023/07/15 14:14:35 - Table input.0 - Finished processing (I=18280433, O=0, R=0, W=18280432, U=0, E=1)
[JOB_ID_n_m] 2023/07/15 14:15:01 - DelphixTableUpdate.0 - Finished processing (I=3540000, O=0, R=3540000, W=3539999, U=3539999, E=1)

 

More about the logs

Logs are explored in more detail in this section. The log output might differ and there may be other job steps that cause the 'Connection reset' (such as a Custom Algorithm which has a connection to a database).

Salient entries in the log:

  • The key entry is Connection reset
    • In this example: both step Table input and step DelphixTableUpdate have failed.  
       
  • Notes around step: DelphixTableUpdate
    • The log entry Error updating batch indicates that the batch couldn't update (due to the Connection error).
    • Due to the error, the job is trying to Rollback to the latest Batch - this fails because The connection is closed. 
       
  • Notes around step: Table input
    • The entries related to getting result set and get value indicate that no data could be fetched (due to the Connection error). The effect can be different between steps.
       
  • The JAVA stacks are from the JDBC client and differ between JDBC connectors (details irrelevant). 
2023/07/15 14:14:33 - DelphixTableUpdate.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Error in step, asking everyone to stop because of:
2023/07/15 14:14:33 - DelphixTableUpdate.0 - Error updating batch
2023/07/15 14:14:33 - DelphixTableUpdate.0 - Connection reset
...
2023/07/15 14:14:33 - DelphixTableUpdate.0 - Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
...
2023/07/15 14:14:33 - DelphixTableUpdate.0 - Caused by: java.net.SocketException: Connection reset
...
2023/07/15 14:14:33 - DelphixTableUpdate.0 - Error performing rollback on connection
2023/07/15 14:14:33 - DelphixTableUpdate.0 - The connection is closed.
2023/07/15 14:14:33 - DelphixTableUpdate.0 - Finished processing (I=18270000, O=0, R=18270000, W=18269999, U=18269999, E=1)
...
2023/07/15 14:14:33 - Table input.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Unexpected error
2023/07/15 14:14:33 - Table input.0 - Couldn't get row from result set
2023/07/15 14:14:33 - Table input.0 - Unable to get value 'String(25)' from database resultset, index 2
2023/07/15 14:14:33 - Table input.0 - Connection reset
...
2023/07/15 14:14:33 - Table input.0 - Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
...
2023/07/15 14:14:35 - Table input.0 - Finished processing (I=18280433, O=0, R=0, W=18280432, U=0, E=1)

 

 

Privacy Policy Terms of Use Legal About
© 2024 Delphix Corp