Oracle dSource SnapSync Fails With Internal Error After Upgrade to 19.3 or Later (KBA6003)

Issue

Following an Oracle upgrade to 19.3 or later, SnapSync operations may fail with an internal error, and suggested action to contact Delphix Support.

DB_SYNC job for "<dsource name>" failed due to an internal error during execution.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0
5.3	5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1

Resolution

This issue is ultimately caused by a combination of changes in Oracle 19.3 and Delphix behavior.  

Oracle 19.3 and later provide OJDBC jar files compiled with Java 8 as well as Java 10, indicated by filename:

$ ls -l $ORACLE_HOME/jdbc/lib 
total 49748 
-rw-r--r--. 1 oracle dba 7473912 Jan 6 22:07 ojdbc10dms_g.jar 
-rw-r--r--. 1 oracle dba 6178866 Jan 6 22:07 ojdbc10dms.jar 
-rw-r--r--. 1 oracle dba 7444205 Jan 6 22:07 ojdbc10_g.jar 
-rw-r--r--. 1 oracle dba 4428368 Jan 6 22:07 ojdbc10.jar 
-rw-r--r--. 1 oracle dba 7430114 Jan 6 22:07 ojdbc8dms_g.jar 
-rw-r--r--. 1 oracle dba 6143034 Jan 6 22:07 ojdbc8dms.jar 
-rw-r--r--. 1 oracle dba 7400595 Jan 6 22:07 ojdbc8_g.jar 
-rw-r--r--. 1 oracle dba 4395452 Jan 6 22:07 ojdbc8.jar 
-rw-r--r--. 1 oracle dba 29205 Jan 6 22:07 simplefan.jar

In affected versions of Delphix, the Java 10 compiled OJDBC libraries may be used during SnapSync, resulting in the internal error. Currently Delphix only supports Java 6 and Java 8 compiled libraries.

This behavior is fixed in 5.3.9.0+ and 6.0.1.0+.  With this fix, the highest supported version of OJDBC driver will be selected for SnapSync operations. This logic also addresses previous issue where an incorrect lower version was selected.

Workaround 

If an upgrade to a fixed version is not immediately possible, the issue can be worked around by temporarily relocating the Java 10 compiled versions of OJDBC jar files from $ORACLE_HOME/jdbc/lib, leaving only Java 8 compiled versions for Delphix to discover.  This change may affect other applications so the possible impact will have to be considered, but generally this is harmless.

An alternative workaround is to create the directory /opt/delphix/server/lib on the target host and copy $ORACLE_HOME/jdbc/lib/ojdbc8_g.jar to /opt/delphix/server/lib/ojdbc8_g-12.2.0.1.jar.  The file must be readable by the Delphix OS user.  The Delphix engine will check for this file on the target host before trying the Oracle libraries from $ORACLE_HOME/jdbc/lib.

Customers can NOT work around this issue by providing their own JDK and using OpenJDK, as Java 10 is not tested or supported. 

Troubleshooting

As the issue logs an Internal Error, this is not fully diagnosable without Support intervention to review a Support bundle. However, the workaround described above can be tested to confirm the issue referenced here.