Following an Oracle upgrade to 19.3 or later, SnapSync operations may fail with an internal error, and suggested action to contact Delphix Support.
DB_SYNC job for "<dsource name>" failed due to an internal error during execution.
The issue occurs after upgrade to 19.3 or later.
The Delphix Engine is running one of the applicable versions listed below.
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases 6.0 188.8.131.52
184.108.40.206 220.127.116.11, 18.104.22.168, 22.214.171.124, 126.96.36.199, 188.8.131.52
This issue is ultimately caused by a combination of changes in Oracle 19.3 and Delphix behavior.
Oracle 19.3 and later provide OJDBC jar files compiled with Java 8 as well as Java 10, indicated by filename:
$ ls -l $ORACLE_HOME/jdbc/lib total 49748 -rw-r--r--. 1 oracle dba 7473912 Jan 6 22:07 ojdbc10dms_g.jar -rw-r--r--. 1 oracle dba 6178866 Jan 6 22:07 ojdbc10dms.jar -rw-r--r--. 1 oracle dba 7444205 Jan 6 22:07 ojdbc10_g.jar -rw-r--r--. 1 oracle dba 4428368 Jan 6 22:07 ojdbc10.jar -rw-r--r--. 1 oracle dba 7430114 Jan 6 22:07 ojdbc8dms_g.jar -rw-r--r--. 1 oracle dba 6143034 Jan 6 22:07 ojdbc8dms.jar -rw-r--r--. 1 oracle dba 7400595 Jan 6 22:07 ojdbc8_g.jar -rw-r--r--. 1 oracle dba 4395452 Jan 6 22:07 ojdbc8.jar -rw-r--r--. 1 oracle dba 29205 Jan 6 22:07 simplefan.jar
In affected versions of Delphix, the Java 10 compiled OJDBC libraries may be used during SnapSync, resulting in the internal error. Currently Delphix only supports Java 6 and Java 8 compiled libraries.
This behavior is fixed in 184.108.40.206+ and 220.127.116.11+. With this fix, the highest supported version of OJDBC driver will be selected for SnapSync operations. This logic also addresses previous issue where an incorrect lower version was selected.
If an upgrade to a fixed version is not immediately possible, the issue can be worked around by temporarily relocating the Java 10 compiled versions of OJDBC jar files from $ORACLE_HOME/jdbc/lib, leaving only Java 8 compiled versions for Delphix to discover. This change may affect other applications so the possible impact will have to be considered, but generally this is harmless.
An alternative workaround is to create the directory /opt/delphix/server/lib on the target host and copy $ORACLE_HOME/jdbc/lib/ojdbc8_g.jar to /opt/delphix/server/lib/ojdbc8_g-18.104.22.168.jar. The file must be readable by the Delphix OS user. The Delphix engine will check for this file on the target host before trying the Oracle libraries from $ORACLE_HOME/jdbc/lib.
Customers can NOT work around this issue by providing their own JDK and using OpenJDK, as Java 10 is not tested or supported.
As the issue logs an Internal Error, this is not fully diagnosable without Support intervention to review a Support bundle. However, the workaround described above can be tested to confirm the issue referenced here.
The following articles may provide more information or related information to this article:
- https://www.oracle.com/database/technologies/faq-jdbc.html - What are the Oracle JDBC releases Vs JDK versions?