LogSync Failures And Host Connection Faults After Upgrade to 11.0.0.0 or Higher (KBA10258)

Issue

Following upgrade to 11.0.0.0 or higher, Oracle LogSync failures may be encountered, along with associated faults indicating a host connectivity issue.  

These faults will appear with the Title:

LogSync failed to connect to remote host

and Description:

Cannot connect to host <HOSTNAME>: Failed to perform backup of log files on instance <INSTANCENAME> on host <HOSTNAME>.

No issues with LogSync are observed prior to upgrade.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Date	Release
Jul 24, 2023	13.0.0.0
Jun 21, 2023	12.0.0.0
May 25, 2023	11.0.0.0

Troubleshooting

In the instance that the source database has triggers enabled, if those triggers cause an error during RMAN execution it may cause LogSync to improperly exit and mark the archive log retrieval unsuccessful.

During LogSync RMAN backup, the typical execution results in an expected failure, but we are successful in receiving the log. We exit "abnormally" so as to ensure the backup is marked as failed in your RMAN catalog, otherwise any 'cleanup' operations may remove archive logs considered to be successfully backed up to our storage, but not to any other backup platform. An example of the executed RMAN backup execution is provided below:

connected to target database: ExampleDB (DBID=123123123123)

echo set on

executing command: SET encryption
using target database control file instead of recovery catalog

executing command: SET NOCFAU

allocated channel: DLPX_CHANNEL_1
channel DLPX_CHANNEL_1: SID=1099 device type=SBT_TAPE
channel DLPX_CHANNEL_1: Delphix SBT API

executing command: SET COMMAND ID

Starting backup at 07/20/2023 19:27:48
channel DLPX_CHANNEL_1: starting archived log backup set
channel DLPX_CHANNEL_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=187508 RECID=600549 STAMP=1142710064
channel DLPX_CHANNEL_1: starting piece 1 at 07/20/2023 19:27:50
released channel: DLPX_CHANNEL_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on DLPX_CHANNEL_1 channel at 07/20/2023 19:27:57
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   dlpxSbtclose: dlpxSbtclose normal LogSync exit

This behavior is also discussed in the following KB article:

https://support.delphix.com/Continuous_Data_Engine_(formerly_Virtualization_Engine)/Oracle/KBA1011_RMAN_Backup_Log_%22Failure%22_in_Oracle_is_Normal_in_5.0.4_and_Greater

For example, if a trigger is configured to capture and email details of an Oracle error in normal operation, the Oracle and RMAN errors generated by Delphix may result in a trigger issue, causing disruption to the LogSync process. Example:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on DLPX_CHANNEL_1 channel at 07/20/2023 19:27:57
ORA-04088: error during execution of trigger 'EMAIL_ORA_ERRORS'
ORA-00604: error occurred at recursive SQL level 1
ORA-06502: PL/SQL: numeric or value error
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   dlpxSbtclose: dlpxSbtclose normal LogSync exit

Resolution

A bug is filed to improve LogSync handling in these conditions, but it is not yet resolved at the time of this publication.

To confirm this behavior, the trigger can be disabled to determine if LogSync is functional after the change.

If the trigger needs to remain in place, the expected RMAN and ORA errors referenced can be 'ignored' by the trigger to avoid its execution.

• RMAN-00571

• RMAN-00569

• RMAN-03009

• ORA-27192

• ORA-19511