TB027 Delphix May Delete Archive Log Info from Oracle Control File
Alert Type
Data Availability / Data Loss
Background
The Oracle Recovery Manager (RMAN) utility is used, among other purposes, for backing up Oracle Databases and associated archive logs. The Delphix Engine uses RMAN when creating Oracle dSources, creating new Delphix Snapshots for existing dSources and when maintaining a continuous timeflow of data using the Delphix LogSync function. As RMAN completes backup operations, including those done on behalf of Delphix, information about the operations are included in one or more Oracle control files.
RMAN supports two channel types for writing backup data. These channels can be either disk channels for normal disk I/O or System Backup to Tape (SBT) channels for data streamed through a media manager. The Delphix Engine exclusively uses the SBT channel.
Impact
The Delphix Engine may issue RMAN commands that delete control file records for SBT archive log backups, including those for backups initiated outside of Delphix. Although the issue does not impact the normal operation of Delphix, it can interfere with the ability to restore backups not associated with Delphix. In such cases, restores may become difficult and protracted because RMAN will not have a complete history of the available archive logs for restoration.
Note that the backed up archive logs themselves are not removed.
Contributing Factors
- The issue may occur in the following Delphix Engine Releases:
- Delphix Engine 4.2.1.0 to 4.2.5.0
- Delphix Engine 4.3.1.0 to 4.3.2.0
- The issue can only occur if the LogSync is enabled on an affected Oracle database
- The issue can only occur if RMAN backups to an SBT channel are done outside of Delphix for the same databases managed as a Delphix dSource with LogSync enabled
Symptoms
RMAN restore operations may fail with errors relating to missing archive logs, for example.
RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 10/06/2015 03:00:04 RMAN-06053: unable to perform media recovery because of missing log RMAN-06025: no backup of archived log for thread 1 with sequence 18 and starting SCN of 966915 found to restore
If a 3rd party backup solution is used ( such as Commvault, Data Domain, etc ) the error message may vary, however messages will likely suggest that archive logs are missing.
Relief/Workaround
For immediate relief from this issue, disable LogSync.
If recovery is required then the missing backup pieces will need to be catalogued. To do this the backup piece name will need to be known. The name should either be found in RMAN logs or in the logs of a 3rd party backup solution. If using a 3rd party backup solution, it is recommended that a case to be opened with the vendor to ensure the correct procedure is followed.
The following RMAN example catalogs the missing backup piece "0eqj0ar6_1_1". This example uses the oracle SBT library that will need to be configured, as appropriate, to function with the local backup configuration. This procedure also requires auto channels to be configured. The auto channel configuration and settings should be verified prior to executing the catalog procedure. If changes to the RMAN configuration are required, they should be saved prior to performing this step.
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' parms="SBT_LIBRARY=oracle.disksbt, ENV=(BACKUP_DIR=/datafile/backup)"; catalog device type 'SBT_TAPE' backuppiece '0eqj0ar6_1_1'; CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' CLEAR;
The success of this step can be confirmed by verifying that RMAN knows where to find the previously missing archive log:
RMAN> list backup of archivelog sequence 18;
using target database control file instead of recovery catalog
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time
------- ---------- ----------- ------------ ---------------
15 Unknown SBT_TAPE 00:00:00 06-OCT-15
BP Key: 85 Status: AVAILABLE Compressed: NO Tag: ARCHLOGS
Handle: 0eqj0ar6_1_1 Media: /backup,0eqj0ar6_1_1
List of Archived Logs in backup set 15
Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 18 966915 06-OCT-15 966924 06-OCT-15
RMAN>
Note that the example above lists the archive log for sequence 18, which was identified in an earlier recover error message.
Once the missing backup pieces have been catalogued, the RMAN recover command ( or 3rd party recovery process ) can be attempted again. If another missing log is found then the procedure can be repeated until recovery is successful.
Further information on this process can be found in Oracle support note - 550082.1
If re-cataloging of the backup pieces is not an option, it may be possible to retrieve the missing archive logs from the Delphix Engine itself, depending on the timeframe between the deletion incident and discovery. Please contact Delphix Support to assist with this process if required.
Resolution
The issue is fully resolved in Delphix Engine version 4.2.5.1, Delphix Engine 4.3.2.1, and later releases.
Additional Information
The following query can be used to confirm if the dSource has seen this issue within recent history (the views used are reset when the instance is shutdown and can hold 32768 lines of history). The output will show the number of recent Delphix initiated backups which have deleted non Delphix archive log backups, if the result is 0 then no recent non Delphix archive log backups have been deleted.
select count(*)
from v$rman_output a
,v$rman_output b
,v$rman_status s
where s.operation = 'DELETE'
and s.session_recid = a.session_recid
and s.session_stamp = a.session_stamp
and s.session_recid = b.session_recid
and s.session_stamp = b.session_stamp
and a.output like '%tag=DLPX%'
and b.output like 'backup piece handle=%'
and b.output not like '%' || substr(a.output,14, instr(a.output,'tag') - 15 ) || '%';
To check longer term backup history a cross reference of the physical backup pieces and RMAN tables would need to be performed, the process for this will depend on the RMAN SBT plugins being used.