Skip to main content
Delphix

Recover vPDB after physical CDB crash KBA 7034

 

KBA

KBA# 7034

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases
6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Troubleshooting <What>

From the physical CDB's alert log we can verify the crash such as - 

cd to the alert log location 

view alert_<CDB>.log

From the bottom shift G move up in the alert log should see something like the following or a shutdown abort - 

2021-01-22T19:09:39.622685-05:00
Performing implicit shutdown abort due to dead PMON
Shutting down instance (abort) (OS id: 417)
License high water mark = 292
USER (ospid: 417): terminating the instance
2021-01-22T19:09:40.692957-05:00
Instance terminated by USER, pid = 417
2021-01-22T19:09:48.045808-05:00
Instance shutdown complete (OS id: 417)
2021-01-22T19:09:48.045918-05:00
ORA-1092 : opitsk aborting process
2021-01-22T21:37:50.275155-05:00
Starting ORACLE instance (normal) (OS id: 9568)

Followed by messages like the following - 

2021-01-22T21:38:03.762709-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/cdb_dbw0_15196.trc:
ORA-01157: cannot identify/lock data file 575 - see DBWR trace file
ORA-01110: data file 575: '/u01/delphix/VDB/VDPB-1-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/system.333.995683467'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

<cut>

2021-01-22T21:38:03.768676-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/cdb_dbw0_15196.trc:
ORA-01157: cannot identify/lock data file 602 - see DBWR trace file
ORA-01110: data file 602: '/u01/delphix/VDB/VPDB-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/system.333.995683467'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

<cut>

*********************************************************
Pluggable Database VPDB_1 with id - 5 will be marked OFFLINE
since one or more of the datafiles belonging to the
pluggable database is either missing or corrupted.
The pluggable database needs to be either dropped or
recovered from an earlier backup
*********************************************************
*********************************************************
Pluggable Database VPDB with id - 6 will be marked OFFLINE
since one or more of the datafiles belonging to the
pluggable database is either missing or corrupted.
The pluggable database needs to be either dropped or
recovered from an earlier backup
*********************************************************

Meaning these vPDBs require recovery.

Prerequisites 

text

Unable to start|enable|disable|rewind|refresh the affected vPDB

These operations fail with the following - 

fail_message_desc: Failed to unplug the pluggable database.
fail_message_id: exception.oracle.targetscripts.pdb.unplug
fail_message_params: ["command: umask 027; . $DB_SCRIPT_DIR/setup-oraenv.sh; $DLPX_SHELL $DB_SCRIPT_DIR/doUnplugPDB.sh \"/u01/delphix/toolkit\"\noutput:END_OF_SETUP\nEND_OF_SETUP\nORA-01113: file 639 needs media recovery\nORA-01110: data file 639:\nalter pluggable database \"VPDB\" unplug into '/u01/delphix/VDB/VPDB-cdb/datafile/delphix_group_writable/VPDB.xml'\n*\nERROR at line 1:\nORA-01113: file 639 needs media recovery\nORA-01110: data file 639:\n'/fisc/delphix/VDB/VPDB-cdb1/datafile/+DB_ORCL_DAT/dir3\n34BFF74DEE053C11B8A0A5D63/DATAFILE/foo_ind.347.996497651'\nUNPLUG_PDB_ERROR\n"]

 

The alert log will show something similar to the following - 

2021-02-11T10:40:26.386599-05:00
alter pluggable database "VDPB" open read write
2021-02-11T10:40:26.563125-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/cdb_m003_5382.trc:
ORA-01110: data file 629: '/u01/delphix/VDB/VPDB-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/foo_ind.277.1046234929'
ORA-01208: data file is an old version - not accessing current version
2021-02-11T10:40:27.925186-05:00
Pdb VPDB hit error 1113 during open read write (1) and will be closed.
2021-02-11T10:40:27.933148-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/vcspsd1_ora_7499.trc:
ORA-01113: file 639 needs media recovery
ORA-01110: data file 639: '/u01/delphix/VDB/VPDB-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/foo_ind.347.996497651'
VWISPSA17(6):JIT: pid 7499 requesting stop
ORA-1113 signalled during: alter pluggable database "VPDB" open read write ...
2021-02-11T10:40:28.201295-05:00
VPDB(6):                alter database recover automatic database
2021-02-11T10:40:28.202233-05:00
VPDB(6):Media Recovery Start
2021-02-11T10:40:28.202574-05:00
VPDB(6):Serial Media Recovery started
VPDB(6):WARNING! Recovering data file 639 from a fuzzy backup. It might be an online
VPDB(6):backup taken without entering the begin backup command.

Resolution

To resolve xyz...

Complete the following procedure to xyz.

  1. Determine what archive log is missing -
    1. sqlplus / as sysdba
    2. show pdbs
    3. alter session set container = VPDB ;
    4. alter pluggable database recover automatic ;
  2. Can this archive log be restored from backup - 
    1. rman target /    ## or to your rman catalog where the backups reside
    2. From within rman can use a command similar to - 
      1. restore archivelog from logseq=<> until logseq=<> thread <> ; 
  3. If the needed archive log(s) are not available from backup then contact Delphix Support via a support case as the Delphix Engine "may" have these archive logs which we can use.
  4. Recover the vPDB
    1. sqlplus / as sysdba
    2. show pdbs
    3. alter session set container = VPDB ;
    4. alter pluggable database recover automatic ;
  5. From the Delphix GUI start the vPDB.
  6. Take a snaphot of the vPDB from the Delphix GUI.