Recover vPDB after physical CDB crash KBA 7034

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
4.3	4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0
4.2	4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1
4.1	4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Troubleshooting <What>

From the physical CDB's alert log we can verify the crash such as - 

cd to the alert log location 

view alert_<CDB>.log

From the bottom shift G move up in the alert log should see something like the following or a shutdown abort - 

2021-01-22T19:09:39.622685-05:00
Performing implicit shutdown abort due to dead PMON
Shutting down instance (abort) (OS id: 417)
License high water mark = 292
USER (ospid: 417): terminating the instance
2021-01-22T19:09:40.692957-05:00
Instance terminated by USER, pid = 417
2021-01-22T19:09:48.045808-05:00
Instance shutdown complete (OS id: 417)
2021-01-22T19:09:48.045918-05:00
ORA-1092 : opitsk aborting process
2021-01-22T21:37:50.275155-05:00
Starting ORACLE instance (normal) (OS id: 9568)

Followed by messages like the following - 

2021-01-22T21:38:03.762709-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/cdb_dbw0_15196.trc:
ORA-01157: cannot identify/lock data file 575 - see DBWR trace file
ORA-01110: data file 575: '/u01/delphix/VDB/VDPB-1-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/system.333.995683467'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

<cut>

2021-01-22T21:38:03.768676-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/cdb_dbw0_15196.trc:
ORA-01157: cannot identify/lock data file 602 - see DBWR trace file
ORA-01110: data file 602: '/u01/delphix/VDB/VPDB-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/system.333.995683467'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7

<cut>

*********************************************************
Pluggable Database VPDB_1 with id - 5 will be marked OFFLINE
since one or more of the datafiles belonging to the
pluggable database is either missing or corrupted.
The pluggable database needs to be either dropped or
recovered from an earlier backup
*********************************************************
*********************************************************
Pluggable Database VPDB with id - 6 will be marked OFFLINE
since one or more of the datafiles belonging to the
pluggable database is either missing or corrupted.
The pluggable database needs to be either dropped or
recovered from an earlier backup
*********************************************************

Meaning these vPDBs require recovery.

fail_message_desc: Failed to unplug the pluggable database.
fail_message_id: exception.oracle.targetscripts.pdb.unplug
fail_message_params: ["command: umask 027; . $DB_SCRIPT_DIR/setup-oraenv.sh; $DLPX_SHELL $DB_SCRIPT_DIR/doUnplugPDB.sh \"/u01/delphix/toolkit\"\noutput:END_OF_SETUP\nEND_OF_SETUP\nORA-01113: file 639 needs media recovery\nORA-01110: data file 639:\nalter pluggable database \"VPDB\" unplug into '/u01/delphix/VDB/VPDB-cdb/datafile/delphix_group_writable/VPDB.xml'\n*\nERROR at line 1:\nORA-01113: file 639 needs media recovery\nORA-01110: data file 639:\n'/fisc/delphix/VDB/VPDB-cdb1/datafile/+DB_ORCL_DAT/dir3\n34BFF74DEE053C11B8A0A5D63/DATAFILE/foo_ind.347.996497651'\nUNPLUG_PDB_ERROR\n"]

2021-02-11T10:40:26.386599-05:00
alter pluggable database "VDPB" open read write
2021-02-11T10:40:26.563125-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/cdb_m003_5382.trc:
ORA-01110: data file 629: '/u01/delphix/VDB/VPDB-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/foo_ind.277.1046234929'
ORA-01208: data file is an old version - not accessing current version
2021-02-11T10:40:27.925186-05:00
Pdb VPDB hit error 1113 during open read write (1) and will be closed.
2021-02-11T10:40:27.933148-05:00
Errors in file /u01/oracle/diag/rdbms/cdb/cdb/trace/vcspsd1_ora_7499.trc:
ORA-01113: file 639 needs media recovery
ORA-01110: data file 639: '/u01/delphix/VDB/VPDB-cdb/datafile/+DB_ORCL_DAT/7DC09A334BFF74DEE053C11B8A0A5D63/DATAFILE/foo_ind.347.996497651'
VWISPSA17(6):JIT: pid 7499 requesting stop
ORA-1113 signalled during: alter pluggable database "VPDB" open read write ...
2021-02-11T10:40:28.201295-05:00
VPDB(6):                alter database recover automatic database
2021-02-11T10:40:28.202233-05:00
VPDB(6):Media Recovery Start
2021-02-11T10:40:28.202574-05:00
VPDB(6):Serial Media Recovery started
VPDB(6):WARNING! Recovering data file 639 from a fuzzy backup. It might be an online
VPDB(6):backup taken without entering the begin backup command.