ORA-01503: CREATE CONTROLFILE failed Encountered with ORA-00600 During VDB Provision or Refresh using dNFS (KBA9021)

Issue

During VDB provsion or refresh, the job may fail with the event Failed to recreate control file.  

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

Troubleshooting

Reviewing the Oracle alert log for the VDB, or the auxiliary CDB created during recovery, ORA-01503 and ORA-00600 may be encountered with the following signature (in the instance ORA-00600 internal errors are encountered, the distinct arguments are used to further correlate events):

Thu Mar 17 14:22:04 2022
WARNING: Default Temporary Tablespace not specified in CREATE DATABASE command
Default Temporary Tablespace will be necessary for a locally managed database in future release
Errors in file /u02/app/oracle/diag/rdbms/cxvjrfuvmnd/CXVJRFUvmnd/trace/CXVJRFUvmnd_ora_177613.trc  (incident=12721) (PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [ksfd_odmcrt3], [10008], [], [], [], [], [], [], [], [], [], []
Incident details in: /u02/app/oracle/diag/rdbms/cxvjrfuvmnd/CXVJRFUvmnd/incident/incdir_12721/CXVJRFUvmnd_ora_177613_i12721.trc

Additionally, ORA-17500 events may also be logged, but this has not been observed consistently.

Resolution

Further investigation with Oracle has correlated this behavior with a known issue, detailed in MOS under Bug 20720667  Operations over remounted dNFS raise ORA-17500 or write files in old mountpoint (linked below).

In addition to application of the patch referenced, alternative workaround are offered such as:

• Restarting the database
• Use a different NFS mount point (mount base, in VDB provision context)
• Disable Direct NFS (dNFS)

As the database instance started up is distinct in Delphix VDB recovery, the suggested restart of the affected instance is not applicable, but the core issue is related to mount details in SGA, presumably shutting down ALL database instances running on the affected host and restarting may successfully work around this behavior.

Ultimately, Oracle Support should be engaged when this behavior is encountered, and the alert log as well as trace files referenced in the alert log provided for further review.