Removing Temporary NFS Files That Are Left In The Filesystem (KBA6534)

Issue  

If you drop a tablespace and create datafiles that reuse the datafile numbers of the dropped datafile, it is possible that temporary NFS files are left in the filesystem. When that happens, during doRenameDatafiles.sh in VDB provisioning/refresh/rewind, Delphix finds these invalid copies. This causes a problem, as it results in having two copies for the same datafile. And, depending on the order in which they are found, Delphix might end up cataloging the NFS temporary file instead of the current datafile, causing a RESETLOGS error mismatch.

Normally Delphix would remove the .nfs temporary files, but when the files are owned by the "oracle" OS user, Delphix cannot delete them. Instead, a 'Permission denied' error is provided (as shown below). The Delphix scripts used to remove these files are run by the "delphix" user.

[2020-09-14 13:49:32,911][INFO][jcm.impl.JobInstance#execute:355][Worker-3474|JOB-1848|DB_REFRESH(ORACLE_DB_CONTAINER-69)][] job failed with user error
com.delphix.common.exception.messages.OracleTargetscriptsExceptions$RenameDatafiles: exception.oracle.targetscripts.rename.datafiles {command: umask 027; . $DB_SCRIPT_DIR/setup-oraenv.sh; $DLPX_SHELL $DB_SCRIPT_DIR/doRenameDatafiles.sh $DB_SCRIPT_DIR "$LOGON_STR" "/delphix/hellodb/datafile"
output:END_OF_SETUP
END_OF_SETUP
Check if /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000001200000026 is a datafile
rm: cannot remove '/delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000001200000026': Permission denied
Check if /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000002100000030 is a datafile
rm: cannot remove '/delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000002100000027': Permission denied
........
NUMBER_OF_DATAFILES_MDS=53, NUMBER_OF_FILES_IN_FS=70
Extra files in directory remaining:
/delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000001200000026
/delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000002100000030
........
List of Files Unknown to the Database
=====================================
File Name: /delphix/hellodb/datafile/spfile.ora
File Name: /delphix/hellodb/datafile/init.ora.createControlfile
File Name: /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000004500000030
File Name: /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/acses_ts_encrypt.278.1036365707
File Name: /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs00000000000000870000002d
.........
File Name: /delphix/hellodb/datafile/init.ora.rename
File Name: /delphix/hellodb/datafile/init.ora.recovery
File Name: /delphix/hellodb/datafile/source_init.ora
cataloging files...
cataloging done
.........
List of Files Which Were Not Cataloged
=======================================
File Name: /delphix/hellodb/datafile/spfile.ora
  RMAN-07518: Reason: Foreign database file DBID: 0  Database Name:
File Name: /delphix/hellodb/datafile/init.ora.createControlfile
  RMAN-07517: Reason: The file header is corrupted
File Name: /delphix/hellodb/datafile/VACSES/onlinelog/o1_mf_2_hovnhn73_.log
  RMAN-07529: Reason: catalog is not supported for this file type
.........
BEGIN SYS.DBMS_BACKUP_RESTORE.switchtocopy(68, 1051105764, false); END;
*
ERROR at line 1:
ORA-19654: must use backup control file to switch file incarnations
ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 4473
ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 4467
ORA-06512: at line 1

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

Resolution

There are two options to workaround this: 

1. Add the "oracle" user to this environment, then provision the VDB using that "oracle" user rather than the "delphix" user. To do this, you need to provision a new VDB and would not be able to refresh an existing VDB.
2. Take a new snapshot of the source VDB, and replicate this to the target Delphix environment. Then, attempt the refresh from the latest snapshot. It will be successful. You can also check on the source VDB filesystem for the presence of these files and what processes are accessing them. Delphix support can assist with this task if required.