Removing Temporary NFS Files That Are Left In The Filesystem (KBA6534)
KBA
KBA# 6534Issue
If you drop a tablespace and create datafiles that reuse the datafile numbers of the dropped datafile, it is possible that temporary NFS files are left in the filesystem. When that happens, during doRenameDatafiles.sh in VDB provisioning/refresh/rewind, Delphix finds these invalid copies. This causes a problem, as it results in having two copies for the same datafile. And, depending on the order in which they are found, Delphix might end up cataloging the NFS temporary file instead of the current datafile, causing a RESETLOGS error mismatch.
Normally Delphix would remove the .nfs temporary files, but when the files are owned by the "oracle" OS user, Delphix cannot delete them. Instead, a 'Permission denied' error is provided (as shown below). The Delphix scripts used to remove these files are run by the "delphix" user.
[2020-09-14 13:49:32,911][INFO][jcm.impl.JobInstance#execute:355][Worker-3474|JOB-1848|DB_REFRESH(ORACLE_DB_CONTAINER-69)][] job failed with user error com.delphix.common.exception.messages.OracleTargetscriptsExceptions$RenameDatafiles: exception.oracle.targetscripts.rename.datafiles {command: umask 027; . $DB_SCRIPT_DIR/setup-oraenv.sh; $DLPX_SHELL $DB_SCRIPT_DIR/doRenameDatafiles.sh $DB_SCRIPT_DIR "$LOGON_STR" "/delphix/hellodb/datafile" output:END_OF_SETUP END_OF_SETUP Check if /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000001200000026 is a datafile rm: cannot remove '/delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000001200000026': Permission denied Check if /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000002100000030 is a datafile rm: cannot remove '/delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000002100000027': Permission denied ........ NUMBER_OF_DATAFILES_MDS=53, NUMBER_OF_FILES_IN_FS=70 Extra files in directory remaining: /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000001200000026 /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000002100000030 ........ List of Files Unknown to the Database ===================================== File Name: /delphix/hellodb/datafile/spfile.ora File Name: /delphix/hellodb/datafile/init.ora.createControlfile File Name: /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs000000000000004500000030 File Name: /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/acses_ts_encrypt.278.1036365707 File Name: /delphix/hellodb/datafile/+DG_ACSES/ACSESDG/DATAFILE/.nfs00000000000000870000002d ......... File Name: /delphix/hellodb/datafile/init.ora.rename File Name: /delphix/hellodb/datafile/init.ora.recovery File Name: /delphix/hellodb/datafile/source_init.ora cataloging files... cataloging done ......... List of Files Which Were Not Cataloged ======================================= File Name: /delphix/hellodb/datafile/spfile.ora RMAN-07518: Reason: Foreign database file DBID: 0 Database Name: File Name: /delphix/hellodb/datafile/init.ora.createControlfile RMAN-07517: Reason: The file header is corrupted File Name: /delphix/hellodb/datafile/VACSES/onlinelog/o1_mf_2_hovnhn73_.log RMAN-07529: Reason: catalog is not supported for this file type ......... BEGIN SYS.DBMS_BACKUP_RESTORE.switchtocopy(68, 1051105764, false); END; * ERROR at line 1: ORA-19654: must use backup control file to switch file incarnations ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 4473 ORA-06512: at "SYS.X$DBMS_BACKUP_RESTORE", line 4467 ORA-06512: at line 1
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1 5.3
5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
Resolution
There are two options to workaround this:
- Add the "oracle" user to this environment, then provision the VDB using that "oracle" user rather than the "delphix" user. To do this, you need to provision a new VDB and would not be able to refresh an existing VDB.
- Take a new snapshot of the source VDB, and replicate this to the target Delphix environment. Then, attempt the refresh from the latest snapshot. It will be successful. You can also check on the source VDB filesystem for the presence of these files and what processes are accessing them. Delphix support can assist with this task if required.
Related Articles
The following articles may provide more information or related information to this article: