VPDB provisioning fails due to error "Failed to create directories for temporary tablespace files" (KBA10899)

Issue

Attempts to provision or refresh a VPDB may fail due to

 messageCode: exception.oracle.targetscripts.create.tempfile.dirs
        messageCommandOutput: Failed to create temporary directory "/mnt/provision/MWVPDB-MWVCDB/temp/MWVCDB/126846DA37E7305EE0638E362B0ABA53/datafile" on host mw-rac19191.dcol2.delphix.com (10.43.54.142) using user "dlpxqa":
/mnt/provision/MWVPDB-MWVCDB/temp/MWVCDB/126846DA37E7305EE0638E362B0ABA53/datafile does not exist.
/mnt/provision/MWVPDB-MWVCDB/temp/MWVCDB/126846DA37E7305EE0638E362B0ABA53 does not exist.
/mnt/provision/MWVPDB-MWVCDB/temp/MWVCDB exists.
/mnt/provision/MWVPDB-MWVCDB/temp/MWVCDB is not writable. <--- To fix: grant write permission to the user.

        messageDetails: Failed to create directories for temporary tablespace files.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Date	Release
Feb 21, 2024	20.0.0.0
Jan 25, 2024	19.0.0.0
Dec 20, 2023 | Jan 10, 2024	18.0.0.0 | 18.0.0.1
Nov 21, 2023	17.0.0.0
Oct 18, 2023	16.0.0.0
Sep 21, 2023	15.0.0.0
Aug 24, 2023	14.0.0.0
Jul 24, 2023	13.0.0.0
Jun 21, 2023	12.0.0.0
May 25, 2023	11.0.0.0
Apr 13, 2023	10.0.0.0 | 10.0.0.1
Mar 13, 2023 | Mar 20, 2023	9.0.0.0 | 9.0.0.1
Feb 13, 2023	8.0.0.0
Jan 12, 2023	7.0.0.0
Releases Prior to 2023
Major Release	All Sub Releases
6.0	6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2

Resolution

To resolve the issue, apply Oracle patch 35750253. If this is not available for your version of Oracle, please request a backport from Oracle.

Alternatively any of the following actions should also allow provisioning to be successful:

• Change the Delphix OS user to match the Oracle instance owner ( typically "oracle" ).
• Disable dNFS on the Oracle Home
• Take a fresh snapshot of the dSource, then provision/refresh from the latest snapshot.

Troubleshooting

The issue is caused by an unpublished Oracle bug 35750253. In these specific versions of Oracle, when a new directory is created by the instance on the NFS filesystem, it is created with the incorrect permissions.

An example can be seen below:

[oracle@mw-rac19191 datafile]$ ls -la
total 5
drwxr-x---. 5 oracle asmadmin 6 Mar  1 00:49 .
drwxr-x--x. 7 oracle asmadmin 7 Feb 29 22:36 ..
drwxr-x---. 3 oracle asmadmin 3 Feb 29 22:36 +DATA
drwxrwx--x. 2 oracle asmadmin 3 Feb 29 22:41 delphix_group_writable
-rw-r-----. 1 oracle asmadmin 0 Feb 29 22:36 includedParameters.ora
drwxr--rwx. 3 oracle asmadmin 3 Mar  1 00:49 MYCDB   <------ CREATED BY ORACLE

Note that the permissions on the directory created by the Oracle instance are incorrect, as it has been created with world read/write/execute, and importantly, no group execute permissions.  The correct permissions should be:

drwxr-x---. 3 oracle asmadmin 3 Mar  1 00:54 MYCDB 

This permissions issue prevents the Delphix workflows from interacting with any files created within this directory.