Skip to main content
Delphix

Provisioning fails with 'Cannot write to directory'

Troubleshooting a failed VDB provision

Issue

Attempting to provision a virtual database (VDB) fails with:

Unable to push scripts to target host "<ip_or_hostname>": Cannot push provisioning script content to host <ip_or_hostname> [<ip_or_hostname>]: Cannot write to directory "/oracle/delphix/toolkit/Delphix_4211234b_a123_a123_1ad2_4de12e3b1234_oracle_host/tmp" on host "<ip_or_hostname>" as user "delphix_os".
 
Error Code 
exception.oracle.target.push.failed
 
Suggested Action 
Make sure that the directory exists on the target host and is writable by "delphix_os".

Troubleshooting

When this error is encountered, the typical troubleshooting process includes checking the Delphix toolkit filesystem to see if it is full, verifying that the delphix_os user has write permissions to the Delphix toolkit directory and the subdirectories and confirming that the delphix_os user can create files in the directory specified in the fault.

Begin by ssh'ing, as the delphix_os user, to the target host specified in the fault 

ssh delphix_os@<ip_or_hostname>

Is the filesystem is full?

cd /oracle/delphix/toolkit
df -h .
 
Filesystem Size Used Avail Use% Mounted on 
/dev/mapper/vg00-lv_oracle 40G 22G 16G 59% /oracle

If the filesystem is not full, it is worth checking to see if there are available inodes.

Are there any remaining inodes?

In the example below, all of the inodes have been allocated:


# df -Ti /oracle
Filesystem    Type    Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg00-lv_oracle
              ext2    190848  190848       0  100% /oracle

Any attempt to create more files will fail:

$ cd /oracle/delphix/toolkit
$ ls -lart
total 20
drwxr-xr-x 3 delphix_os dba 4096 Jun 11 2014 ..
-rw-r--r-- 1 delphix_os dba 2153 Jun 25 2014 .delphix_profile
drwxrwxr-x 6 delphix_os dba 4096 Aug 23 12:33 Delphix_COMMON_4219666b_a172_a514_3ad7_4de52e3b6800_oracle_host
drwxrwx--- 4 delphix_os dba 4096 Aug 23 12:34 .
drwxrwx--- 6 delphix_os dba 4096 Aug 23 13:15 Delphix_4219666b_a172_a514_3ad7_4de52e3b6800_oracle_host
$ touch xx
touch: cannot touch `xx': No space left on device

Resolution

If there is space available in the filesystem, but there are insufficient inodes available to write to the filesystem, it may be necessary to remove some files in order to free up some inodes. 

Do the following to find where all the inodes are being used:

cd /oracle

How many K bytes does each subdirectory use
du -ks * | sort -n -b | column -t

Find the directory that is the largest
cd <largest_subdirectory>

How many files are in each subdirectory
for dp in *; do [ -d $dp ] && echo $(find $dp|wc -l) $dp; done | sort -n -b | column -t

Hint: Check the audit directory as it can grow quite large and exhaust available inodes.