Provisioning fails with 'Cannot write to directory' (KBA1273)
KBA
KBA#1273Issue
Attempting to provision a virtual database (VDB) fails with:
Unable to push scripts to target host "<ip_or_hostname>": Cannot push provisioning script content to host <ip_or_hostname> [<ip_or_hostname>]: Cannot write to directory "/oracle/delphix/toolkit/Delphix_4211234b_a123_a123_1ad2_4de12e3b1234_oracle_host/tmp" on host "<ip_or_hostname>" as user "delphix_os". Error Code exception.oracle.target.push.failed Suggested Action Make sure that the directory exists on the target host and is writable by "delphix_os".or
Cannot extract archive file b259c5b4-beb8-43c9-8cd9-ed26063c4c41_scripts_89608cdb-9f05-4a23-8cf1-f493bc68047c.tar from /tmp/b259c5b4-beb8-43c9-8cd9-ed26063c4c41_scripts_89608cdb-9f05-4a23-8cf1-f493bc68047c.tar to /work/Delphix_COMMON_e68430f09c3a_ed26063c4c41_host/tmp/b259c5b4-beb8-43c9-8cd9-ed26063c4c41_scripts_89608cdb-9f05-4a23-8cf1-f493bc68047c.tar on host rhel-79-orac-9559-lab-8-63d20ac8.dlpxdc.co [rhel-79-orac-9559-lab-8-63d20ac8.dlpxdc.co] Error Code exception.dhm.toolkit.unarchive.failed
Troubleshooting a Failed VDB Provision
When this error is encountered, the typical troubleshooting process includes checking the Delphix toolkit filesystem to see if it is full, verifying that the delphix_os user has write permissions to the Delphix toolkit directory and the subdirectories and confirming that the delphix_os user can create files in the directory specified in the fault.
Begin by ssh'ing, as the delphix_os user, to the target host specified in the fault
ssh delphix_os@<ip_or_hostname>
Checking if the filesystem is full?
cd /oracle/delphix/toolkit df -h . Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg00-lv_oracle 40G 22G 16G 59% /oracle
If the filesystem is not full, it is worth checking to see if there are available inodes.
Are there any remaining inodes?
In the example below, all of the inodes have been allocated:
# df -Ti /oracle
Filesystem Type Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg00-lv_oracle
ext2 190848 190848 0 100% /oracle
Any attempt to create more files will fail:
$ cd /oracle/delphix/toolkit $ ls -lart total 20 drwxr-xr-x 3 delphix_os dba 4096 Jun 11 2014 .. -rw-r--r-- 1 delphix_os dba 2153 Jun 25 2014 .delphix_profile drwxrwxr-x 6 delphix_os dba 4096 Aug 23 12:33 Delphix_COMMON_4219666b_a172_a514_3ad7_4de52e3b6800_oracle_host drwxrwx--- 4 delphix_os dba 4096 Aug 23 12:34 . drwxrwx--- 6 delphix_os dba 4096 Aug 23 13:15 Delphix_4219666b_a172_a514_3ad7_4de52e3b6800_oracle_host $ touch xx touch: cannot touch `xx': No space left on device
Resolution
If there is space available in the filesystem, but there are insufficient inodes available to write to the filesystem, it may be necessary to remove some files in order to free up some inodes.
To find where all the inodes are being used:
cd /oracle How many K bytes does each subdirectory use du -ks * | sort -n -b | column -t Find the directory that is the largest cd <largest_subdirectory> How many files are in each subdirectory for dp in *; do [ -d $dp ] && echo $(find $dp|wc -l) $dp; done | sort -n -b | column -t