Oracle VDB Provision Failures Of Type "exception.oracle.targetscripts.environment.validate" (KBA9771)

Issue

When provisioning, refreshing, or enabling an Oracle Virtual Database (VDB), Delphix validates a number of required Environment conditions are met, including the ability to mount a temporary filesystem mounted from the Delphix Appliance, as well as subsequent permissions to navigate the given filesystem.

If the configured Delphix OS user is unable to successfully list the named file in the temporary mount point, and the environment user is not the oracle installation user, the following Environment validation error may be reported.

Example:

Error
Environment validation failed for environment "myTargetHost" host "myTargetHost". Cause: Validation of user environment failed.

Error Code
exception.oracle.vdb.environment.validate

Suggested Action
Correct the user environment error, or provide a privileged user, and try again.

Command Output
 exception.oracle.targetscripts.environment.validate {command: $DLPX_SHELL -x '/work/toolkit/Delphix_COMMON_c950e23f8227_a821e43288fc_36_host/scripts/provision/oracle/common/validate_provision_reqs' '/work/toolkit/Delphix_COMMON_c950e23f8227_a821e43288fc_36_host' '/mnt/provision' 'VDBS_7NJ' '10.1.1.217' '/public/ORACLE_VIRTUAL_SOURCE-55-iL3Ts' 'VDBS_7NJ' '_NOT_PDB_' 'true' '4' '1001' 'VDBS_7NJ_2400266547161004133_VALIDATE'
output:Enter the password for user "delphix_os" (Hit return if no password is needed) :
stty: standard input: Inappropriate ioctl for device
stty: standard input: Inappropriate ioctl for device
Checking if 10.1.1.217:/public/ORACLE_VIRTUAL_SOURCE-55-iL3Ts is mounted to /mnt/provision/VDBS_7NJ
Trying "umount" using dlpx_pfexec -
umount: /mnt/provision/VDBS_7NJ: mountpoint not found
Trying "mkdir" without dlpx_pfexec -
mkdir: cannot create directory ‘/mnt/provision/VDBS_7NJ’: Permission denied
Trying "mkdir" using dlpx_pfexec -

Trying "mount" using dlpx_mount -

ls: cannot access /mnt/provision/VDBS_7NJ/VDBS_7NJ_2400266547161004133_VALIDATE: Permission denied
Trying "umount" using dlpx_pfexec -

#####DELPHIX_START_DATA#####
REQUIRED.BACKDOOR_USED=no
REQUIRED.CURRENT_USER=delphix_os
REQUIRED.USER_CAN_MOUNT=no
REQUIRED.USER_CAN_UNMOUNT=yes
REQUIRED.USER_CAN_CD_MOUNT_BASE=yes
REQUIRED.SID_IN_USE=no
SYSTEM.ULIMITS_VAL=65536
#####DELPHIX_END_DATA#####
#####DELPHIX_START_ERROR#####
ERROR_CODE=5
ERROR : Could not verify file ownership under mount for user  "delphix_os"
ERROR : Details : ls: cannot access /mnt/provision/VDBS_7NJ/VDBS_7NJ_2400266547161004133_VALIDATE: Permission denied
#####DELPHIX_END_ERROR#####
} 

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Date	Release
Mar 13, 2023	9.0.0.0, 9.0.0.1
Feb 13, 2023	8.0.0.0
Jan 12, 2023	7.0.0.0
Releases Prior to 2023
Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2

Troubleshooting

Reviewing the message block between the DELPHIX_START_ERROR header and DELPHIX_END_ERROR footer will provide the specific Environment Validation failure, which explicitly states the OS user cannot access the *_VALIDATE file in the NFS mount.

#####DELPHIX_START_ERROR#####
ERROR_CODE=5
ERROR : Could not verify file ownership under mount for user  "delphix_os"
ERROR : Details : ls: cannot access /mnt/provision/VDBS_7NJ/VDBS_7NJ_2400266547161004133_VALIDATE: Permission denied
#####DELPHIX_END_ERROR#####

This event also confirms the issue is not related to NFS mount permissions, but rather the permissions within the filesystem.

Examples are provided below, with command output from both an example Delphix OS user (delphix_os)and the oracle install user:

[oracle@myTargetHost}$ ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x. 1 oracle asmadmin 323613264 Aug 30 12:53 /u01/app/oracle/product/12.1.0.2/dbhome_1/bin/oracle

[oracle@myTargetHost]$ id 
uid=1001(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(asmdba),1003(asmoper),1005(dba),1006(oper),1007(backupdba),1008(dgdba),1009(kmdba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[oracle@myTargetHost]$ groups 
oinstall asmdba asmoper dba oper backupdba dgdba kmdba

[delphix_os@myTargetHost]$ id 
uid=1001(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(asmdba),1003(asmoper),1005(dba),1006(oper),1007(backupdba),1008(dgdba),1009(kmdba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[delphix_os@myTargetHost]$ groups 
oinstall asmdba asmoper dba oper backupdba dgdba kmdba

Resolution

To resolve this behavior, an admin must confirm the Delphix OS user matches the initial login group and supplementary groups as the oracle user and if not present, a secondary group membership of the oracle binary found at $ORACLE_HOME/bin/oracle.  As this often requires elevated privileges to add or modify OS user group membership, this issue can not be resolved by the Delphix OS user, but rather some other admin user (root, etc).

[root@myTargetHost]# usermod -a -G asmadmin delphix_os

Following this change, confirm the supplemental group addition with the same OS commands (id, groups, etc):

[delphix_os@myTargetHost]$ id
uid=1001(oracle) gid=1001(oinstall) groups=1001(oinstall),1002(asmdba),1003(asmoper),1004(asmadmin),1005(dba),1006(oper),1007(backupdba),1008(dgdba),1009(kmdba) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

[delphix_os@myTargetHost]$ groups
oinstall asmdba asmoper asmadmin dba oper backupdba dgdba kmdba

Once the change is applied, a subsequent dataset provision/refresh/enable should be successful.