Provisioning or Refreshing Virtual Databases Using the CLI Fails After Approximately 30 Minutes (KBA6562)
KBA
KBA# 6562
Issue
When the provisioning or refreshing of Delphix Virtual Databases is performed using the Delphix Command Line Interface (CLI) and the provision time exceeds 30 minutes in duration, it is possible for the Delphix CLI session to timeout. This timeout is defined at a user level in the Delphix Virtualization Engine with the default being 30 minutes for a CLI session to wait on a task completing.
Prerequisites
Provision or refresh of Virtual Databases is being performed through the Delphix Command Line Interface (CLI) or a CLI-based script.
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0 5.3
5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0 5.2
5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1
5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0
5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
Troubleshooting
The following example provides the behavior seen during an Oracle Virtual Database provision attempt made through a unix shell script that is logging into the Delphix CLI and issuing commands to kick off the provision. These commands or database platform specifics are not the focus of this knowledge base article. What is examined here are some of the symptoms that appear that will allow you to identify this CLI timeout appearing in your own job execution.
The example demonstrates how an Oracle recovery of a VDB taking longer than 30 minutes to complete can result in the CLI timeout being reached and the CLI session being terminated. The script performing the provision which also has post provision tasks occurring understands the time out as a sign the provision has completed and performs further work, one component of which is to shut the VDB down. This results in the VDB being shutdown in the middle of recovering the VDB creating a failure in the overall provision process. The timeout in this case was increased to prevent the CLI from timing out and the provision script from moving on and performing further work that potentially impacts the provision itself.
The CLI script output shows something similar to the following where a session timeout is reported.
`ORACLE_DB_CONTAINER-446
Dispatched job JOB-4912
DB_PROVISION job started for "MyVDBs/vdb1".
Starting provision of the virtual database "vdb1".
..SNIP..
Recovering Oracle database with created control file.
The current session has timed out, exiting.
..SNIP
Examining the VDB alert log shows recovery starting at 17:32 and prior to Oracle indicating it has completed the VDB is shutdown at 18:03. This shutdown has occurred as the provision appears to the script to have finished due to the CLI session timeout and the script has continued. The shutdown then results in the overall provision failure.
Fri Aug 28 17:32:00 2020 alter database recover automatic database until change 6649954392923 using backup controlfile Media Recovery Start ..SNIP.. Fri Aug 28 17:59:56 2020 Media Recovery Log /mnt/delphix/vdb1/source-archive/arch_1_15146_1024433369.log Fri Aug 28 18:02:59 2020 Media Recovery Log /mnt/delphix/vdb1/source-archive/arch_1_15147_1024433369.log Fri Aug 28 18:03:18 2020 Shutting down instance (immediate) Shutting down instance: further logons disabled Stopping background process MMNL Fri Aug 28 18:03:18 2020 Errors with log /mnt/delphix/vdb1/source-archive/arch_1_15147_1024433369.log Stopping background process MMON License high water mark = 19 Fri Aug 28 18:03:44 2020 Recovery interrupted! Fri Aug 28 18:05:12 2020 Recovered data files to a consistent state at change 6649812038064 Media Recovery failed with error 448 Errors in file /u01/app/oracle/diag/rdbms/vdb1/vdb1/trace/vdb1_pr00_4354.trc: ORA-00283: recovery session canceled due to errors ORA-00448: normal completion of background process
Resolution
There are a few options available for resolution when this behavior is encountered. Utilizing the API and programmatically checking for a successful provision before moving on to other post provision processing as opposed to scripting through the CLI and increasing the CLI timeout is a far better approach to both provisioning VDBs and checking the state of the VDB provision. If utilizing the API is not possible, the following alternative solutions are offered.
For job execution with an inherent variability in time-to-completion, the job can be executed asynchronously, by altering the wait option in the CLI session.
setopt wait=false
This feature is detailed in the Delphix CLI documentation:
If interactive CLI job execution is required (ie, the session waits for the job to complete), the user CLI timeout can be adjusted / extended for the user performing the provisioning . Exactly what value needs to be set for this is dependent on the maximum time a provision may take and will differ from site to site and potentially across provisions. In this case the user performing the provision is the Delphix user named "admin" and the timeout for this user is increased from the default of 30 minutes to a value of 60 minutes through the Delphix Administration CLI.
Delphix5390> user
Delphix5390 user> ls
Objects
NAME EMAILADDRESS
admin p.b@delphix.com
..SNIP..
Delphix5390 user> select admin
Delphix5390 user 'admin'> ls
Properties
type: User
name: admin
apiUser: true
authenticationType: NATIVE
emailAddress: p.b@delphix.com
enabled: true
firstName: (unset)
homePhoneNumber: (unset)
isDefault: true
lastName: (unset)
locale: en-US
mobilePhoneNumber: (unset)
passwordUpdateRequest: NONE
principal: admin
publicKey: (empty)
reference: USER-2
sessionTimeout: 30min
userType: DOMAIN
workPhoneNumber: (unset)
..SNIP..
updateCredential
Delphix5390 user 'admin'> update
Delphix5390 user 'admin' update *> set sessionTimeout=60
Delphix5390 user 'admin' update *> commit
The extension of CLI session timeout may have other security implications, as some organizations will have a maximum timeout requirement for remote sessions.