Skip to main content
Delphix

Provisioning or Refreshing Virtual Databases Using the CLI Fails After Approximately 30 Minutes (KBA6562)

 

KBA

KBA# 6562

 

Issue

When the provisioning or refreshing of Delphix Virtual Databases is performed using the Delphix Command Line Interface (CLI) and the provision time exceeds 30 minutes in duration, it is possible for the Delphix CLI session to timeout.  This timeout is defined at a user level in the Delphix Virtualization Engine with the default being 30 minutes for a CLI session to wait on a task completing.

Prerequisites

Provision or refresh of Virtual Databases is being performed through the Delphix Command Line Interface (CLI) or a CLI-based script.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases
6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Troubleshooting

The following example provides the behavior seen during an Oracle Virtual Database provision attempt made through a unix shell script that is logging into the Delphix CLI and issuing commands to kick off the provision. These commands or database platform specifics are not the focus of this knowledge base article.  What is examined here are some of the symptoms that appear that will allow you to identify this CLI timeout appearing in your own job execution.

The example demonstrates how an Oracle recovery of a VDB taking longer than 30 minutes to complete can result in the CLI timeout being reached and the CLI session being terminated. The script performing the provision which also has post provision tasks occurring understands the time out as a sign the provision has completed and performs further work, one component of which is to shut the VDB down.  This results in the VDB being shutdown in the middle of recovering the VDB creating a failure in the overall provision process.  The timeout in this case was increased to prevent the CLI from timing out and the provision script from moving on and performing further work that potentially impacts the provision itself. 

The CLI script output shows something similar to the following where a session timeout is reported.

    `ORACLE_DB_CONTAINER-446
    Dispatched job JOB-4912
    DB_PROVISION job started for "MyVDBs/vdb1".
    Starting provision of the virtual database "vdb1".
..SNIP..
    Recovering Oracle database with created control file.
The current session has timed out, exiting.
..SNIP

Examining the VDB alert log shows recovery starting at 17:32 and prior to Oracle indicating it has completed the VDB is shutdown at 18:03.  This shutdown has occurred as the provision appears to the script to have finished due to the CLI session timeout and the script has continued. The shutdown then results in the overall provision failure.

Fri Aug 28 17:32:00 2020
 alter database recover automatic  database  until change 6649954392923 using backup controlfile
Media Recovery Start
..SNIP..
Fri Aug 28 17:59:56 2020
Media Recovery Log /mnt/delphix/vdb1/source-archive/arch_1_15146_1024433369.log
Fri Aug 28 18:02:59 2020
Media Recovery Log /mnt/delphix/vdb1/source-archive/arch_1_15147_1024433369.log

Fri Aug 28 18:03:18 2020
Shutting down instance (immediate)
Shutting down instance: further logons disabled
Stopping background process MMNL
Fri Aug 28 18:03:18 2020
Errors with log /mnt/delphix/vdb1/source-archive/arch_1_15147_1024433369.log
Stopping background process MMON
License high water mark = 19
Fri Aug 28 18:03:44 2020
Recovery interrupted!
Fri Aug 28 18:05:12 2020
Recovered data files to a consistent state at change 6649812038064
Media Recovery failed with error 448
Errors in file /u01/app/oracle/diag/rdbms/vdb1/vdb1/trace/vdb1_pr00_4354.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process

Resolution

There are a few options available for resolution when this behavior is encountered.  Utilizing the API and programmatically checking for a successful provision before moving on to other post provision processing as opposed to scripting through the CLI and increasing the CLI timeout is a far better approach to both provisioning VDBs and checking the state of the VDB provision.  If utilizing the API is not possible, the following alternative solutions are offered.

 

For job execution with an inherent variability in time-to-completion, the job can be executed asynchronously, by altering the wait option in the CLI session. 

setopt wait=false

This feature is detailed in the Delphix CLI documentation:

https://docs.delphix.com/docs/developer-s-guide/command-line-interface-guide/delphix-objects/asynchronous-jobs

If interactive CLI job execution is required (ie, the session waits for the job to complete), the user CLI timeout can be adjusted / extended for the user performing the provisioning . Exactly what value needs to be set for this is dependent on the maximum time a provision may take and will differ from site to site and potentially across provisions. In this case the user performing the provision is the Delphix user named "admin" and the timeout for this user is increased from the default of 30 minutes to a value of 60 minutes through the Delphix Administration CLI.

Delphix5390> user
Delphix5390 user> ls
Objects
NAME   EMAILADDRESS            
admin  p.b@delphix.com
..SNIP..
Delphix5390 user> select admin
Delphix5390 user 'admin'> ls
Properties
    type: User
    name: admin
    apiUser: true
    authenticationType: NATIVE
    emailAddress: p.b@delphix.com
    enabled: true
    firstName: (unset)
    homePhoneNumber: (unset)
    isDefault: true
    lastName: (unset)
    locale: en-US
    mobilePhoneNumber: (unset)
    passwordUpdateRequest: NONE
    principal: admin
    publicKey: (empty)
    reference: USER-2
    sessionTimeout: 30min
    userType: DOMAIN
    workPhoneNumber: (unset)
..SNIP..
updateCredential
Delphix5390 user 'admin'> update
Delphix5390 user 'admin' update *> set sessionTimeout=60
Delphix5390 user 'admin' update *> commit

The extension of CLI session timeout may have other security implications, as some organizations will have a maximum timeout requirement for remote sessions.