TB081 Delphix Engine API and User Interface May Become Non-Responsive

Impact

The internal application on the Delphix Engine that supports the browser interface (GUI), command line interface (CLI) and Web APIs may restart unexpectedly or become unresponsive.  Virtual Databases (VDBs) will continue to run, but it may become impossible to administer the Delphix Engine. Scheduled jobs, e.g. including SnapSync jobs that generate new snapshots from dSources may be delayed, may not run, or may not complete. Other administration functions of the Delphix Engine may be similarly affected.

Contributing Factors

The issue can only occur when running the following version of Delphix software:

The issue is only known to affect Delphix Virtualization Engines. Masking Engines are not susceptible to the issue. 

The issue is more likely to occur under heavier loads, including:

• Engines sustaining high Virtual Database (VDB) I/O activity.
• Engines where there are a larger number of VDBs or Sources, e.g. more than dozens of objects.
• Engines running many simultaneous jobs, e.g. SnapSync, Provision, or refresh.

Symptoms

• Existing sessions using the Delphix Admin or System Setup applications may become unresponsive.
• Attempts to login to an affected Delphix Engine may cause the browser to continually load or become unresponsive, and new sessions may fail.
• Commands issued in the CLI may not complete.
• Delphix jobs, e.g. SnapSync, running at the time an issue occurs may fail and may not automatically restart. Jobs may be failed with an error similar to:

  <job_type> for "<object>" failed due to server restart during execution

Relief/Workaround

If a Delphix Engine becomes non-responsive due to this problem, it should still be possible to manually restart the management service to restore access to the system:

1. ssh into the delphix Engine using a user credential with the system administrator role. (Windows users may use an ssh utility like putty)
2. Run the following CLI command

   system restart ; commit
   

   Command output similar to the following should be seen

   delphix> system restart ; commit
   Restarting the management service. The current session will be re-established once the "service is available.
   

Related Documents

Starting, Stopping, and Restarting Your Engine (see Restarting the Delphix Engine Management Process)

Managing System Administrators