TB023 Delphix Admin Application (4.2.x) May Restart Unexpectedly
Alert Type
Availability
Impact
The internal application on the Delphix Engine that supports the browser interface, CLI, and Web APIs may run out of memory and restart unexpectedly. While this will not impact virtual databases, it may result in the abnormal termination of any Delphix Jobs/Actions, including long running actions like SnapSync, Replication, or Virtual-to-Physical (V2P). The problem may recur intermittently.
Contributing Factors
The issue may occur in the following Delphix Releases:
- Delphix Engine 4.2.1.0 and 4.2.1.1
- Delphix Engine 4.2.2.0 and 4.2.2.1
Symptoms
-
The error:
Delphix Engine Communication Error
may be seen either when navigating a browser to the Delphix Engine or when an existing Delphix Admin or Server Setup application is disrupted by the issue.
-
A Delphix Engine alert and accompanying email may occur with the following text:
The server is starting up following an unexpected shutdown around <date>."
or
The management service is starting up following an unexpected shutdown around <date>."
-
Jobs or Actions running at the time of failure will terminate abnormally. A Delphix Engine alert and accompanying email will be issued with the following text:
<job_type> for "<object>" failed due to an error during execution: <job_type> for <object> failed due to server restart during execution."
where <job_type> is the type of job running, for example, DB_REFRESH, DB_PROVISION, or DB_SYNC and <object> is the name of the Delphix group and database name for which the job was being processed.
Relief/Workaround
The Delphix Engine will automatically resume normal operations. Jobs/Actions that were aborted will not be automatically restarted.
Resolution
The issue is fully resolved in Delphix Engine 4.2.3.0 and later releases.
Using a deferred os upgrade, it’s possible to activate the 4.2.3.0 upgrade with this fix without shutting down VDBs. See the "Deferred OS Upgrade" section of "Upgrading to a New Version of the Delphix Engine" for more information. A deferred os upgrade will activate just the Delphix management software portion of the upgrade, while operating system fixes in the release will not be activated. For information detailing all fixes in the 4.2.3.0 release, please see the Release 4.2.x.x Known Issues and Changes document.