TB088 Delphix Engine May Become Permanently Unmanageable
Alert Type
Availability, Data Loss
Impact
In rare cases following an error during upgrade, a Delphix Engine may experience a significant loss of configuration data that may cause the affected engine to enter a persistently degraded state or to become unmanageable. This can include:
-
Loss of visibility to prior snapshots or other objects created before the loss of configuration data
-
Loss of prior configuration changes or updates
-
Inability to complete operations through the Delphix Admin Application, CLI, or APIs, for example, creating new Snapshots, refreshing Virtual Databases (VDBs), and executing Self-Service operations
-
Replication jobs may not run or may not complete
-
Inability to login or access the Delphix Admin and Setup Applications
Although running VDBs are not directly impacted by the problem, VDB access could be impacted if a VDB was disabled or stopped by administrator action and a subsequent enable or start operation cannot be completed.
Because the system can be unmanageable, access to currently unused snapshots, timeflows, and bookmarks may be lost and unavailable for refreshes, rewinds, or provisions.
An impacted engine is not repairable, because there is no way to recover lost configuration data. The only way to restore normal operation is to deploy a new Delphix engine with a re-created configuration.
Contributing Factors
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1
The issue cannot occur if a Delphix engine is being upgraded from a release or to a release that is not enumerated above. For example, an upgrade from Delphix 5.3.9.0 to 6.0.5.0 is not susceptible, because 5.3.9.0 is not an affected version.
The issue occurs rarely. Although still rare, the issue is thought to be more likely to occur on Delphix Virtualization Engines, especially where Windows-based cluster environments are present in the configuration.
The issue can only be triggered by a special upgrade-related job of type POST_UPGRADE_CLEANUP experiencing an error (see symptoms below) or by the job being explicitly canceled by a Delphix administrator. Further, the issue can only occur after first being triggered by an abnormal completion of this job and a subsequent reboot of the affected Delphix engine.
Symptoms
Determining if a susceptible upgrade occurred
-
Log in to the Delphix Setup application as a system administrator.
-
From the Dashboard panel, select the View label on the Software Version panel.
- On the left side of the display, the Upgrade Images panel will be displayed showing current and previous versions. If the Running Version and most recent Previous version are both susceptible releases (see Contributing Factors), an upgrade has occurred that is susceptible to the issue.
In the example below, the system is currently running the 6.0.5.0 release, and the most recent previous release is 6.0.4.0. Since both of these releases are susceptible, a susceptible upgrade has occurred on this system.
Determining if an error occurred during a susceptible upgrade
-
Log in into the Delphix Management app using a user with an administrator role.
-
Navigate to the System -> Jobs display.
-
Select a custom range beginning on March 1, 2020 and extending to the current time.
-
Examine the output for any jobs of Type POST_UPGRADE_CLEANUP that have a status other than COMPLETED.
Note: you may enter the string "POST_UPGRADE" in the filter field to reduce the number of jobs shown.
Relief/Workaround
Do not upgrade a Delphix engine from an affected release to an affected release.
If you believe a Delphix Engine has already been upgraded and is susceptible, check to see if a job of type POST_UPGRADE_CLEANUP has completed abnormally. If so:
-
Contact Delphix Support Immediately.
-
Do not reboot the affected Delphix Engine.
-
Defer any configuration changes or other operations until after Delphix Support has been engaged.
Resolution
The issue is resolved in the Delphix 6.0.7.0 software release.