TB033 Device Removal May Lead to Delphix Engine Crash

Impact

The Delphix Engine includes the Delphix Storage Migration feature that allows for the removal of storage devices if they are no longer needed. During or following a device removal using this feature, the Delphix Engine may crash and restart unexpectedly. The system will normally return to service without manual operation; however, Delphix jobs running at the time of the failure, e.g. SnapSync or Provision jobs, may be interrupted and may need to be manually restarted.

Delphix Engine crashes can cause temporary loss of access to Virtual Databases (VDBs), or could cause VDBs to crash.

Recurring Delphix Engine crashes are possible. 

Contributing Factors

The problem can only occur in the Delphix Engine Releases in which the Delphix Storage Migration Feature is available, including all of the following:

• Delphix Engine 4.2.0.0 and 4.2.0.3

• Delphix Engine 4.2.1.0 and 4.2.1.1

• Delphix Engine 4.2.2.0 and 4.2.2.1

• Delphix Engine 4.2.3.0

• Delphix Engine 4.2.4.0

• Delphix Engine 4.2.5.0 and 4.2.5.1

• Delphix Engine 4.3.1.0

• Delphix Engine 4.3.2.0 and 4.3.2.1

• Delphix Engine 4.3.3.0

• Delphix Engine 4.3.4.0 and 4.3.4.1

• Delphix Engine 4.3.5.0

• Delphix Engine 5.0.1.0 and 5.0.1.1

• Delphix Engine 5.0.2.0 and 5.0.2.1 and 5.0.2.2

• Delphix Engine 5.0.3.0 and 5.0.3.1
• Delphix Engine 5.0.4.0 and 5.0.4.1

The problem can only occur during removal or following the successful removal of a storage device using the Delphix Storage Migration feature.

The problem is typically associated with the use of one or more VDBs subsequent to completion of the device removal feature. If all VDBs have been in use following a device removal, the probability of the issue occurring rapidly decreases over time. Likewise, the problem is more likely to occur as VDBs are first used following a device removal.

If a Delphix Engine experiences this problem, multiple recurrences are likely. 

Storage devices added by the hypervisor to the Delphix guest machine but not configured in the Delphix storage pool as data devices cannot contribute to this problem. See Adding and Expanding Storage Devices for more information on the procedure for configuring devices into a Delphix storage pool. 

Symptoms

• Following an unexpected reboot, an alert may be be created with the following descriptive text:
  
  Unexpected server restart   The server is starting up following an unexpected shutdown around <date>. Contact Delphix Support
   
• Delphix jobs running at the time of the failure may fail with the alert:
  
  <job_type> for "<object>" failed due to server restart during execution
   
• When a reboot occurs VDBs may experience temporary suspension of service. SQL Server VDBs may be inaccessible until they are restarted. In the log of affected Oracle target hosts, messages like:
  
  NFS server <ip address> not responding
  
  may be seen on the console or in the system log, where <ip address> is the network address of the affected Delphix Engine. 

Relief/Workaround

Avoid using the device removal feature on affected Delphix Engine releases.

Resolution

The issue is fully resolved in Delphix OS Versions, 5.0.2016.08.23, 5.1.2016.09.01, and later releases. These OS versions were delivered in Delphix Engine releases 5.0.5.1 and 5.1.2.0, respectively. 

Fresh installations and full upgrades will be running the Delphix OS version specified as included OS Version in the table below.  For Delphix Engine releases where the Included OS Version and Minimum OS Version are different, it's possible that a Deferred Upgrade was used and that the affected Delphix Engine could be running an older operating system version. See the "Deferred OS Upgrade" section of "Upgrading to a New Version of the Delphix Engine"  for information about how to determine the current OS version of a Delphix Engine. 

Additional Information

Delphix Storage Migration

Adding and Expanding Storage Devices