Skip to main content
Delphix

Device Removal if Used After Upgrade to Specific Delphix Versions May Lead to Delphix Engine Crash (TB035)

 

Alert Type

Availability

Impact

The Delphix Engine, as of Delphix Engine 4.3.1.0, has included the Delphix Storage Migration a feature that allows for the removal of attached storage devices. While using this feature, the Delphix Engine may crash and restart unexpectedly. The system will normally return to service without manual operation; however, Delphix jobs running at the time of the failure, e.g. SnapSync or Provision jobs, may be interrupted and may need to be manually restarted.

Delphix Engine crashes can cause temporary loss of access to Virtual Databases (VDBs), or could cause VDBs to crash.

Recurring Delphix Engine crashes are possible. 

Contributing Factors

The problem can only occur in the Delphix Engine Releases in which the Delphix Storage Migration Feature is available, including all of the following:

  • Delphix Engine 4.3.1.0

  • Delphix Engine 4.3.2.0 and 4.3.2.1

  • Delphix Engine 4.3.3.0

  • Delphix Engine 4.3.4.0 and 4.3.4.1

  • Delphix Engine 4.3.5.0

  • Delphix Engine 5.0.1.0 and 5.0.1.1

  • Delphix Engine 5.0.2.0 and 5.0.2.1 and 5.0.2.2 and 5.0.2.3

  • Delphix Engine 5.0.3.0 and 5.0.3.1
  • Delphix Engine 5.0.4.0 and 5.0.4.1 
     

    This problem can occur **during** the device removal process.  
    This problem only impacts Delphix Engines that were upgraded from releases prior to 4.3 to release 4.3 or more recent.

    This problem does not impact Delphix Engines that were installed on, or after, version 4.3.

    This problem only impacts Delphix Engines where there were no reboots, or disks added since the upgrade.

Symptoms

  • Following an unexpected reboot, an alert may be be created with the following descriptive text:

    Unexpected server restart   
    The server is starting up following an unexpected shutdown around <date>. 
    Contact Delphix Support
     
  • Delphix jobs running at the time of the failure may fail with the alert:

    <job_type> for "<object>" failed due to server restart during execution
     
  • When a reboot occurs VDBs may experience temporary suspension of service. SQL Server VDBs may be inaccessible until they are restarted. In the log of affected Oracle target hosts, messages like:

    NFS server <ip address> not responding

    may be seen on the console or in the system log, where <ip address> is the network address of the affected Delphix Engine. 

Relief/Workaround

Once this issue is encountered and the Delphix Engine has rebooted the issue should not present again.

Prior to a device removal operation Delphix Support can determine if the Delphix Engine is likely to encounter this issue.  If the Delphix Engine is likely to encounter this issue Delphix Support can also pro-actively take steps to prevent this issue.

Resolution

This issue has been resolved in Delphix Engine version 5.0.5.1 and Delphix Engine versions 5.1 and later