Skip to main content
Delphix

TB029 Delphix Engine May Hang During Heavy I/O Load

 

 

Alert Type

Availability

Impact

The Delphix Engine employs a large filesystem cache as a component of its I/O engine. As a result of cache activity, in some circumstances the Delphix Engine's kernel memory may become fragmented. If the fragmentation is sufficiently severe, the entire Delphix Engine may persistently hang. This results both in an inability to manage the Delphix Engine and in a disruption of any I/O to all virtual databases (VDBs) hosted on the affected Delphix Engine.

Contributing Factors

The issue may occur in the following Delphix Engine Releases:

  • Delphix Engine 4.3.1.0
  • Delphix Engine 4.3.2.0 and 4.3.2.1
  • Delphix Engine 4.3.3.0
  • Delphix Engine 4.3.4.0 and 4.3.4.1

The issue is more likely to occur in Delphix Engines hosted in virtual guest machines configured with a large amount of memory, especially configurations with larger than 256GB of memory. 

The issue is more likely to occur as I/O activity increases.

The issue is more likely to occur when using DB2 dSources and VDBs.

Symptoms

An affected Delphix Engine will exhibit some or all of the following:

  • The system may not respond to network requests, including ICMP "pings"
  • Existing GUI and CLI sessions will hang
  • New GUI and CLI sessions will fail or timeout
  • WebAPI calls will fail
  • VDBs on the affected Delphix Engine will become unresponsive and may crash

Relief/Workaround

Immediate relief is possible by rebooting the Delphix Engine. When in a hung state, the affected Delphix Engine can only be rebooted from the hypervisor (e.g. VMware/ESXi) hosting the Delphix Engine virtual guest machine.   

Resolution

The issue is fully resolved in Delphix Engine OS Version 4.3.2016.02.06, and later OS releases.  

Fresh Installations and Full Upgrades to Delphix Engine 4.3.5.0 will run OS Version 4.3.2016.02.06.

Deferred OS upgrades to Delphix 4.3.5.0 will run a prior version of the Delphix OS and are still susceptible to the problem described in this bulletin. 

Fresh installations and upgrades to Delphix Engine 5.0.1.0 will be running OS Version 5.0.2016.01.28 and are not susceptible to this issue. 

See the "Deferred OS Upgrade" section of "Upgrading to a New Version of the Delphix Engine"  for information about how to determine the current OS version of a Delphix Engine. 

Delphix Engine Release Included OS Version Minimum OS Version
4.3.5.0 4.3.2016.02.06 4.3.2015.08.12 
5.0.1.0

5.0.2016.01.28

5.0.2016.01.28

Additional Information

The only way to determine the root cause of the hang resulting from this problem is to obtain an operating system crash dump from the Delphix Engine. Please see the KB article How to Generate an NMI for more information, or contact Delphix Support.