The Delphix Engine employs a large filesystem cache as a component of its I/O engine. As a result of cache activity, in some circumstances the Delphix Engine's kernel memory may become fragmented. If the fragmentation is sufficiently severe, the entire Delphix Engine may persistently hang. This results both in an inability to manage the Delphix Engine and in a disruption of any I/O to all virtual databases (VDBs) hosted on the affected Delphix Engine.
The issue may occur in the following Delphix Engine Releases:
- Delphix Engine 220.127.116.11
- Delphix Engine 18.104.22.168 and 22.214.171.124
- Delphix Engine 126.96.36.199
- Delphix Engine 188.8.131.52 and 184.108.40.206
The issue is more likely to occur in Delphix Engines hosted in virtual guest machines configured with a large amount of memory, especially configurations with larger than 256GB of memory.
The issue is more likely to occur as I/O activity increases.
The issue is more likely to occur when using DB2 dSources and VDBs.
An affected Delphix Engine will exhibit some or all of the following:
- The system may not respond to network requests, including ICMP "pings"
- Existing GUI and CLI sessions will hang
- New GUI and CLI sessions will fail or timeout
- WebAPI calls will fail
- VDBs on the affected Delphix Engine will become unresponsive and may crash
Immediate relief is possible by rebooting the Delphix Engine. When in a hung state, the affected Delphix Engine can only be rebooted from the hypervisor (e.g. VMware/ESXi) hosting the Delphix Engine virtual guest machine.
The issue is fully resolved in Delphix Engine OS Version 4.3.2016.02.06, and later OS releases.
Fresh Installations and Full Upgrades to Delphix Engine 220.127.116.11 will run OS Version 4.3.2016.02.06.
Deferred OS upgrades to Delphix 18.104.22.168 will run a prior version of the Delphix OS and are still susceptible to the problem described in this bulletin.
Fresh installations and upgrades to Delphix Engine 22.214.171.124 will be running OS Version 5.0.2016.01.28 and are not susceptible to this issue.
See the "Deferred OS Upgrade" section of "Upgrading to a New Version of the Delphix Engine" for information about how to determine the current OS version of a Delphix Engine.
|Delphix Engine Release||Included OS Version||Minimum OS Version|
The only way to determine the root cause of the hang resulting from this problem is to obtain an operating system crash dump from the Delphix Engine. Please see the KB article How to Generate an NMI for more information, or contact Delphix Support.