The Delphix Engine may become unresponsive. I/O to Virtual Databases (VDB) may experience severe performance issues or hang. Oracle VDBs may crash.
The issue may occur in the following Delphix Releases:
- All Delphix Engine 4.0 releases
- All Delphix Engine 4.1 releases
- Delphix Engine 126.96.36.199 and 188.8.131.52
- Delphix Engine 184.108.40.206 and 220.127.116.11
The issue only occurs when deleting one or more large files on a Delphix filesystem from a Linux or UNIX target. This may happen under one or more of the following circumstances:
- Using the Linux or UNIX "rm" command to delete individual files or groups of files on an Oracle VDB
- Using the Linux or UNIX "rm" command to delete individual files or groups of files on an unstructured data (vFiles) filesystem on a Delphix Engine.
- Dropping large Oracle datafiles or temporary tablespaces on a VDB using Oracle's "ALTER TABLESPACE DROP DATAFILE", "DROP TABLESPACE INCLUDING CONTENTS AND DATAFILES", and similar SQL statements.
The problem only occurs when large files are being deleted, usually a minimum of dozens of gigabytes in size.
The problem does not occur when deleting entire VDBs, regardless of size, because in those circumstances entire filesystems are deleted.
The problem does not occur when deleting files from VDBs or unstructured data from vFiles on Microsoft Windows environments.
- Existing sessions to an affected Delphix Engine may become unresponsive, including browser sessions to the Delphix Admin application, browser sessions to the Server Setup application, ssh CLI sessions and WebAPI sessions
- Attempts to establish new sessions to the Delphix Engine may hang or fail
In rare cases, messages may appear on the Delphix Engine Console, including:
WARNING: vmxnet3s0: ddl_dma_alloc_handle() failed
WARNING: vmxnet3s :0 : ddi_dma_mem_alloc() failed
- VDBs may hang. New sessions to VDBs may hang or fail
Oracle VDBs may crash with one of the following errors:
ORA-00494: enqueue [CF] held for too long ...
ORA-00238: timeout waiting for control file enqueue held by ...
- An affected Delphix engine that is forcibly rebooted may take much longer than normal to boot, possibly several hours or more.
On Linux- or UNIX-based target environments, do not delete large files from NFS-mounted filesystems provisioned from a Delphix Engine:
- When attempting to clean up space from an unused VDB, instead of removing files manually, delete the entire VDB using the Delphix User Interface. See the instructions for Deleting a VDB in the end-user documentation.
- Avoid the use of Oracle's "ALTER TABLESPACE DROP DATAFILE", "DROP TABLESPACE ... INCLUDING CONTENTS AND DATAFILES" and similar statements that may remove large files on a database
- Avoid the use of the Linux or UNIX "rm", "unlink", or "mv" commands on Oracle files or unstructured data provisioned from vFiles.
An affected Delphix Engine that is completely unresponsive may be forcibly rebooted. See "How to Generate an NMI" on the Delphix knowledge base.
The issue is fully resolved in Delphix Engine OS Version 4.2.2015.05.19, included with Delphix Engine 18.104.22.168.
Fresh Installations and Full Upgrades to Delphix Engine 22.214.171.124 will run OS Version 4.2.2015.05.19.
Deferred OS upgrades to Delphix 126.96.36.199 run a prior version of the Delphix OS and are still susceptible to the problem described in this bulletin.
See the "Deferred OS Upgrade" section of "Upgrading to a New Version of the Delphix Engine" for information about how to determine the current OS version of a Delphix Engine.
The issue may result in protracted disruptions of service, although it occurs rarely.
In some circumstances users have deleted a large number of files which are queued up on a target environment. Even after an affected Delphix Engine is rebooted and recovers, the issue may immediately recur when file deletions continue to be processed from a target environment. This may result in even more significant disruption of service.