TB007 High Pool Utilization Causes Performance Degradation

Impact

Significant performance degradation will occur when the Delphix storage pool utilization exceeds 85%. 

Contributing Factors

The issue can affect any Delphix Engine release, but is more likely to occur in the following Delphix Engine releases:

• Delphix Engine 3.2.x - 5.2.x

In Delphix releases, other than those enumerated above, the Delphix Engine will automatically discontinue the initiation of new SnapSync and Provision jobs when pool utilization exceeds a critical threshold. This critical threshold was configurable in Delphix Engine 2.6 releases but defaults to 80%. In the latest releases of Delphix Engine, the critical threshold is fixed at 85%

Pathological performance is more likely to occur if utilization approaches 90%. 

The problem is more likely to occur if additional disks/LUNs are added to the Delphix pool after utilization has already become excessive, resulting in a scenario with unbalanced disk/LUN utilization. 

For more information about how Delphix manages operations at various capacity thresholds please refer to the product documentation link here:

An Overview of Capacity and Performance Information

Symptoms

Virtual database (VDB) performance, especially for write operations, may be diminished. In extreme circumstances pathological VDB performance may occur, and target servers may report errors such as "NFS server not responding".  

In rare circumstances, the Delphix Engine user interface (GUI and CLI) may perform poorly or become non-responsive. 

Relief/Workaround

The issue may be avoided by maintaining a storage pool with sufficient available space.   For best performance, Delphix recommends a pool utilization of 80% or less. This can be assisted by:

• planning a storage configuration with sufficient reserve capacity
• managing appropriate retention policies for snapshots, dSources, and VDBs
• deleting unnecessary snapshots, dSources, and VDBs
• expanding the size of existing LUNs
• adding new LUNs to the storage configuration well before aggregate utilization approaches 80%
   

It is preferable to expand the size of existing LUNs rather than adding new LUNs, as the former prevents capacity from any individual LUNs from becoming overutilized.  In cases where expanding current LUNs is not possible, new LUNs should be added as soon as possible.  Once aggregate pool utilization approaches 80%, adding additional LUNs may no longer be an effective means of preventing the onset of poor performance due to this issue.    Although a larger portion of allocations will be directed to LUNs that have the most free space, even a small portions of allocations may lead to problems for LUNs that are already near the utilization limit.