TB021 Delphix Engine Performance Issues on Large or Memory-constrained Virtual Guests

Alert Type

Availability/Performance

Impact

The Delphix Engine may experience recurring and short-lived periods of reduced performance. In most instances the issue may be difficult to observe. Individual network connections between an affected Delphix Engine and other hosts may appear to hang for brief periods lasting 2-60 seconds. Network connections between the Delphix Engine and Windows hosts may be reset, although in most instances these connections will automatically be retried or reestablished by Delphix without any required intervention. 

In extremely rare cases, the Delphix Engine may hang, requiring manual intervention to reboot the system. 

Contributing Factors

The issue may occur in the following Delphix Releases:

• All Delphix Engine 4.0 releases
• All Delphix Engine 4.1 releases

The incidence and severity of the problem is related to increasing amounts of memory configured for the virtual guest hosting a Delphix Engine. Large memory guests, greater than 100GB, are more likely to be impacted.

The problem correlates with increased VDB I/O activity, especially where there are increasing numbers of read operations.

Connections between Windows hosts and affected Delphix engines are more likely to be impacted, because Windows TCP connections will attempt fewer segment retransmissions before resetting/aborting a connection. 

Symptoms

• On MS SQL dSources, Validated Sync or transaction log processing may be interrupted due to network connectivity errors. Delphix faults and alerts may generated with text similar to the following:

Validated sync for dSource "<name>" failed with the error: An error occurred when attempting to connect to remote host "<hostname>" for environment "DSOURCE: <name>": "null".

• TCP connections from Windows clients, e.g. Putty, to an affected Delphix Engine may be terminated with a "connection reset" message.
• A single vCPU may stay at 100% capacity for periods of time ranging from 2 to 60 seconds. While this cannot be directly observed from the Delphix system, it may be visible using tools on the virtualization platform. For example, the esxtop utility on VMware's ESXi host console has an option on the CPU Panel that can display the utilization of individual vCPUs.

Resolution

This issue is fully resolved in Delphix Engine 4.2.0.0 and later releases.  

Additional Information

Using esxtop or resxtop in Interactive Mode (external link)