TB075 Operations on Windows-Based Sources and Databases May Stop Responding

Impact

Operations on Windows-based Sources and Virtual Databases (VDBs) may stop responding:

• VDB deletions
• VDB refreshes

Once one VDB operation has hung, all subsequent such operations on any Windows-based source or VDB will also stop responding. In some circumstances, all VDBs on an affected target environment and/or all dSources on an affected target staging host environment will stop responding or crash.

Once the problem occurs the only remedy is to reboot an affected Delphix Engine which may result in hangs or crashes of all VDBs on the affected engine.

Contributing Factors

This bulletin applies to the following releases:

The issue can only occur when using Windows-based target environments. This includes:

• MS SQL Sources and VDBs
• WIndows-based Unstructured Files and App Data (vFiles)
• Windows-based Extensible Data Source Integrations (plugins)

The issue is only known to occur when affected environments are not configured as per the iSCSI Configuration Required for Operational Stability section of Requirements for Windows iSCSI Configuration. 

The issue only occurs secondary to recovery from errors during iSCSI-based operations. These types of errors can be triggered by:

Important:

This is the most significant known trigger of the problem

• Poor performance of back-end storage presented to the Delphix engine
• Poor responsiveness of Delphix Engine (e.g. due to oversubscription of CPU or memory on hypervisor)
• Network problems including packet loss and general loss of connectivity affecting connections between the Delphix Engine and target environments
• Poor responsiveness of target host and target staging host environments

The issue is far less likely to occur on the Delphix 6.0.2.1 release than other affected releases.

Symptoms

• Refreshing existing VDBs or provisioning new VDBs will not complete and the jobs cannot be canceled. 
• Faults with the below text can appear:

  The VDB is not able to access its storage. Reads and writes to the VDB might fail

• The Delphix Engine console may show errors similar to the below:
  
  iSCSI Login negotiation failed. Unable to locate Target Portal Group on iqn.2008-07.com.delphix:02:629e0257-4117-69c2-f381-b303e42e05bb

Relief/Workaround

One or more of the following:

• Defer upgrades to affected versions of Delphix software.
• Implement the Windows iSCSI configuration parameters documented in the iSCSI Configuration Required for Operational Stability section of Requirements for Windows iSCSI Configuration. These parameters must be implemented on all target environments, including both staging target hosts and VDB target hosts. 

Related Documents

Requirements for Windows iSCSI Configuration

KBA1251 Registry Settings for Optimal Database Performance and Stability

Extensible Data Source Integration

Provisioning Unstructured Files as vFiles