Skip to main content
Delphix

TB076 I/O Operations on VDBs and Other Sources May Intermittently Become Non-Responsive

 

 

Alert Type

Availability

Impact

I/O operations on a Delphix Virtualization Engine may intermittently become non-responsive. After onset of the problem, all I/O will become non-responsive persisting from a few seconds to as much as dozens of seconds or several minutes.

Episodes of non-responsiveness may recur multiple times per hour. 

The Delphix Admin application, System Setup Application, and Delphix Engine APIs may temporarily become non-responsive.

The issue may additionally trigger secondary issues on Windows-based data sources that include:

  • Virtual Database (VDB) deletion jobs may become persistently non-responsive
  • VDB refresh jobs may become persistently non-responsive
  • I/O operations for all VDBs hosted on one or more target environments (hosts) may become permanently non-responsive
    • Affected VDBs may become inoperative or crash

Recovering from these secondary issues affecting Windows-based sources requires a Delphix Engine reboot, which will temporarily disrupt operation of all VDBs on an affected Delphix engine.  See TB075 for more information. 

Contributing Factors

The problem can only occur when using one of the following affected Delphix releases:

 
 
Major Release All Sub Releases
6.0 6.0.0.0

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0The issue can only occur when using one of the following affected Delphix releases:

Major Release All Sub Releases
6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1

The issue is only known to affect Delphix Virtualization Engines. Delphix Engines used exclusively for Masking workflows are not known to be affected.

All I/O operations on an affected Delphix engine can both trigger and be affected by the problem:

  • Normal I/O operations on Virtual Databases (VDBs)
  • Source operations such as linking, Snapshot creation, LogSync, and Validated Sync 
  • All types of data sources including:
    • Oracle 
    • MS SQL
    • Extensible Data Source Integrations (plugins)
    • Unstructured Files and App Data (vFIles)

The problem is more likely to occur and to recur more frequently on Delphix engines with a high I/O workload and/or many VDBs. 

The duration of episodes of non-responsiveness is likely to increase on Delphix Engines with larger amounts of memory.  Extended durations of non-responsiveness increase the likelihood of secondary issues on Windows-based data sources.

Symptoms

Oracle Enterprise Manager may issue alerts on affected VDBs as average latencies for metrics such as "Average Synchronous Single-Block Read Latency" can surpass thresholds.

Windows-based VDBs may enter a stopped state and may not be restartable. 

MSSQL VDBs may report I/O "stall errors", e.g.

I/O Stall Time: The file 'E:\Program Files\Delphix\DelphixConnector\
b05efdda-4c94-41cf-b45d-e753619fc2d6-vdb-22\DATA\db\Sample.LDF' on database 'Sample' has an I/O stall time of 17311.2 ms/IO averaged over the last 15 minutes.

Relief/Workaround

Defer upgrades to affected versions of Delphix software.

The probability of encountering secondary issues on Windows-based data sources can be reduced by implementing the iSCSI configuration parameters documented in the iSCSI Configuration Required for Operational Stability section of Requirements for Windows iSCSI Configuration. These parameters must be implemented on all target environments, including both staging target hosts and VDB target hosts.