Skip to main content
Delphix

KBA1047 How to Resolve Persistent "Error 21" Errors in SQL Server VDBs

 

Issue

Following a disruption of network connectivity between the Delphix Engine and a Target SQL Server environment, you may receive the following error:

Msg 823, Level 24, State 2, Line 2
The operating system returned error 21(The device is not ready.) to SQL Server during a read at offset 0x00000000012000 in file 'C:\Program Files\Delphix\DelphixConnector\ENGINE-UUID-vdb-1\DATA\db\databasename.mdf'. 

Additional messages in the SQL Server error log and system event log may provide more detail. 
This is a severe system-level error condition that threatens database integrity and must be corrected immediately. 
Complete a full database consistency check (DBCC CHECKDB). 
This error can be caused by many factors; for more information, see SQL Server Books Online.

This SQL Server error will prevent normal use of the affected database(s), and may prevent operations on VDBs and dSources (including Disable and Enable operations) from being performed by the Delphix Engine.

The Delphix Engine will typically detect this issue during VDB health monitoring, and affected databases will appear in a Stopped state in the Datasets page.

Troubleshooting

This error is raised by SQL Server after a disruption of iSCSI traffic between the Delphix Engine and a SQL Server Environment, if the disruption lasts longer than 60 seconds. Causes for this include, but are not limited to:

  • Planned maintenance of network, storage or compute infrastructure supporting the Delphix Engine
  • An unplanned outage of network, storage or compute infrastructure supporting the Delphix Engine
  • An unplanned restart of the Delphix Engine
  • A planned restart of the Delphix Engine where dSources and Virtual Databases (VDBs) were not disabled
  • Resource contention at the hypervisor (VMware) level
  • VMware snapshots, snapshot consolidation, or vMotion operations, resulting in delayed or stalled disk access

The "Error 21" errors will persist even after connectivity to the iSCSI disks has been re-established, preventing successful operation of the VDB.

To identify when the underlying issue may have started, look for the first instances of the strings "error 21" or "Msg 823, Level 24, State 2, Line 2" in your SQL Server ERRORLOG.

In some cases, only a subset of dSources or VDBs will be affected. Databases which are not accessed during a network disruption may remain online.

Resolution

Before proceeding with resolution, confirm that both the Delphix Engine and the SQL Server Environment can be reached and are responding normally.

The resolution steps will depend on the status of the database in the Management GUI:

Container Type Status in Management GUI Action to Resolve
dSource Active Disable and Enable dSource

If this fails due to KB4094893, restart the SQL Server instance.
VDB Stopped Start VDB

If this fails due to KB4094893, restart the SQL Server instance.
VDB Running Disable and Enable VDB

If this fails due to KB4094893, restart the SQL Server instance.

Workaround for KB4094893

Due to the issue described in Microsoft KB4094893, several versions of SQL Server are unable to clear the "Error 21" state automatically when a Start, Disable or Enable VDB operation is performed:

  • SQL Server 2014 RTM (unresolved)
  • SQL Server 2014 SP1 (unresolved)
  • SQL Server 2014 SP2 (fixed in CU12)
  • SQL Server 2016 RTM (unresolved)
  • SQL Server 2016 SP1 (fixed in CU9)
  • SQL Server 2016 SP2 (fixed in CU1)
  • SQL Server 2017 (unresolved, as of CU7)

To resolve this issue on a version affected by this bug, it may be necessary to restart the SQL Server instance on the affected host. Please be aware that restarting the instance will temporarily disrupt access to all databases on the SQL Server instance, whether or not they are hosted by the Delphix Engine.

Preventing recurrence of the issue

To minimize disruptions, we recommend performing a Disable operation on dSources and VDBs before performing any disruptive network maintenance.

We also recommend increasing disk timeouts as part of our Recommendations for Windows iSCSI Configuration. This has not been shown to affect the 60 second timeout described in this article, but it will decrease the likelihood of a network disruption affecting a SQL Server VDB for other reasons.

Finally, we encourage any customers affected by Microsoft KB4094893 to upgrade any Staging and Target SQL Server instances to a Cumulative Update or Service Pack where this issue is resolved.