Skip to main content
Delphix

Resolving persistent "error 21" errors in SQL Server VDBs

 

Issue

Following a disruption of network connectivity between the Delphix Engine and a Target SQL Server environment, you may receive the following error:

Msg 823, Level 24, State 2, Line 2
The operating system returned error 21(The device is not ready.) to SQL Server during a read at offset 0x00000000012000 in file 'C:\Program Files\Delphix\DelphixConnector\ENGINE-UUID-vdb-1\DATA\db\databasename.mdf'. 

Additional messages in the SQL Server error log and system event log may provide more detail. 
This is a severe system-level error condition that threatens database integrity and must be corrected immediately. 
Complete a full database consistency check (DBCC CHECKDB). 
This error can be caused by many factors; for more information, see SQL Server Books Online.

If present, this error will be visible when viewing the details of failed Delphix operations, and in the SQL Server ERRORLOG. It will prevent:

  • Routine use of the affected database(s)
  • Successful Disable and Enable operations from the Delphix Engine

Troubleshooting

This error is typically raised after a disruption of iSCSI network traffic between the Delphix Engine and a SQL Server Environment, or if the the Delphix Engine becomes temporarily inaccessible. Causes for this include, but are not limited to:

  • Planned maintenance of network, storage or compute infrastructure supporting the Delphix Engine
  • An unplanned outage of network, storage or compute infrastructure supporting the Delphix Engine
  • An unplanned restart of the Delphix Engine
  • A planned restart of the Delphix Engine where virtual databases (VDBs) were not disabled
  • Resource contention at the hypervisor (VMware) level
  • VMware snapshots or snapshot consolidation, resulting in delayed or stalled disk access

The "Error 21" errors will often persist once connectivity to the iSCSI disks has been re-established, preventing successful operation of the VDB.

To identify when the underlying issue may have started, look for the first instances of "error 21" or "Msg 823, Level 24, State 2, Line 2" in your SQL Server ERRORLOG.

It is normal for only a subset of VDBs on a single engine to be affected. Databases which are not accessed during a network disruption will usually remain online.

Resolution

Before proceeding with resolution, confirm that both the Delphix Engine and the SQL Server Environment can be reached and are responding normally.

To resolve this issue, restart the affected SQL Server Instance.

This will temporarily disrupt access to all databases on the Instance, including databases not hosted by the Delphix Engine.

This will clear SQL Server's internal state, and ensure that the SQL Server instance re-tests whether each data file is accessible.

Preventing recurrence of the issue

This issue is most often triggered by network disruptions that are longer than Windows' default iSCSI timeout (60 seconds).

Delphix recommends significantly increasing this timeout as part of our Recommendations for Windows iSCSI Configuration. This dramatically decreases the likelihood of a network disruption affecting a SQL Server VDB.