Skip to main content
Delphix

TB041 VDB Operations on Windows Environments May Fail With iSCSI Mount Errors

  1. Last updated
  2. Save as PDF
 

 

  1. Alert Type
  2. Impact
  3. Contributing Factors
  4. Symptoms
  5. Relief/Workaround
  6. Resolution
  7. Additional Info
    1. Related Links

Alert Type

Data Availability

Impact

Delphix presents iSCSI targets that are used by Microsoft Windows Server environments (hosts) used for Virtual Database (VDB) target and Staging Hosts.  In some circumstances, mount operations from affected environments may not succeed causing associated Delphix workflows to fail including:

  • Provisioning new VDBs
  • Refreshing VDBs
  • Enabling VDBs

This may result in affected VDBs being completely unavailable or in the inability to refresh VDBs with new data or snapshots. 

Contributing Factors

The issue may occur in the following Delphix releases:

Major Release

All Sub Releases

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

4.0

4.0.0.0, 4.0.0.1, 4.0.1.0, 4.0.2.0, 4.0.3.0, 4.0.4.0, 4.0.5.0, 4.0.6.0, 4.0.6.1

3.2

3.2.0.0, 3.2.1.0, 3.2.2.0, 3.2.2.1, 3.2.3.0, 3.2.4.0, 3.2.4.1, 3.2.4.2, 3.2.5.0, 3.2.5.1, 3.2.6.0, 3.2.7.0, 3.2.7.1

3.1

3.1.0.1, 3.1.1.0, 3.1.2.0,  3.1.2.1, 3.1.3.0 , 3.1.3.1, 3.1.3.2, 3.1.4.0, 3.1.5.0, 3.1.6.0

3.0

3.0.0.3, 3.0.0.4, 3.0.1.0, 3.0.1.1, 3.0.1.2, 3.0.1.3, 3.0.2.0, 3.0.2.1, 3.0.3.0, 3.0.3.1, 3.0.4.0, 3.0.4.1, 3.0.5.0, 3.0.6.0, 3.0.6.1

The issue can only occur on environments (hosts) running Windows Server 2012 R2 and functioning as one or more of:

  • a target environment hosting one or more Microsoft SQL Server VDBs
  • an environment acting as a Delphix Staging Host with Microsoft SQL Server Instances
  • a target environment hosting one or more Windows vFiles VDBs

This issue may occur after one or more of the following Microsoft Windows Updates is installed on a susceptible target environment:

  • KB4025335 - Preview of Monthly Rollup (July 18, 2017)
  • KB4034672 - Security-only update (August 8, 2017)
  • KB4034681 - Monthly Rollup (August 8, 2017)

The following updates are still being monitored to confirm whether they may also be contributing factors:

  • KB4034663 - Preview of Monthly Rollup (August 15, 2017)
  • KB4038792 - Monthly Rollup (September 12, 2017)

This issue may first occur immediately after installing the patches, or up to a month afterwards. Once the problem occurs it may then occur persistently, including after reboots of an affected system.

Symptoms

Affected VDB operations will report the following error, visible within the Delphix Management interface:

Error Code
exception.windowshost.mount.failed

Suggested Action
Make sure that the iSCSI initiator is running and that the user "DOMAIN\delphix_user" has privileges to perform disk management on the host.

Command Output
Includes ERROR_LUN_NOT_FOUND:

die : Error: ERROR_LUN_NOT_FOUND: Could not find a device for 0
At C:\Program Files\Delphix\DelphixConnector\[ENGINE-UUID]
-vdb-1\SCRIPT\MountLunData.ps1:229 char:9
+ die "ERROR_LUN_NOT_FOUND: Could not find a device for $lunid"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorExcep tion
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,die

Customers may also notice unexpected results when viewing iSCSI configuration through Windows:

  • The iSCSI command iscsicli sessionlistreports no disks for the target
  • The iSCSI Initiator applet does not show any LUNs if the Delphix Engine is selected and the "Devices" button is pressed

VDBs that were already provisioned and enabled before Windows Updates were installed may continue to function as expected.

Relief/Workaround

Although there is not a definitive correlation between Microsoft patches and the issue, the issue has not been observed on any environment that has not recently had Microsoft patches applied. Defer or suspend Microsoft patch updates on susceptible Windows target environments until a full resolution is available from Delphix.

If the issue has already manifested, it may be possible to recover by one or more of the following:

  • Uninstall Microsoft patches KB4025335, KB4034672, and/or KB4034681 using the normal Windows uninstallation facilities.
  • Migrating VDBs or staging instances to another environment (host)
    • Disable affected VDBs using the Delphix Admin App (GUI) or CLI
    • Use the "Migrate VDB" feature available on the dSource card (5.0 and earlier) or Configuration tab (5.1 and later) to move the VDB to an unaffected server

      Note: Migrating VDBs to a new target environment may require application changes to reference the newly selected target instance. 

 

Resolution

This issue is fully resolved in Delphix Engine release 5.1.8.1 and later releases. 

Additional Info

After some difficulty, Delphix has reproduced this problem internally and is working closely with Microsoft. Delphix will update this bulletin as further information is available. 

After the given patch updates for Microsoft Windows Server 2012 R2 have been applied, the Devices property the MSiSCSIInitiator_SessionClass Class may be empty. This will result in the failure of iSCSI disks to be properly identified and mounted.

Privacy Policy Terms of Use Legal About
© 2024 Delphix Corp