Skip to main content
Delphix

TB001 Duplicate iSCSI Qualified Names may Lead to Failure of Delphix iSCSI Service

 

Alert Type

Data Availability

Impact

Two or more hosts with duplicate iSCSI qualified names (IQNs) attempting to mount LUNs from the same Delphix Engine may generate a flood of failed iSCSI connection requests, ultimately causing the iSCSI service on the Delphix Engine to become unresponsive. This may prevent the hosts with duplicate IQNs from mounting LUNs, and may also interrupt data access by unrelated hosts. When the iSCSI service becomes unresponsive in this manner, the only recourse may be to reboot the Delphix Engine. Until the duplicate IQNs are resolved, recurring failure is likely.

Affected Virtual Databases (VDBs) may hang and/or fail to restart.

Contributing Factors

The issue may occur in the following Delphix Releases:

  • Delphix Engine 3.0.0.3, and 3.0.0.4
  • Delphix Engine 3.0.1.0, 3.0.1.1, 3.0.1.2, and 3.0.1.3
  • Delphix Engine 3.0.2.0 and 3.0.2.1
  • Delphix Engine 3.0.3.0 and 3.0.3.1
  • Delphix Engine 3.0.4.0 and 3.0.4.1
  • Delphix Engine 3.0.5.0
  • Delphix Engine 3.0.6.0 and 3.0.6.1
  • Delphix Engine 3.1.0.1
  • Delphix Engine 3.1.1.0
  • Delphix Engine 3.1.2.0 and 3.1.2.1
  • Delphix Engine 3.1.3.0, 3.1.3.1, and 3.1.3.2
  • Delphix Engine 3.1.4.0
  • Delphix Engine 3.1.5.0
  • Delphix Engine 3.2.0.0
  • Delphix Engine 3.2.1.0
  • Delphix Engine 3.2.2.0 and 3.2.2.1

The issue can only occur when SQL DB VDBs are being used. 

In order for the problem to occur, two or more iSCSI clients must exist with identical IQNs, and these clients must attempt to mount iSCSI LUNs from the same Delphix Engine. The clients may be Target Environments or Pre-Provisioning Target Environments. In order for the client to attempt to mount an iSCSI LUN, either a dSource or a VDB would need to be configured on the Environment, i.e. merely adding the Environment to the Delphix Engine is insufficient to trigger this issue.

Symptoms

When an Environment with a duplicate IQN is added to a Delphix Engine, the first symptom of this issue will be a failure to create a dSource or a failure to provision a VDB on that Environment. The error received by the user will reference an inability to mount an iSCSI LUN. Following this, the other Environment with the same IQN will begin experiencing errors, which may include dSources reporting errors on pre-provisioning runs and falling out of sync, or VDBs going offline. In the case of VDBs, end users may experience the database becoming unresponsive. These symptoms will be progressive and may eventually affect any and all Environments on the Delphix Engine.

Relief/Workaround

Do not clone any any systems that are existing Target Environments or Pre-Provisioning Target Environments for Delphix. Do not use duplicate IQNs.

If a cloned Environment is created, the Environment with the duplicate IQN should be deleted from the Delphix Engine after deleting any VDB or dSource configured on that Environment. On the Windows Server that was deleted from Delphix, the system administrator should take the following steps:

  1. Using the iSCSI Initiator tool in the Control Panel, delete the Delphix Engine's IQN from the Favorite Targets tab. (If more than one IQN is present here, the Delphix Engine's IQN can be identified by having the format "iqn.2010-08.org.illumos" or "iqn.2007-08.com.delphix" followed by a 32-character hexadecimal string
  2. In the Targets tab, click on the Delphix Engine's IQN and click Disconnect to disconnect any existing sessions
  3. In the Configuration tab, change the Initiator Name to a unique value
  4. Restart the Microsoft iSCSI Initiator service from the Services tool in Server Manager
  5. Reboot the Delphix Engine

Resolution

This issue is resolved in Delphix version 3.2.3.0 and later.

Additional Information

As defined in RFC 3270 (https://datatracker.ietf.org/doc/rfc3270/), IQNs must be globally unique. However, in practice, the IQN of an iSCSI initiator on a Microsoft Windows Server is constructed such that duplication is possible. An example IQN from a Windows Server might be:

iqn.1991-05.com.microsoft:myserver.mydomain.com

where "myserver.mydomain.com" is the fully-qualified domain name of the Windows Server. In an environment where duplicate host names exist, such as may be the case in some lab environments, two Windows Servers may, by default, configure their iSCSI initiators with identical IQNs. Additionally, in Windows Server, the IQN can be manually changed by the system administrator, another possible vector for introducing a duplicate name. 

DLPX-24628-  COMSTAR should enforce IQN uniqueness during session setup based on remote IP address