Two or more hosts with duplicate iSCSI qualified names (IQNs) attempting to mount LUNs from the same Delphix Engine may generate a flood of failed iSCSI connection requests, ultimately causing the iSCSI service on the Delphix Engine to become unresponsive. This may prevent the hosts with duplicate IQNs from mounting LUNs, and may also interrupt data access by unrelated hosts. When the iSCSI service becomes unresponsive in this manner, the only recourse may be to reboot the Delphix Engine. Until the duplicate IQNs are resolved, recurring failure is likely.
Affected Virtual Databases (VDBs) may hang and/or fail to restart.
The issue may occur in the following Delphix Releases:
- Delphix Engine 22.214.171.124, and 126.96.36.199
- Delphix Engine 188.8.131.52, 184.108.40.206, 220.127.116.11, and 18.104.22.168
- Delphix Engine 22.214.171.124 and 126.96.36.199
- Delphix Engine 188.8.131.52 and 184.108.40.206
- Delphix Engine 220.127.116.11 and 18.104.22.168
- Delphix Engine 22.214.171.124
- Delphix Engine 126.96.36.199 and 188.8.131.52
- Delphix Engine 184.108.40.206
- Delphix Engine 220.127.116.11
- Delphix Engine 18.104.22.168 and 22.214.171.124
- Delphix Engine 126.96.36.199, 188.8.131.52, and 184.108.40.206
- Delphix Engine 220.127.116.11
- Delphix Engine 18.104.22.168
- Delphix Engine 22.214.171.124
- Delphix Engine 126.96.36.199
- Delphix Engine 188.8.131.52 and 184.108.40.206
The issue can only occur when SQL DB VDBs are being used.
In order for the problem to occur, two or more iSCSI clients must exist with identical IQNs, and these clients must attempt to mount iSCSI LUNs from the same Delphix Engine. The clients may be Target Environments or Pre-Provisioning Target Environments. In order for the client to attempt to mount an iSCSI LUN, either a dSource or a VDB would need to be configured on the Environment, i.e. merely adding the Environment to the Delphix Engine is insufficient to trigger this issue.
When an Environment with a duplicate IQN is added to a Delphix Engine, the first symptom of this issue will be a failure to create a dSource or a failure to provision a VDB on that Environment. The error received by the user will reference an inability to mount an iSCSI LUN. Following this, the other Environment with the same IQN will begin experiencing errors, which may include dSources reporting errors on pre-provisioning runs and falling out of sync, or VDBs going offline. In the case of VDBs, end users may experience the database becoming unresponsive. These symptoms will be progressive and may eventually affect any and all Environments on the Delphix Engine.
Do not clone any any systems that are existing Target Environments or Pre-Provisioning Target Environments for Delphix. Do not use duplicate IQNs.
If a cloned Environment is created, the Environment with the duplicate IQN should be deleted from the Delphix Engine after deleting any VDB or dSource configured on that Environment. On the Windows Server that was deleted from Delphix, the system administrator should take the following steps:
- Using the iSCSI Initiator tool in the Control Panel, delete the Delphix Engine's IQN from the Favorite Targets tab. (If more than one IQN is present here, the Delphix Engine's IQN can be identified by having the format "iqn.2010-08.org.illumos" or "iqn.2007-08.com.delphix" followed by a 32-character hexadecimal string
- In the Targets tab, click on the Delphix Engine's IQN and click Disconnect to disconnect any existing sessions
- In the Configuration tab, change the Initiator Name to a unique value
- Restart the Microsoft iSCSI Initiator service from the Services tool in Server Manager
- Reboot the Delphix Engine
This issue is resolved in Delphix version 220.127.116.11 and later.
As defined in RFC 3270 (https://datatracker.ietf.org/doc/rfc3270/), IQNs must be globally unique. However, in practice, the IQN of an iSCSI initiator on a Microsoft Windows Server is constructed such that duplication is possible. An example IQN from a Windows Server might be:
where "myserver.mydomain.com" is the fully-qualified domain name of the Windows Server. In an environment where duplicate host names exist, such as may be the case in some lab environments, two Windows Servers may, by default, configure their iSCSI initiators with identical IQNs. Additionally, in Windows Server, the IQN can be manually changed by the system administrator, another possible vector for introducing a duplicate name.