Db2 Host Environment Refresh Fails for Db2 Instance When Trying to Start DSP Session (KBA10317)

Issue

During the post refresh steps of a Delphix Continuous Data engine, both an environment refresh and unquiesce (Enable) fail with the following:

The dSource enable job failed with message:

Lost SSH connection to host "db2_target_host.acme.com".

The Environment refresh job failed with this message:

Failed to start the DSP client on remote host "db2_target_host.acme.com".

Without resolving these issues, subsequent work on the environment, including adding Db2 instance users and refreshing the environment and attempting to link dSources, will result in the "Lost SSH connection" errors.

Prerequisites

Target host is used for Db2 DBMS with Db2 plugin applied (such as 4.X)
The Db2 instance user(s) and the Db2 target host OS user are not sharing a common group with access to various Db2 resources

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Date	Release
Aug 24, 2023	14.0.0.0
Jul 24, 2023	13.0.0.0
Jun 21, 2023	12.0.0.0
May 25, 2023	11.0.0.0
Apr 13, 2023	10.0.0.0 \| 10.0.0.1
Mar 13, 2023 \| Mar 20, 2023	9.0.0.0 \| 9.0.0.1
Feb 13, 2023	8.0.0.0
Jan 12, 2023	7.0.0.0
Releases Prior to 2023
Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Resolution

The issue is due to the fact the Db2 instance user was not setup with Db2 or similar common group permissions/membership.
There are times when following documentation (see below) will still result in the issue.
Continue to read to understand how to resolve.

Configuring Toolkit Directory

From this documentation link one can find the information on how to configure the toolkit directory depending on certain criteria:

Additional environment requirements

If the delphix_os user and instance users (responsible for the Delphix Engine operations such as linking and provisioning) are not sharing any common group, then the toolkit path directory must have -rwxrwxrwx (0777) permissions.

In addition, as is documented, when not using Db2 common group, where instance users are sharing access to Db2 utilities and resources, our documentation states from this link (using Db2 4.X plugin example):
Custom Db2 plugin parameters for 4.x plugins
Per the documentation (Appended to Delphix_COMMON is a UUID that uniquely identifies the directory)

Common group (usersHaveCommonGroup)

This parameter sets the permission for the "mnts" directory located under <Toolkit dir>/DB2 directory and for the "logs" directory located under the <Toolkit directory>/<Delphix_COMMON>/plugin/DB2_18f4ff11-b758-4bf2-9a37-719a22f5a4b8/ directory.

Set the usersHaveCommonGroup parameter as True, if the primary group of the primary environment user (user which is used to do discovery) is shared with the instance user.

Set the usersHaveCommonGroup parameter as False, if the primary environment user and the instance user do not share any user group.

Once the necessary changes are made, refresh the environment in order to apply the changes.

By default, the usersHaveCommonGroup parameter is set to True. The valid values for this parameter are true or false.

Mitigation When Error occurs about Lost ssh Connection After Environment Refresh

There are times when these steps, if followed, might still result in the failure, particular after making the adjustments and refreshing the environment.

The following solution has potential to mitigate the issue:

After configuring the db2_plug.conf file and setting toolkit directory with 777 permissions, run the command from within the toolkit directory:

chmod -R 777

The Db2 instance user(s) will also have their own toolkit directories with names similar to the following, under the toolkit directory:

Delphix_c4d493f454b1_f3cdfe94d06e_13_host

The first portion of the directory name of "random" characters is obtained from the Delphix Engine UUID.
From Delphix engine GUI, on the Menu go to Help | About:
The last portion of the Server ID/Serial number is used in the directory name.
At this point, as the Db2 instance user owns this directory, the blanket permission application with above chmod command may not apply to this directory.
Login as the Db2 instance user to this host and run the chmod command to apply the 777 permissions to this directory.
Repeat on other Db2 instance user directories if applicable.
After the configuration steps are completed refresh the environment and proceed with dSource links.

Troubleshooting

It may not be readily apparent why this error occurs. Troubleshooting might initially indicate that the permissions appear to be proper.

Check permissions below the actual toolkit directory to make sure the Db2 instance users have access to the Delphix_COMMON directory as well as the subdirectories below.
- There are repositories for the various scripts necessary to run operations
- However, if unable to execute the java class, client.jar, the issue occurs.
Make the changes offered in the resolution if at first things are not working.
If the situation persists report the issue to Delphix Technical Support.

KBA