Skip to main content
Delphix

Control Tower shows Delphix Engines are Offline (KBA7117)

 

KBA

KBA# 7117

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases
6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Troubleshooting <What>

text

A Delphix Engine (DE) was showing as Offline in Control Tower.

Prerequisites 

text

At some point prior the DE was upgraded from 5.3.6.0 to 6.0.5.0.

We do not know at this time if other versions are in play here.

Analyzing the support logs from the upgraded 6.0.5.0 DE we see - 

cd os/

view ps_aux_--sort=-%mem

We should normally see three java processes - 

 For example here we only see two our stack and sso but no cloud agent java process - 

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     14469  5.5  0.6 11608044 3173840 ?    Sl   Feb09 1727:22 /usr/bin/java -server -d64 -enableassertions -XX:MaxMetaspaceSize=512m -XX:MetaspaceSize=16m -XX:MaxMetaspaceFreeRatio=40 -XX:CompressedClassSpaceSize=256m -Xfuture -Xmx2g -Xms2g -Xss512k -XX:+UseCompressedOops -XX:InlineSmallCode=500 -XX:-OmitStackTraceInFastThrow -XX:+PreserveFramePointer -javaagent:/opt/delphix/server/lib/exec/tomcat-launcher/libs/com.google.code.java-allocation-instrumenter/java-allocation-instrumenter-3.0.jar=manualOnly -XX:ErrorFile=/var/delphix/server/log/hs_err_pid%p.log -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/crash -Dcom.sun.management.jmxremote -Ddelphix.group=0 -Ddelphix.server.root=/opt/delphix/server -Ddelphix.user=0 -Djava.security.properties==/opt/delphix/server/etc/java.security -Djna.dump_memory=true -Dlog.dir=/var/delphix/server/log -Dsun.net.inetaddr.negative.ttl=0 -Dsun.net.inetaddr.ttl=0 -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true -Djdk.tls.ephemeralDHKeySize=2048 -Djava.security.auth.login.config=/opt/delphix/server/etc/krbjaas.conf -Dsun.security.krb5.debug=true -Dsun.security.jgss.debug=true -Djava.security.debug=gssloginconfig,configfile,configparser,logincontext -Djavax.security.auth.useSubjectCredsOnly=false -jar /opt/delphix/server/lib/exec/tomcat-launcher/tomcat-launcher.jar /var/tmp /opt/delphix/server/lib/module/ROOT.war /opt/delphix/server/lib/module/api-json.war /opt/delphix/server/lib/module/dxcore.war /opt/delphix/server/lib/module/jetstream.war /opt/delphix/server/lib/module/api.war /opt/delphix/server/lib/module/connector.war /opt/delphix/server/lib/module/dxtest.war /opt/delphix/server/lib/module/styleguide.war /opt/delphix/server/lib/module/resources.war /opt/delphix/server/lib/module/login.war
root     27877  0.0  0.2 38131600 1044796 ?    Sl   Jan23  48:02 /usr/lib/jvm/adoptopenjdk-java8-jdk-amd64/bin/java -Djdk.tls.ephemeralDHKeySize=2048 -XX:-OmitStackTraceInFastThrow -XX:+PreserveFramePointer -enableassertions -XX:+CrashOnOutOfMemoryError -XX:ErrorFile=/var/delphix/server/log/sso-app/hs_err_pid%p.log -XX:MaxMetaspaceSize=256m -XX:CompressedClassSpaceSize=256m -Xfuture -Dlog.dir=/var/delphix/server/log/sso-app -Dserver.tomcat.accesslog.rename-on-rotate=true -Dserver.address=127.0.0.1 -Dserver.tomcat.accesslog.prefix=access -Dserver.tomcat.accesslog.directory=/var/delphix/server/log/sso-app -Dserver.tomcat.accesslog.enabled=true -Dserver.useForwardHeaders=true -Dlog.level=DEBUG -jar /opt/delphix/server/lib/sso-app/sso-app.jar --spring.config.location=/opt/delphix/server/etc/sso.properties

The logs for the cloud agent are in /var/delphix/server/log/agent

Analyzing these logs there were no updates after the upgrade occur.

From mds we can verify when the upgrade occurred - 

hercules=> select * from dlpx_version order by version desc;
 version_id | version | min_version |           path           |      status       |     build_date      |      install_date       |       verify_date       | was_deferred | min_reboot_optional_version | verification_version 
------------+---------+-------------+--------------------------+-------------------+---------------------+-------------------------+-------------------------+--------------+-----------------------------+----------------------
         14 | 6.0.5.0 | 5.3.6.0     | /var/dlpx-update/6.0.5.0 | CURRENTLY_RUNNING | 2020-11-04 10:29:18 | 2021-01-23 15:41:22.608 | 2021-01-23 15:35:14.022 | f            | 0.0.0.0                     | 0.0.0
         13 | 5.3.7.1 | 5.1.0.0     |                          | PREVIOUS          | 2019-12-31 14:59:03 | 2020-01-25 15:36:27.618 | 2020-01-25 15:22:20.462 | f            | 0.0.0.0                     | 0.0.0
<cut>

Now we move to analyzing the journal on this 6.0.5.0 DE - 

Jan 23 15:37:21 delphix-vm-n-6 systemd[1]: Starting Initial cloud-init job (pre-networking)...Jan 23 15:37:25 delphix-vm-n-6 cloud-init[1918]: Cloud-init v. 19.2-delphix-2020.10.26.16 running 'init-local' at Sat, 23 Jan 2021 15:37:23 +0000. Up 64.20 seconds.
Jan 23 15:37:25 delphix-vm-n-6 systemd[1]: Started Initial cloud-init job (pre-networking).
Jan 23 15:37:26 delphix-vm-n-6 systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: Cloud-init v. 19.2-delphix-2020.10.26.16 running 'init' at Sat, 23 Jan 2021 15:37:27 +0000. Up 67.87 seconds.
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info: | ens160 | False |     .     |     .     |   .   | 00:50:56:92:25:08 |
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |  host |         .         |
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
Jan 23 15:37:27 delphix-vm-n-6 cloud-init[2886]: ci-info:
Jan 23 15:37:27 delphix-vm-n-6 systemd[1]: Started Initial cloud-init job (metadata service crawler).
Jan 23 15:37:27 delphix-vm-n-6 systemd[1]: Reached target Cloud-config availability.
Jan 23 15:39:03 delphix-vm-n-6 systemd[1]: Starting Apply the settings specified in cloud-config...
Jan 23 15:39:03 delphix-vm-n-6 cloud-init[9695]: Generating locales (this might take a while)...
Jan 23 15:39:04 delphix-vm-n-6 cloud-init[9695]:   en_US.UTF-8... done
Jan 23 15:39:04 delphix-vm-n-6 cloud-init[9695]: Generation complete.
Jan 23 15:39:04 delphix-vm-n-6 cloud-init[9695]: Cloud-init v. 19.2-delphix-2020.10.26.16 running 'modules:config' at Sat, 23 Jan 2021 15:39:03 +0000. Up 164.18 seconds.
Jan 23 15:39:04 delphix-vm-n-6 systemd[1]: Started Apply the settings specified in cloud-config.
Jan 23 15:39:14 delphix-vm-n-6 apply[9586]: skipping: [localhost] => (item={u'key': u'Provisioning.UseCloudInit', u'value': u'n'})  => {
Jan 23 15:39:14 delphix-vm-n-6 apply[9586]:         "key": "Provisioning.UseCloudInit",
Jan 23 15:39:29 delphix-vm-n-6 systemd[1]: Starting Execute cloud user/final scripts...
Jan 23 15:39:30 delphix-vm-n-6 cloud-init[13976]: Cloud-init v. 19.2-delphix-2020.10.26.16 running 'modules:final' at Sat, 23 Jan 2021 15:39:29 +0000. Up 190.59 seconds.
Jan 23 15:39:30 delphix-vm-n-6 cloud-init[13976]: ci-info: no authorized ssh keys fingerprints found for user delphix.
Jan 23 15:39:30 delphix-vm-n-6 cloud-init[13976]: Cloud-init v. 19.2-delphix-2020.10.26.16 finished at Sat, 23 Jan 2021 15:39:30 +0000. Datasource DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud][dsmode=net].  Up 190.74 seconds
Jan 23 15:39:30 delphix-vm-n-6 systemd[1]: Started Execute cloud user/final scripts.
Jan 23 15:39:30 delphix-vm-n-6 systemd[1]: Reached target Cloud-init target.

After Cloud-init target we see - 

Jan 23 15:39:30 delphix-vm-n-6 systemd[1]: Started Execute cloud user/final scripts.
Jan 23 15:39:30 delphix-vm-n-6 systemd[1]: Reached target Cloud-init target.
Jan 23 15:39:30 delphix-vm-n-6 sudo[14898]:     root : TTY=unknown ; PWD=/usr/lib/postgresql/10/bin ; USER=postgres ; COMMAND=//opt/delphix/server/bin/dx_pg_post_start -r /usr/lib/postgresql/10 -d /mds/db -e /mds/mds_external -p 5432
Jan 23 15:39:30 delphix-vm-n-6 svc-postgres[13984]: Initialize postgres data dir /mds/db external /mds/mds_external
Jan 23 15:39:30 delphix-vm-n-6 svc-postgres[13984]: File /mds/db/force-reindex is present, re-indexing requested.
Jan 23 15:39:30 delphix-vm-n-6 svc-postgres[13984]: Starting re-indexing ...
Jan 23 15:39:30 delphix-vm-n-6 svc-postgres[13984]: DROP INDEX
Jan 23 15:39:30 delphix-vm-n-6 svc-postgres[13984]: DROP INDEX
<cut>
Jan 23 15:39:31 delphix-vm-n-6 svc-postgres[13984]: DROP INDEX
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Incorrect number of arguments: 2
Jan 23 15:39:33 delphix-vm-n-6 delphix-stat-service[13967]: Usage: [-v] zfs_ids_to_path <pool> <objset id> <object id>
Jan 23 15:39:47 delphix-vm-n-6 svc-postgres[13984]: REINDEX
Jan 23 15:39:47 delphix-vm-n-6 svc-postgres[13984]: Reindexing database took 17 seconds.
Jan 23 15:39:47 delphix-vm-n-6 systemd[1]: Started Delphix default PostgreSQL database server.
Jan 23 15:39:47 delphix-vm-n-6 systemd[1]: cgroup compatibility translation between legacy and unified hierarchy settings activated. See cgroup-compat debug messages for details.
Jan 23 15:39:47 delphix-vm-n-6 systemd[1]: Starting Delphix management service...
Jan 23 15:39:47 delphix-vm-n-6 sudo[19190]:     root : TTY=unknown ; PWD=/export/home/delphix ; USER=root ; COMMAND=/bin/chown 0:0 /var/delphix/server
Jan 23 15:39:47 delphix-vm-n-6 sudo[19195]:     root : TTY=unknown ; PWD=/export/home/delphix ; USER=root ; COMMAND=/bin/chmod g+rwx /var/delphix/
Jan 23 15:39:47 delphix-vm-n-6 sudo[19202]:     root : TTY=unknown ; PWD=/export/home/delphix ; USER=root ; COMMAND=/bin/rm -rf /var/tmp/work/Catalina/localhost
Jan 23 15:39:47 delphix-vm-n-6 sudo[19204]:     root : TTY=unknown ; PWD=/export/home/delphix ; USER=root ; COMMAND=/bin/rm -rf /var/tmp/webapps
Jan 23 15:39:47 delphix-vm-n-6 start_mgmt_server_jvm[19187]: Upgrading application to 6.0.5.0
<cut>
Jan 23 15:41:09 delphix-vm-n-6 systemd[1]: Reloading.
Jan 23 15:41:09 delphix-vm-n-6 systemd[1]: Starting Delphix SSO Service...
Jan 23 15:41:09 delphix-vm-n-6 sso-app[27875]: SSO app started
<cut>

Therefore something occurred the cloud-agent should have been downloaded from the Control Tower. However nothing occurred after this.

 

 

 

Resolution

To resolve xyz...

Complete the following procedure to xyz.

  1. Created a bug with engineering - DLPX-74652
  2. Instructed the customer to re-enable the cloud agent on the DEs shows as Offline.