Skip to main content
Delphix

KBA1783 Delphix Java Process May Hang on AIX

 

 

Applicable Delphix Versions

All

Issue

On AIX, customers may find that there are orphaned Delphix java processes that do not appear to be functioning, but continue to consume memory resources. These processes may be hung (deadlocked) and never exit which means that a server could be deprived of memory resources to the point that it results in a loss of service due to a complete lack of memory. It may also dramatically impact the performance of the system prior to this as standard paging and swapping algorithms are used to free physical memory.
 

Troubleshooting

Utilize the "ps" command to see if there are processes owned by the Delphix operating system user that are left hanging for extended periods of time (days). In the following example, the Delphix operating system user is named "delphix":

$ ps -ef | grep delphix
delphix  4980954 50331840   0                  0:00 <defunct>
delphix 12910728        1   0 07:58:19      -  0:02 /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/java/jdk/bin/java -ea -XX:-UseVMInterruptibleIO -Ddelphix.host.os=unix -Ddelphix.toolkit.base.dir=/var/opt/delphix/Toolkit -Ddelphix.max.worker=16 -Djava.io.tmpdir=/var/opt/delphix/Toolkit/Delphix_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/tmp -jar /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/client/dsp/client.jar
delphix 13303810        1   0   Aug 19      -  0:02 /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/java/jdk/bin/java -ea -XX:-UseVMInterruptibleIO -Ddelphix.host.os=unix -Ddelphix.toolkit.base.dir=/var/opt/delphix/Toolkit -Ddelphix.max.worker=16 -Djava.io.tmpdir=/var/opt/delphix/Toolkit/Delphix_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/tmp -jar /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/client/dsp/client.jar
delphix 16253078 35192968   0                  0:00 <defunct>
delphix 16973938 59048178   0                  0:00 <defunct>
delphix 20971624        1   0   Aug 19      -  0:02 /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/java/jdk/bin/java -ea -XX:-UseVMInterruptibleIO -Ddelphix.host.os=unix -Ddelphix.toolkit.base.dir=/var/opt/delphix/Toolkit -Ddelphix.max.worker=16 -Djava.io.tmpdir=/var/opt/delphix/Toolkit/Delphix_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/tmp -jar /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/client/dsp/client.jar
delphix 22282392 41680912   0                  0:00 <defunct>
delphix 22937658        1   0   Aug 19      -  0:02 /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/java/jdk/bin/java -ea -XX:-UseVMInterruptibleIO -Ddelphix.host.os=unix -Ddelphix.toolkit.base.dir=/var/opt/delphix/Toolkit -Ddelphix.max.worker=16 -Djava.io.tmpdir=/var/opt/delphix/Toolkit/Delphix_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/tmp -jar /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/client/dsp/client.jar
delphix 35192968        1   0   Aug 17      -  0:02 /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/java/jdk/bin/java -ea -XX:-UseVMInterruptibleIO -Ddelphix.host.os=unix -Ddelphix.toolkit.base.dir=/var/opt/delphix/Toolkit -Ddelphix.max.worker=16 -Djava.io.tmpdir=/var/opt/delphix/Toolkit/Delphix_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/tmp -jar /var/opt/delphix/Toolkit/Delphix_COMMON_423e004e_8079_502d_14bb_82d1bfdd3532_delphix_host/client/dsp/client.jar

It is also likely that each of these processes will have a child process with an executable name of “<defunct>” (zombie). If these conditions are observed, it is likely that the Java process is hung in memory allocation and cannot make forward progress. This is not a Delphix or Java issue as the hang occurs in the AIX memory management facility.

Resolution

In order to mitigate the hang, simple environment variable additions for the Delphix operating system user can be made that will only affect the Delphix OS user and have no impact to the rest of the system. Adding the following environment variable and options will prevent the accumulation of Java processes and the depletion of memory:

$ cat .ssh/environment
MALLOCOPTIONS=multiheap:32,pool

NOTE: The environment variable must be defined for non-interactive logins. Many of the commands Delphix issues are done through a non-interactive login and as such, they need to be defined in the ".ssh/environment" file. Depending on the operating system user's shell, there may be other alternatives. To test that the environment variable is defined correctly for non-interactive logins the following test can be completed:

$ ssh ora11202@aix101-14.delphix.com "env | grep MALLOCOPTIONS"
ora11202@aix101-14.delphix.com's password: 
MALLOCOPTIONS=multiheap:32,pool

From IBM’s documentation these two options have the following impact:

multiheap

  • Configures the number of parallel heaps to be used by memory allocators. You can set the multiheap by exporting MALLOCOPTIONS=multipheap:n. The value n can vary from 1 through 32. The default value is 32, if n is not specified. This option is advisable for multithreaded applications, as it can significantly improve the performance.

pool

  • Maintains the bucket for each thread and provides a lock-free allocation and deallocation for blocks less than 513 bytes. This option improves the performance of multithreaded applications as it avoids the time that is spent on locking of memory size less than 513 bytes. The pool option makes small memory block allocations fast and efficient.

Additional Information

In a future release of Delphix, the environment variable will be set in the user's environment automatically as needed.