Increasing Java Heap Memory to Avoid Out Of Memory Errors (KBA10608)

Issue

Under some circumstances, the Java management application on a Delphix Continuous Data engine may exhaust its heap memory, causing the application to restart.  For instance, this may happen when running multiple large replication jobs at the same time on an engine.

If the management application restarts, any running jobs will be impacted but running VDBs will not be affected.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Date	Release
Nov 21, 2023	17.0.0.0
Oct 18, 2023	16.0.0.0
Sep 21, 2023	15.0.0.0
Aug 24, 2023	14.0.0.0
Jul 24, 2023	13.0.0.0
Jun 21, 2023	12.0.0.0
May 25, 2023	11.0.0.0
Apr 13, 2023	10.0.0.0 | 10.0.0.1

Resolution

By default, the Java application runs with a maximum heap size of 2GB which is sufficient for most environments.  If out of memory errors are encountered during the operation of a Delphix engine running version 10.0.0.0 or later, the maximum heap size can be increased using the maxHeapSizeGb setting in the CLI or API.

1. Log into the Delphix CLI as a sysadmin user.
2. Navigate to the system section.  Listing the properties using ls will show the current maxHeapSizeGb value.

delphix-engine> system
delphix-engine system> ls
Properties
...
    maxHeapSizeGb: 2GB

3. Update the maxHeapSizeGb setting to increase the amount of memory allocated to the Java management application's heap.  Values up to 10GB can be used.

delphix-engine system> update
delphix-engine system update *> set maxHeapSizeGb=6
delphix-engine system update *> ls
Properties
    type: SystemInfo
    engineType: VIRTUALIZATION
    hostname: delphix-engine
    maxHeapSizeGb: 6GB (*)
    maxNativeMemoryGb: 4GB
delphix-engine system update *> commit

4. Restart the Delphix engine's management application for the new setting to take effect.

delphix-engine system> restart
delphix-engine system restart> commit

If this issue is encountered on a Delphix engine running a version prior to 10.0.0.0, please contact Delphix Support for assistance.

Troubleshooting

If this occurs and SMTP is configured, an alert e-mail with a message similar to the following will be sent.

Description   The Delphix Engine has experienced memory exhaustion which caused the management service to be restarted. Jobs active at that time may have been canceled.
Action        Contact Delphix Support for assistance.
Severity      WARNING

The issue may occur when replicating large numbers of objects with a single job or across multiple jobs as the required dependency generation work can consume large amounts of heap memory.

If the issue is seen on a version prior to version 10.0.0.0 then an escalation is required and a hotfix generated.