Skip to main content
Delphix

Memory - Errors and How much (KB1036)

 

Issue 

Masking or Profiling job fails with the error messages:

  • Masking Engine: "java.lang.OutOfMemoryError: GC overhead limit exceeded".
  • Oracle message:  "Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory and no more memory could be made available by the garbage collector.".

OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable. 

Troubleshooting Out of Memory Error

Some jobs need more memory than others. It depends on the number of tables, how many columns you are masking, and the type of algorithms assigned to the inventory. The current memory allocation may not be enough or the job is configured with too many threads/streams, or both.

Note

Note:

The default memory allocation is 1GB.

Example of Masking Engine -  DMsuite log with out of memory condition from Masking job:

image2016-5-12%2B12-21-48.png

How much memory is needed?

The memory size needed is dependent on a few things and can be estimated using the following formula. The max value is always used in order to simplify the calculation and to define the safe upper limit.

  • Max job memory needed:
    • Memory needed: per algorithm: max "lookup value length" x "number of lookup values".
    • Memory needed: per table: max "per algorithm" x "number of masked columns in table".
    • Memory needed: per job: max "per table" x "number of streams".
       
  • Engine memory needed: 
    • On the engine: 4 GB (OS etc) + max "per job" x "number of jobs concurrently".
       
  • Min memory:
    • Enough on the VM host is if the host is overcommitted or not.
    • If the memory has not been reserved or there is not enough memory on the VM host - set Min = Max / 4.
    • If the memory is reserved and there is enough on the VM host - set Min = Max.
  • Min = Max will improve performance (if memory is available). 
     
  • Warnings:
    • If you don't have enough Job Max Memory - the job will crash.
    • If you don't have enough memory on the engine - the engine will crash. 
       
  • NOTE: 
    • The performance will not improve with more memory. 
    • On standard jobs - there is no need to set the Min/Max memory.
    • If there are errors with memory, these usually result in crashes. 

Resolution

To resolve Out of Memory error, the memory allocated to the job must be increased and/or decrease the number of streams and threads which are consuming memory.  Rerun the job, once these changes are made to the memory.

Note

Note:

Do not exceed the amount of memory allocated to the VM machine.

To increase or decrease memory configuration parameters use MIN Memory and Max Memory fields on the Create Masking Job page.

masking_mem_config.png