Skip to main content
Delphix

Memory - How much and Errors (KB1036)

 

How much memory is needed?

 

Note

Note:

The default Job memory allocation is 1 GB. This is usually ok - even when streams are used. 

The memory size needed is dependent on a few things and can be estimated using the following formula. The max value is always used in order to simplify the calculation and to define the safe upper limit.

  • Warnings:
    • If you don't have enough Job Max Memory - the job will crash.
    • If you don't have enough memory on the engine - the engine will crash. 

Engine Memory 

  • Engine memory needed: 
    • Simple formula: 4 GB (OS etc) + max( max_job_mem) x "number of jobs running concurrently".
    • More exact: 4 GB (OS etc) + sum( max_job_memfor all jobs running concurrently.

Job Memory - Notes

Here are some useful notes about Masking Job Memory. 

  • For standard jobs, default memory (MIN/MAX) is usually enough (1 GB).
  • Setting Min = Max will allocate all memory and improve performance slightly. 
     
  • Job memory MAX (xmx) reserves virtual address space.
  • Job memory MIN (xms) allocates (commits) the memory.  
  • The Java heap space needs to be contiguous (meta-space does not need to be contiguous).

Job Memory - Database Masking

These estimates are defined based on using In-Place Masking on masking a database.

  • Max job memory needed:
    • Memory needed per job: max_job_mem =  max "per table" x "number of streams".
    • Memory needed per table: max "per algorithm" x "number of masked columns in table".
    • Memory needed per algorithm: max "lookup value length" x "number of lookup values".
       
  • Min memory:
    • Enough on the VM host is if the host is over-committed or not.
    • If the memory has not been reserved or there is not enough memory on the VM host - set Min = Max / 4.
    • If the memory is reserved and there is enough on the VM host - set Min = Max.
    • Min = Max will improve performance slightly. 
       
  • NOTE: 
    • The performance will not improve with more memory. 
    • On standard jobs - there is no need to set the Min/Max memory.
    • If there are errors with memory, these usually result in crashes. 

Job Memory - File Masking

File masking is different as the whole row will be read into the masking engine and the memory. When all rows are masked, the rows will then be sorted back into the original order and this sort uses both memory and disk but if the sort needs to swap to disk the job is much slower.

The details below have been divided into Rows with many bytes and Rows with few bytes.

  • Max job memory needed:
    • Memory needed per job: max_job_mem =  max "per table" x "number of streams".
    • Rows with many bytes (long rows): 
      • Memory needed per table: max "bytes per row" x 10,000 x "number of masked columns in table" x 5 (est steps per algorithm).
    • Rows with few bytes (short rows):
      • Memory needed per table: max "per algorithm" x "number of masked columns in table".
      • Memory needed per algorithm: max "lookup value length" x "number of lookup values".
         
  • Min memory:
    • Enough on the VM host is if the host is over-committed or not.
    • If the memory has not been reserved or there is not enough memory on the VM host - set Min = Max / 4.
    • If the memory is reserved and there is enough on the VM host - set Min = Max.
    • Min = Max will improve performance slightly. 
       
  • Warnings:
    • If you don't have enough Job Max Memory - the job will crash.
    • If you don't have enough memory on the engine - the engine will crash. 
       
  • NOTE: 
    • The performance will not improve with more memory. 
    • On standard jobs - there is no need to set the Min/Max memory.
    • If there are errors with memory, these usually result in crashes. 

Issue 

Masking or Profiling job fails with the error messages:

  • Masking Engine: "java.lang.OutOfMemoryError: GC overhead limit exceeded".
  • Oracle message:  "Thrown when the Java Virtual Machine cannot allocate an object because it is out of memory and no more memory could be made available by the garbage collector.".

OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable. 

Troubleshooting Out of Memory Error

Some jobs need more memory than others. It depends on the number of tables, how many columns you are masking, and the type of algorithms assigned to the inventory. The current memory allocation may not be enough or the job is configured with too many threads/streams, or both.

Note

Note:

The default memory allocation is 1GB.

Example of Masking Engine -  DMsuite log with out of memory condition from Masking job:

image2016-5-12%2B12-21-48.png

Resolution

To resolve Out of Memory error, the memory allocated to the job must be increased and/or decrease the number of streams and threads which are consuming memory.  Rerun the job, once these changes are made to the memory.

Note

Note:

Do not exceed the amount of memory allocated to the VM machine.

To increase or decrease memory configuration parameters use MIN Memory and Max Memory fields on the Create Masking Job page.

masking_mem_config.png