Best Practice: Job Configuration on the Masking Engine (KBA1307)

Last updated
Save as PDF
Share
1. Share
2. Tweet
3. Share

This article defines the best practice for:

initial configuration of masking and profiling jobs
general engine configuration

Job and Engine Configuration

Detailed here are:

Profiling Job configuration
Masking Job configuration
Masking Engine configuration

Originally the Masking Engine was usually located on the same server as the database and as such the requirements and limiting factors were different. Today, we are seeing more remote connections and larger databases. As a result, the requirements for configuration have changed.

This article has been created to address the optimal initial configuration. Once the job has been confirmed working, these can be tweaked to increase performance.

Profiling Job Configuration

There are two different types of Profiling:

Column Level
Data Level.

The type is defined in the Profiling Set.

The Column Level profiling uses RegEx on the Column name in the Inventory. The Data Level profiling uses RegEx on a sample set of the data in the column. If there is no match using the Column Level and there is a Data Level defined, a profile job is started for that column.

When Profiling using Column Level only (default) there is no need to change the job configurations.

When using Data Level profiling there can be many thousands of columns to profile. The best practice initial configuration is to start with 1 stream:

Streams: 1
Memory (JVM xms and xmx)
- Min: 2048 MB
- Max: 4056 MB

Masking Job Configuration

There are two different types of masking jobs: Database and File. Note that the configurations are fairly similar. The objective for these initial settings is to have a successful execution.

Job Configuration: Database

Best practice for initial masking job configuration:

Masking Method: In-Place
No. of Streams: 1
Update Threads: 1
Memory (JVM xms and xmx)
- Min: 2048 MB (needs to be defined)
- Max: 4096 MB (or larger as needed)
Feedback and Commit Size: leave blank (default)
Drop Indexes, Disable Constraints, and Triggers:
- Optional - Set these as it will improve performance and Indexes, Constraints and Triggers are a common cause for issues.

Job Configuration: File

Best practice for initial masking job configuration:

Masking Method: On-the-fly
No. of Streams: 1
Memory
- Min: 2048 MB (needs to be defined)
- Max: 4096 MB (or larger as needed)
Feedback Size: leave blank (default)

Feedback Size

The Feedback Size defines how frequent logs are written to the log files. These values are guidelines and one way to determine the size is that the logs should preferably fit into one log file.

Database Size	Max number of Records in a table	Feedback Size value
Small to medium	Up to 10,000,000	50,000 (default)
Large	Up to 500,000,000	500,000
Very large	Over 500,000,000	5,000,000

Max Memory (xmx)

The amount of memory needed is defined by the number of columns masked and the masking algorithms used. The size, the number of records, has no impact on the memory requirements as the data is processed in a stream.

When would the Max need to be increased?

A large number of columns per tables masked - more than 10.
A large number of lookup values in Secure Lookup algorithm.
A large number of lookup values in Mapping algorithm.
Complex Segmented Mapping algorithms.
Complex custom mapplets.

Troubleshooting Diagnosis (Error Signatures)

Some signs that the engine is not configured correctly are:

Masking Engine (dmsuite/debug) logs contains:
- "OutOfMemoryError" such as "OutOfMemoryError: GC overhead limit exceeded".
- "ERROR: insert or update on table 'xxx' violates foreign key constraint...".
The Masking/Profiling Job exits with incorrect status.
Hanging Jobs where the status shows "Running".
The Masking Engine restarts.