Best Practice: Job Configuration on the Masking Engine (KBA1307)
This article defines the best practice for:
- initial configuration of masking and profiling jobs
- general engine configuration
Job and Engine Configuration
Detailed here are:
- Profiling Job configuration
- Masking Job configuration
- Masking Engine configuration
Originally the Masking Engine was usually located on the same server as the database and as such the requirements and limiting factors were different. Today, we are seeing more remote connections and larger databases. As a result, the requirements for configuration have changed.
This article has been created to address the optimal initial configuration. Once the job has been confirmed working, these can be tweaked to increase performance.
Profiling Job Configuration
There are two different types of Profiling:
- Column Level
- Data Level.
The type is defined in the Profiling Set.
The Column Level profiling uses RegEx on the Column name in the Inventory. The Data Level profiling uses RegEx on a sample set of the data in the column. If there is no match using the Column Level and there is a Data Level defined, a profile job is started for that column.
When Profiling using Column Level only (default) there is no need to change the job configurations.
When using Data Level profiling there can be many thousands of columns to profile. The best practice initial configuration is to start with 1 stream:
- Streams: 1
- Memory (JVM xms and xmx)
- Min: 2048 MB
- Max: 4056 MB
Masking Job Configuration
There are two different types of masking jobs: Database and File. Note that the configurations are fairly similar. The objective for these initial settings is to have a successful execution.
Job Configuration: Database
Best practice for initial masking job configuration:
- Masking Method: In-Place
- No. of Streams: 1
- Update Threads: 1
- Memory (JVM xms and xmx)
- Min: 2048 MB (needs to be defined)
- Max: 4096 MB (or larger as needed)
- Feedback and Commit Size: leave blank (default)
- Drop Indexes, Disable Constraints, and Triggers:
- Optional - Set these as it will improve performance and Indexes, Constraints and Triggers are a common cause for issues.
Job Configuration: File
Best practice for initial masking job configuration:
- Masking Method: On-the-fly
- No. of Streams: 1
- Memory
- Min: 2048 MB (needs to be defined)
- Max: 4096 MB (or larger as needed)
- Feedback Size: leave blank (default)
Feedback Size
The Feedback Size defines how frequent logs are written to the log files. These values are guidelines and one way to determine the size is that the logs should preferably fit into one log file.
Database Size | Max number of Records in a table | Feedback Size value |
Small to medium | Up to 10,000,000 | 50,000 (default) |
Large | Up to 500,000,000 | 500,000 |
Very large | Over 500,000,000 | 5,000,000 |
Max Memory (xmx)
The amount of memory needed is defined by the number of columns masked and the masking algorithms used. The size, the number of records, has no impact on the memory requirements as the data is processed in a stream.
When would the Max need to be increased?
- A large number of columns per tables masked - more than 10.
- A large number of lookup values in Secure Lookup algorithm.
- A large number of lookup values in Mapping algorithm.
- Complex Segmented Mapping algorithms.
- Complex custom mapplets.
Troubleshooting Diagnosis (Error Signatures)
Some signs that the engine is not configured correctly are:
- Masking Engine (dmsuite/debug) logs contains:
- "OutOfMemoryError" such as "OutOfMemoryError: GC overhead limit exceeded".
- "ERROR: insert or update on table 'xxx' violates foreign key constraint...".
- The Masking/Profiling Job exits with incorrect status.
- Hanging Jobs where the status shows "Running".
- The Masking Engine restarts.
Related Articles
- Masking Performance for Database
- Column and Data Level Profiling
- Best Practice: File Masking Job Configuration
- Export/Import Masking Environments
- https://docs.delphix.com/docs/delphix-masking/delphix-masking-engine-system-requirements
- https://docs.delphix.com/docs/delphix-masking/masking-engine-admin-guide/managing-profiler-settings
- https://docs.delphix.com/docs/delphix-masking/delphix-masking-quick-start-guide/masking-engine-activities/mask-data
- Masking Prerequisite