Skip to main content
Delphix

Algorithm - Secure Lookup (SL)

 

Applicable Delphix Masking Versions

  • 5.2.x
  • 5.1.x
  • 5.0.x
  • 4.x

At a Glance

Description:

This is a very commonly used algorithm and there are around 20 predefined secure lookup algorithms provided out of the box. 
The algorithm is fast and lightweight (when used correctly).

Characteristics:

This algorithm produces a 1:N mapping (the same masked value will appear many times).

Fairly lightweight and fast.  

Character 
Encodings:

Works with all characters encodings (some bugs are known). Encoding should give the same result. Data is trimmed. 
SL is Case Sensitive (Anders and ANDERS are different).
See Issue 2 below for the workaround.  

Lookup Pool Size: Recommended size up to 500,000
Limitations: Large pool size (rows in the lookup file) will take a long time to load into the Transformation 
Engine. Please see below.
Customization:

Creation: Lookup values are loaded when the algorithm is created from a text file.

Modify: The Description and the Lookup Value cab be changed. 

 

High-Level Overview

Secure Lookup is the most common algorithm and there are around 20 predefined secure lookup algorithms provided out of the box. 

This document describes how Secure Lookup works and how to create a new one. 

The Secure Lookup algorithm is a simple algorithm which uses an XML file. These values are imported in the creation of the lookup algorithm. A Modulus of a hashcode of the encrypted input is used as a lookup value. This method ensures that the lookup value returns the same result each time and that the number of lookup values never runs out. Since the values in the output are reused, this algorithm is not suited for indexes and when the values need to be unique. 

It is recommended to limit the number of items in the Secure Lookup to under 500,000.  
 

UI - Creation and Modification  

The algorithms are accessed from the tab Settings and under the submenu Algorithm. A custom algorithm can be created and modified (edit):

  • To create click "Add Algorithm".
  • To modify click the green pen in the "Edit" column.

User Interface 

The following popup is accessed when creating and modifying the algorithm:

image.png

  1. Algorithm Properties
    • Algorithm Name
    • Description
  2. Lookup Details
    • Lookup File Name

Modification 

Editable parameters are:

  • Description
  • List of lookup values

Note - to change any other parameter the algorithm needs to be deleted and then recreated. 

Considerations 

Pool Size and Performance 

The pool size (number of rows in the lookup file) affects both the creation time and the load time. An estimate is that the increase in load time is 10 to 3 - increase the number of values to load by 3 and it will take 10 times longer. On an average system, it will take 4 min to load 500,000 values. Loading 1,500,00 will take 40 min. 

The graph below shows the load values in milliseconds (y) and in the number of values loaded on the horizontal axis (x). 

SL_ToTLoadTime.png

Memory Requirements 

The ingested values are loaded into RAM when the masking job starts. For large pool sizes, it might be required to increase the Min/Max Memory settings in the Job Configuration. 

Best Practice: Job Configuration on the Masking Engine

Case Sensitivity  

The Secure Lookup is case sensitive. This can be a problem if the same result is expected independent of the case.

To solve this, use this workaround:

  1. Open and edit the Rule Set
  2. Add a Custom SQL statement with the following amendment:

Change : ..., maskMeCol, ...

to: ..., UPPER(maskMeCol) as maskMeCol, ...

Example:

SELECT ID, UPPER(maskMeCol) as maskMeCol from myTable;

Masked Data Encoded with Non-Standard Code Page 

The data to be masked might be encoded and contain characters that are not converted correctly to UTF-8. The masked data is masked as "????". This issue is resolved in version 5.1.

If experiencing this issue please raise a case with Delphix support. 

Examples 

Below is an example of mask result. Note the case sensitivity. 

FIRST NAME SL was used in this example: 

+-----+--------+----------+
| Ref | Source | Masked   |
+=====+========+==========+
| 1   | Peter  | Flo      |
+-----+--------+----------+
| 2   | PETER  | Yoshiko  |
+-----+--------+----------+
| 3   | peter  | Paul     |
+-----+--------+----------+
| 4   |  Peter | Flo      |  << Leading and/or trailing whitespaces are trimmed.
+-----+--------+----------+     Result same as Ref 1.

Additional Information

Delphix Documentation:

 

 

 

 

  • Was this article helpful?