File Masking Record Types - Headers, Trailers, and Filters (KBA7064)

At a Glance

Description:	The Continuous Compliance Engine can mask Files using different file formats (different fields and algorithm configuration) in the same file by using Record Types. This page describes how to define record types for file masking including header, trailer, and multiple bodies with filters for delimited and fixed files.
How To:	The Workflow for Defining Record Types are: Define File Format (one for All Records and one per format). Create Rule Set and configure the File Format: Set the All Records - File Format. Set the EOR, delimiter (if applicable), and enclosure. Define Inventory - where applicable: Define the Record Type - Header/Trailer Define the Record Type - Body and assign file format and RegEx conditions.
Multiple RTs:	Records Types (RT) conditions can be defined on 1 to 3 fields.
RT RegEx:	Record Type filter conditions are defined on the value in the specified field - matched using RegEx.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
4.3	4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0
4.2	4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1
4.1	4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Record Type Configurations

The first step is to create Masking Connectors for file masking. File masking is best configured as an On-The-Fly job as this job type is faster (shortest runtime) and it provides a better workflow.

Step 1 - Create File Format Files

This article covers three different file types. To define the file format, the first step is to create the file format in a text editor.

Example of Delimited and Fixed Width

Below are two examples of delimited and fixed width file formats (consult the IBM documentation for Copybook as it is too complex for a quick sample).

Delimited - File Format example

Name
Address
City
State

Fixed Width - File Format example

Name,25
Address,40
City,20
State,2

Special Characters

There are some characters that are invalid when creating the file format - these are:

!@#$%^&*()+=-[]\';,./{}|":<>?~`

All Records - File Formats

The file format for all records is the default file format for the file.

Multiple File Formats

If multiple record types are to be used (the masked rows need to be filtered), you then need to create one file format for each additional record type.

It is recommended to name each file format so it is easy to see the record type (RT) and column name. The column names should also match the actual field name.

The example below is for a delimited file:

RT1_Description1.txt

RT1_col1
RT1_col2
RT1_col3
...

RT2_Description2.txt

RT2_col1
RT2_col2
RT2_col3
...
RT2_col9

Step 2 - Import the File Format

The second step is to Import and create the default file format. This is only done on the main (default) file format which will be called All Records (all additional file formats are added in the inventory).

Navigate to Settings > File Format, and then click Import File Format.
Select Import Format Type.
Import fields by clicking Select, and then select the default File Format.
Save the setting.

Step 3 - Create Rule Set

When you create the Rule Set, set the format details (delimiter, EOR, and Enclosed character if used). This will be the default record type called All Records.

Step 4 - Define Inventory

To define record types in the user interface:

Navigate to Environment > Inventory and click Record Types.

Define Header

If applicable, define the Header (predefined unmasked rows at the top of the file).

Click + Add a Record Type.
Define the Header details:
- Name the Record Type.
- For the Header/Body/Trailer - select Header.
- Define the number of rows (the first n rows will be ignored when masking).

Define Trailer

If applicable, define the Trailer (predefined unmasked rows at the top of the file).

Click + Add a Record Type.
Define the Header details:
- Name the Record Type.
- For the 'Header/Body/Trailer' - select Trailer.
- Define the number of rows (the last n rows will be ignored when masking).

Define Body (File Format)

For each File Format:

Click + Add a Record Type.
Define the Body details:
- Name the Record Type.
- For the Header/Body/Trailer - select Body.
- # of identifier fields - this is how many fields should be used in the filter value (1, 2, or 3 fields)
- Import Fields - Import the file format for this record type - select the specific file created above.
- Define the ID #n Regular Expression and Position #
  - Depending on Id fields (1, 2, or 3 above there will be 1, 2, or 3 fields to fill in).
  - Define the RegEx matching the values in this field (if exact then write this value).
  - Detail which field Position # contains the record type identifier.

Example

Record Type and Regular Expression
The regular expression (RegEx) is used to say mask the file format using the same algorithms.

If for example, these record types are to be masked (N1, N2, N3, ...) and they all had the same file format, you could use RegEx to create just one record type.

For example, the RegEx would be:

N[0-9]

The configuration would be:

Record type: Header (1 row)
Record type Nx
- # of identifier fields = 1
- ID #1 Regular Expression = N[0-9]
- Position # = 1

Input file:

RT*MASK*ORIG
N1*1*1~
N2*12*12~
N3*123*123~
M1*1xyz*1xyz~
A2*2xyz*2xyz~

Masked file:

RT*MASK*ORIG
N1*7*1~
N2*73*12~
N3*739*123~
M1*1xyz*1xyz~
A2*2xyz*2xyz~

Errors and Resolution

Masking job becomes unresponsive

The masking job will likely become unresponsive if there is no All Records. The masking job will read the data and for records that do not match any record type the engine will buffer these until the buffer is full, and then stop responding.

Resolution

Make sure to create a default All Records file format and do not define any filter on this record type.

Masked file has additional Fields

If the file format for a row is defined to have 4 fields and the input only has 3 fields - the engine will create a row with all 4 fields and set the last field to NULL (empty).

The file format is an absolute definition of the row, and the engine will use the file format as a specification for how many rows to create in the masked output.

KBA