Skip to main content
Delphix

Masking Inventory Import and Export (KBA1815)

 

 

KBA

KBA#1815

 

This document describes the Inventory Export and Import process.  Masked columns can be changed in multiple environments, Rule Sets, and Tables using the CSV import process. 

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Date Release
Nov 21, 2023 17.0.0.0
Oct 18, 2023 16.0.0.0
Sep 21, 2023 15.0.0.0
Aug 24, 2023 14.0.0.0
Jul 24, 2023 13.0.0.0
Jun 21, 2023 12.0.0.0
May 25, 2023 11.0.0.0
Apr 13, 2023 10.0.0.0 | 10.0.0.1
Mar 13, 2023 | Mar 20, 2023 9.0.0.0 | 9.0.0.1
Feb 13, 2023 8.0.0.0
Jan 12, 2023 7.0.0.0
Releases Prior to 2023
Major Release All Sub Releases
6.0

6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0, 6.0.12.0, 6.0.12.1, 6.0.13.0, 6.0.13.1, 6.0.14.0, 6.0.15.0, 6.0.16.0, 6.0.17.0, 6.0.17.1, 6.0.17.2

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

At a Glance 

Description:  This KBA describes the Export and Import CSV process to configure and modify the Masking Inventory.
Location: UI: Environments - [Environment] - Inventory 

API: Not available.
Data Exported: All properties in the selected Rule Set are Exported
Data Imported:
Required to: Required fields: 
Define Object These fields define which Inventory object to change: 
  • Environment Name
  • Rule Set
  • Table Name
  • Column Name

Note: The Environment and Rule Set can be any available (even multiple). 

Apply Change These fields define what property in the Inventory to change: 
  • Domain, Algorithm, Is Masked (all three needed)
  • ID Method*
  • Row Type
  • Date Format
  • Notes
  • Multi-Column Logical Field
  • Group Number

* Only available on some connector types, for example, database.

CSV Entries: The imported CSV only needs to contain the data that is modified. All other entries in the CSV can be removed.
 

CSV entries needed: 

  • If a column is to be masked. 
  • If a masked column is no longer to be masked.
  • If the masked column configuration has changed.
Limitation: The values in the Inventory CSV file are not enclosed with quotes. This means that any entry with a comma in it will corrupt the CSV file. 

There are two workarounds that might resolve or limit the impact of this limitation: 
  1. Only CSV records that are modifying the masking configuration are needed. All others can be removed.
  2. Files Masking: Export the Inventory on a File (or Pattern) without commas in the name.
Pup-ups: Most common issue: popups need to be enabled for import to succeed. 
Updates: The CSV Format was updated in version 6.0.9.0. Three fields were added:
  • 'Notes'
  • 'Multi-Column Logical Field'
  • 'Group Number'
Related Docs:  Masking Documentation:

 

User Interface

Export Inventory 

To export the Rule Set Inventory to a CSV:

  1. Navigate to UI: Environments > [Environment Name] > Inventory
  2. Select the Rule Set to export.
  3. Click Export.
    This opens the Export Inventory dialog.
  4. Click Save (the File Name can be changed).
    This will open a popup indicating that the CSV file is being generated.
    After the file has been created, the popup will change. This may take some time, depending on the size of the inventory.
  5. Click the Download File link to download.

KBA1815_-_Export_Steps.png

Import Inventory 

To import a CSV with changes to an Inventory:

  1. Navigate to UI: Environments > [Environment Name] > Inventory.
  2. Click Import.
    This opens the Import Inventory dialog.
  3. Click Select and browse to the file for import and click Save.
  4. Wait for the status popup.

Notes:

  • Browser popups must be enabled.
  • It takes a few minutes to display the status popup.
  • The size of the inventory uploaded impacts the time you need to wait. It can be significant.
  • Speed up Import by removing CSV entries not required (see below).

 

KBA1815_-_Import_Steps.png

Popup 

Import stages and the final import status message are displayed in a browser popup (as in the example above).

Note

Note:

 Popups need to be enabled on the browser for the engine URL.

 

CSV File Format 

The file format for the exported and imported CSV file is detailed below. 

Database

Export Description

Only the selected Rule Set will be exported. 

+-----------------------------+------------+----------+----------------------------------------------------------------+
| CSV Field                   | Value      | Export   | Export comment                                                 |
+-----------------------------+------------+----------+----------------------------------------------------------------+
| Environment Name            | As in UI   | Selected | Only the selected Environment.                 |
| Rule Set                    | As in UI   | Selected | Only the selected Rule Set.                                    |
+-----------------------------+------------+----------+----------------------------------------------------------------+
| Table Name                  | As in UI   | All      | All tables in the Rule Set.                                    |
| Type                        | -          | -        | Normally not used - default '-'.                               |
| Parent Column Name          | -          | -        | Normally not used - default '-'.                               |
| Column Name                 | As in UI   | All      | All columns in the Rule Set.                                   |
| Data Type                   | As in UI   | All      | The Data Type for specified column.                            |
| Domain                      | As in UI   | All      | The assigned Domain for specified column.                      |
| Algorithm                   | As in UI   | All      | The assigned algorithm for specified column.                   |
| Is Masked                   | TRUE/FALSE | All      | TRUE = specified column is masked. FALSE = Not Masked.         |
| ID Method                   | Auto/User  | All      | User = Domain/Algorithm is locked. Auto = profiler can change. |
| Row Type                    | 'All Row'  | All      | Normally not used - default 'All Row'.                         |
| Date Format                 | As in UI   | All      | The assigned Date Format for specified column.                 |
| Notes*                      | As in UI   | All      | To record notes.                                               |
| Multi-Column Logical Field* | As in UI   | All      | Used with Multi-Column algorithm.                              |
| Group Number*               | As in UI   | All      | Used with Multi-Column algorithm.                              |
+-----------------------------+------------+----------+----------------------------------------------------------------+
  • * These columns were added in version 6.0.9.0.

Import Description

+-----------------------------+------------+----------+--------------------------------+
| CSV Field                   | Value      | Import   | Import comment                 |
+-----------------------------+------------+----------+--------------------------------+
| Environment Name            | As in UI   | Req      | Need to match existing.        |
| Rule Set                    | As in UI   | Req      | Need to match existing.        |
| Table Name                  | As in UI   | Req/Mult | Need to match existing.        |
| Type                        | -          | -        | Normally not used.             |
| Parent Column Name          | -          | -        | Normally not used.             |
| Column Name                 | As in UI   | Req/Mult | Need to match existing.        |
+-----------------------------+------------+----------+--------------------------------+
| Data Type                   | -          | -        | No effect.                     |
+-----------------------------+------------+----------+--------------------------------+
| Domain                      | As in UI   | Optional | Set these 3 always together.   |
| Algorithm                   | As in UI   | Optional | Set these 3 always together.   |
| Is Masked                   | TRUE/FALSE | Optional | Set these 3 always together.   |
+-----------------------------+------------+----------+--------------------------------+
| ID Method                   | Auto/User  | Optional | Set by import.                 |
| Row Type                    | 'All Row'  | Optional | Set by import.                 |
| Date Format                 | As in UI   | Optional | Set by import.                 |
| Notes*                      | As in UI   | Optional | Set by import.                 |
| Multi-Column Logical Field* | As in UI   | Optional | Set by import.                 |
| Group Number*               | As in UI   | Optional | Set by import.                 |
+-----------------------------+------------+----------+--------------------------------+
  • "Req" - Filed is required. 
  • "Mult" - Multiple objects can be updated with one CSV file. 
  • "Optional" - Optional, only rows that modify the Inventory are needed. All other rows can be excluded, which reduces import time on large Rule Sets. 
     
  • 'Set these 3 always together'  - always set these three to the desired correct value (if masked and if not masked).
     
  • * These columns were added in version 6.0.9.0.

Example

The example below includes the Header and one column.

Environment Name,Rule Set,Table Name,Type,Parent Column Name,Column Name,Data Type,Domain,Algorithm,Is Masked,ID Method,Row Type,Date Format,Notes,Multi-Column Logical Field,Group Number
MyEnv,MyRS,MyTable,-,-,MyColumn,nchar (10),FIRST_NAME,FIRST NAME SL,true,Auto,All Row,-,-,-,-

Success / Troubleshooting

Popup must be enabled  

The import process requires popups to be enabled.

When importing a CSV file, a notification will be shown in the browser if popups are disabled. Look for Browser notifications. 

Resolution  

Navigate to the browser settings and enable popups for the masking engine page. 

Performance

The Inventory Export and Import can take a long time depending on the size of the inventory involved.  It can take many minutes to process millions of columns in an inventory (this is a very large Rule Set). 

In both export and import scenarios, you have to wait for the popup which shows the process to be complete. The Export/Import will terminate if the browser is closed or the session expires.

A message in the popup will indicate when the process is finished or if there is an error.  

Reducing import time 

The time taken to import can be improved by removing CSV entries that have not changed (usually this is all fields not masked).

If the Rule Set has 100,000 columns and only 1000 is masked - to set all masked columns will only require 1/100th of the entries which will reduce the import time from many minutes to a few seconds.

Comma in Fields

If there is a comma in a field value then the CSV has an invalid format. The Inventory CSV is a standard CSV and values are not enclosed (quoted). As such values cannot have a comma in them.

File Format Pattern or File

If the File Format Pattern or File has a comma this can be worked around by creating (or editing) a Pattern without a comma. Then Export and Import the Rule Set. The Rule Set can then be edited to include the comma.

tip

Tip:

This works because the Inventory is assigned to the File Format (and not each individual file or pattern).

 

Remove row with comma

Remove the row (or rows) with comma from the CSV file and Import the file. 

This usually has no impact on the Inventory as only changes are needed in the CSV. If needed, manually change the Inventory. 

Specified object is not valid 

If a referenced object (such as a Table, Column, Domain, or Algorithm) is not available on the Masking Engine a warning is shown in the popup. 

Warnings are:

  • "Specified Environment name [name] is not valid"
  • "Specified Ruleset name [name] is not valid"
  • "Specified Table name [name] is not valid"
  • "Specified Column name [name] is not valid"
  • "Specified domain [name] is not valid"
  • "Specified Algorithm [name] is not valid"
  • "Specified Date Format [format] is not valid"
  • "Unmatched Inventory not updated into Application"

Any object with this warning was not applied and updated in the Inventory. Please verify the name and if the object exists in the Rule Set and try again.

Note

Notes:

  1. Correct spelling and case are required to reference objects.
  2. The reference to algorithms is the Algorithm Name as shown in the UI.

Examples of these warnings can be seen below:

Import Inventory CSV Warnings - example.png

Masking Logs 

Errors in the import process are logged in the masking log (some object warnings are currently not logged). Below are some examples related to the warning shown in the popup above.

Reading each line of the csv imported Inventory: Environment,RS,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Environment Environment does not exist

Reading each line of the csv imported Inventory: Env,RuleSet,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Rule Set RuleSet does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Column,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Env Env, Rule Set RS_BUG : Column Column does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Col,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,XXXX@MM!dd
Input date format:XXXX@MM!dd Valid:false

Resolution 

Depending on what object is throwing the warning, please check the object on the Masking Engine UI and correct the reference to the object, the spelling, the case, or the date format. 

To get an example of the CSV file, please export a file from the UI.

Incorrect number of columns in CSV 

If the CSV file is missing a comma or column(s), the following error will be thrown in the UI "Error occurred during importing inventory. 


Import Inventory CSV Error.png

Checking the masking logs provides more information:

Proceeding to reading and checking that csv has proper formatting
...
Reading each line of the csv imported Inventory: Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-
Exception :
java.lang.RuntimeException: Unexpected number of fields on line 19 of imported Inventory: expected 13, found 9
...

Resolution 

Check the masking logs, the row, and the error. The error shows how many columns are expected and the failed row is shown above. 

From the example above:

Error: 
  * Unexpected number of fields
  * on line 19 
  * expected 13, found 9
  
Row:
  * Is      : "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-"
  * Expected: "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-,true,Auto,All Row,-"
    

Invalid File Format 

If the header is missing or has incorrect fields, the following error is shown: "Invalid File Format". The popup will look like this:

Import Inventory CSV Invalid File Format.png

The logs will not show any imported rows (and no error message):

User has requested to import a csv file to Inventory
csv file upload successful to path: /var/delphix/dmsuite/import/delphix_admin/ImportInventory
...
If there is no error and inventory does not get updated, make sure popup is enabled.

Resolution  

The best way to resolve this issue is the export a Rule Set (Inventory) and copy the header from this file and paste it into the failed import file.

Invalid characters in the CSV file 

The characters need to match the objects in the Rule Set and algorithm names. Invalid characters in the CSV file can and is likely to cause undesired behaviors which are not easy to detect.

Example of issues:

  • Trailing and leading spaces.
  • Incorrect end-of-line character(s).
  • Invalid character in the row with data.
  • A column name that includes a comma (any column names that include a comma will need to have their algorithm set manually in the GUI)

Resolution 

The best resolution here is to be diligent in editing the CSV file. Open it in a text editor, save it to a standard ANSII character set and check for invalid characters. 

Related Articles

Masking Documentation: