Skip to main content
Delphix

Inventory Export and Import

 

 

Applicable Delphix Masking Versions

This feature is applicable to all known versions:

  • 5.2.x  (version 5.2.3.1 has been used in the examples below)
  • 5.1.x
  • 5.0.x
  • 4.x

Summary

This document describes the Inventory Export and Import process. 

Masked columns can be changed in multiple Environments, Rule Sets, and Tables using the CSV import process. 

The features include:

  • Export will extract and export the selected Rule Set and all:
    • Tables
    • Columns  
    • Column properties:
      • Data Type and Length
      • Is Masked flag 
      • Assigned Domain and Algorithm
      • (For Date) Date format
  • Import and update multiple one or multiple:
    • Environments
    • Rule Sets
    • Tables
    •  Columns
  • Only the modified data needs to be imported.
    • If a column is to be masked this column only need to be included in the CSV file. 
    • If a masked column is not to be masked anymore, only this column needs to be included in the CSV file. 

UI - Export, Import, and Pop-up

Export Inventory

To export the Inventory of a Rule Set, select the Rule Set and press Export. 

Steps

  1. Select the Rule Set to export - in the example below 'RuleSet'.
  2. Press Export.
  3. If desired, change the File Name.
  4. Press Save - this will open a pop-up indicating that the CSV file is being generated.
  5. After the file has been created, the popup will change. Click the link to download the file.

UI Inventory Export w popup .png

Import Inventory

To import changes to the Inventory, follow these steps:

  1. Press Import.
  2. Click Select and browse and select a file to import.
  3. Press Save
  4. Wait for the pop-up to report import status.

UI Inventory Import w popup.png

Pop-up

The stages and the final status message is shown in a browser pop-up (success shown in the two examples above). Therefore, Pop-ups need to be enabled on the browser for the engine URL.

Examples of warnings and errors are shown below. 

Data Format

CSV File

The file format for the exported and imported CSV file is detailed below. 

+--------------------+----------+----------+---------------------------------------------------------------------------------+
| CSV Field          | Export   | Import   | Comment                                                                         |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Environment Name   | Selected | Multiple | Export only the selected Rule Set (Environment), Import works on multiple.      |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Rule Set           | Selected | Multiple | Export only the selected Rule Set, Import works on multiple.                    |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Table Name         | All      | Multiple | Export all tables in the Rule Set, Import as needed (need to be valid Tables).  |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Type               | -        | -        | -                                                                               |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Parent Column Name | -        | -        | -                                                                               |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Column Name        | All      | Multiple | Export all columns in the Rule Set, Import as needed (need to be valid Columns) |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Data Type          | Specific | -        | Export specific data type and length to assist setting Domain and Algorithm.    |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Domain             | Specific | Specific | Export specific current configuration, Import will set this configuration.      |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Algorithm          | Specific | Specific | Export specific current configuration, Import will set this configuration.      |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Is Masked          | Specific | Specific | Export specific current configuration, Import will set this configuration.      |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| ID Method          | Specific | Specific | Export specific current configuration, Import will set this configuration.      |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Row Type           | Specific | Specific | Export specific current configuration, Import will set this configuration.      |
+--------------------+----------+----------+---------------------------------------------------------------------------------+
| Date Format        | Specific | Specific | Export specific current configuration, Import will set this configuration.      |
+--------------------+----------+----------+---------------------------------------------------------------------------------+

Example:

Environment Name,Rule Set,Table Name,Type,Parent Column Name,Column Name,Data Type,Domain,Algorithm,Is Masked,ID Method,Row Type,Date Format
Environment,RuleSet,Table,-,-,Column,NVARCHAR2 (50),LAST_NAME,LAST NAME SL,TRUE,Auto,All Row,-

Possible Issues and Troubleshooting

Pop-up must be enabled 

The import process requires pop-ups to be enabled. When importing a CSV file a notification will be shown in the browser should pop-ups be disabled. 

The example below is from Chrome on Windows. 

doccontrol-faq-2.jpg

Resolution 

Go to the browser settings and enable pop-ups for the masking engine page. 

Referenced object not available on the Masking Engine

If a referenced object (such as Environment, Rule Set, Table, Column, Domain, Algorithm) is not available on the Masking Engine a warning is shown in the pop-up. 

Note:

  1. Correct spelling and case are required to reference objects.
  2. The reference to algorithms is the Algorithm Name as shown in the UI.

Examples of these warnings can be seen below:

Import Inventory CSV Warnings - example.png

Masking Logs

Errors in the import process are logged in the masking log (some object warnings are currently not logged). Below are some examples related to the warning shown in the pop-up above.

Reading each line of the csv imported Inventory: Environment,RS,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Environment Environment does not exist

Reading each line of the csv imported Inventory: Env,RuleSet,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Rule Set RuleSet does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Column,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Env Env, Rule Set RS_BUG : Column Column does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Col,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,XXXX@MM!dd
Input date format:XXXX@MM!dd Valid:false

Resolution

Depending on what object is throwing the warning, please check the object on the Masking Engine UI and correct the reference to the object, the spelling, the case, or for the date the format. 

To get an example of the CSV file, please Export a file from the UI.

Incorrect number of columns in the CSV file

If the CSV file is missing a comma or column(s) the following error will be thrown in the UI "Error occurred during importing inventory. Please refer application logs.", as shown below.
Import Inventory CSV Error.png

Checking the masking logs provides more information:

Proceeding to reading and checking that csv has proper formatting
...
Reading each line of the csv imported Inventory: Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-
Exception :
java.lang.RuntimeException: Unexpected number of fields on line 19 of imported Inventory: expected 13, found 9
...

Resolution

Check the masking logs and the row and the error. The error shows how many columns are expected and the failed row is shown above. 

From the example above :

Error: 
  * Unexpected number of fields
  * on line 19 
  * expected 13, found 9
  
Row:
  * Is      : "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-"
  * Expected: "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-,true,Auto,All Row,-"
    

No (invalid) header fields (Invalid File Format)

If the hearer is missing or has incorrect fields the following error is shown: "Invalid File Format". The pop-up will look like this:
Import Inventory CSV Invalid File Format.png

The logs will not show any imported rows (and no error message):

User has requested to import a csv file to Inventory
csv file upload successful to path: /var/delphix/dmsuite/import/delphix_admin/ImportInventory
...
If there is no error and inventory does not get updated, make sure pop-up is enabled.

Resolution 

The best way to resolve this issue is the export a Rule Set (Inventory) and copy the header from this file and paste it into the failed import file.

Invalid characters in the CSV file

The characters need to match the objects in the Rule Set and algorithm names. Invalid characters in the CSV file can and is likely to cause undesired behaviors which are not easy to detect.

Example of issues:

  • Trailing and leading spaces.
  • Incorrect end-of-line character(s).
  • Invalid character in the row with data.

Resolution

The best resolution here is to be diligent in editing the CSV file. Open it in a text editor, save it to a standard ANSII character set and check for invalid characters. 

Additional Information

Details from Delphix documentation:

 

 

 

 

 

  • The imported row is shown and how many columns are expected and missing.