Skip to main content
Delphix

Inventory Import and Export (KBA1815)

 

This document describes the Inventory Export and Import process.  Masked columns can be changed in multiple environments, Rule Sets, and Tables using the CSV import process. 

At a Glance 

Versions: Applicable Delphix Masking versions: 4.x, 5.0.x, 5.1.x, 5.2.x, 5.3.x
Description:  Export and Import of CSV file in order to review and quickly modify Inventory configurations.
Location: UI: Environments - [Environment] - Inventory 
Exported:
Selected Environment Name, Rule Set.
All All Rule Set properties for all other fields. 
Imported:
Required Environment Name, Rule Set, Table Name, Column Name.
Optional For column properties where these have changed: 
Domain, Algorithm, Is Masked, ID Method, Row Type, Date Format.
Character 
Encoding:
For multi-byte encoding, the CSV file needs to be imported using UTF-8 (no BOM).

 

Note

Note:

Only the modified data needs to be imported. For example: 

  • If a column is to be masked. 
  • If a masked column is not to be masked anymore.
  • If the masked column configuration has changed.

 

User Interface: Export, Import, and Pop-up 

Export Inventory 

To export the Inventory of a Rule Set:

  1. Navigate to Environments > Inventory
  2. Select the Rule Set to export which in this example is 'RuleSet'.
  3. Click Export.
    This opens the Export Inventory dialog.
  4. If desired, change the File Name.
  5. Click Save.
    This will open a pop-up indicating that the CSV file is being generated.
  6. After the file has been created, the popup will change. 
  7. Click the Download File link to download.

UI Inventory Page and Export Pop-up .png

Import Inventory 

To import changes to the Inventory:

  1. Navigate to Environments > Inventory
  2. Click Import.
    This opens the Import Inventory dialog.
  3. Click Select and browse to the file for import.
  4. Click Save.
    It takes a few minutes to display the pop-up which reports import status.

UI Inventory Import w popup .png

Pop-up 

The stages and the final status message are displayed in a browser pop-up (as in the two examples above).

Note

Note:

 Pop-ups need to be enabled on the browser for the engine URL.

See Troubleshooting and Resolution for examples of warnings and errors. 

Data Format 

The file format for the exported and imported CSV file is detailed below. 

CSV File - Export Description

Only the selected Rule Set will be exported. 

+--------------------+----------------+----------+-------------------------------------------------------------------------+
| CSV Field          | Value          | Export   | Export comment                                                          |
+--------------------+----------------+----------+-------------------------------------------------------------------------+
| Environment Name   | As in UI       | Selected | Only the selected Environment as per Rule Set.                          |
| Rule Set           | As in UI       | Selected | Only the selected Rule Set.                                             |
+--------------------+----------------+----------+-------------------------------------------------------------------------+
| Table Name         | As in UI       | All      | All tables in the Rule Set.                                             |
| Type               | -              | -        | Normally not used - default '-'.                                        |
| Parent Column Name | -              | -        | Normally not used - default '-'.                                        |
| Column Name        | As in UI       | All      | All columns in the Rule Set.                                            |
| Data Type          | As in UI       | All      | The Data Type for specified column.                                     |
| Domain             | As in UI       | All      | The assigned Domain for specified column.                               |
| Algorithm          | As in UI       | All      | The assigned algorithm for specified column.                            |
| Is Masked          | 'TRUE'/'FALSE' | All      | Set to TRUE if specified column is masked, otherwise FALSE.             |
| ID Method          | 'Auto'/'User'  | All      | Set to User if Domain/Algorithm is locked. Auto if profiler can change. |
| Row Type           | 'All Row'      | All      | Normally not used - default 'All Row'.                                  |
| Date Format        | As in UI       | All      | The assigned Date Format for specified column.                          |
+--------------------+----------------+----------+-------------------------------------------------------------------------+

CSV File - Import Description

+--------------------+----------------+-----------+----------------------------------------------------+
| CSV Field          | Value          | Import    | Import comment                                     |
+--------------------+----------------+-----------+----------------------------------------------------+
| Environment Name   | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
| Rule Set           | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
| Table Name         | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
| Type               | -              | -         | Normally not used.                                 |
| Parent Column Name | -              | -         | Normally not used.                                 |
| Column Name        | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
+--------------------+----------------+-----------+----------------------------------------------------+
| Data Type          | -              | -         | Not used.                                          |
+--------------------+----------------+-----------+----------------------------------------------------+
| Domain             | As in UI       | Optional* | Import will set this configuration.                |
| Algorithm          | As in UI       | Optional* | Import will set this configuration.                |
| Is Masked          | 'TRUE'/'FALSE' | Optional* | Import will set this configuration.                |
| ID Method          | 'Auto'/'User'  | Optional* | Import will set this configuration.                |
| Row Type           | 'All Row'      | Optional* | Import will set this configuration.                |
| Date Format        | As in UI       | Optional* | Import will set this configuration.                |
+--------------------+----------------+-----------+----------------------------------------------------+

* Optional - these rows are only needed for columns where the configuration needs to be changed. All other rows can be excluded. This reduces import and processing time on large Rule Sets. 

Example

Environment Name,Rule Set,Table Name,Type,Parent Column Name,Column Name,Data Type,Domain,Algorithm,Is Masked,ID Method,Row Type,Date Format
Environment,RuleSet,Table,-,-,Column,NVARCHAR2 (50),LAST_NAME,LAST NAME SL,TRUE,Auto,All Row,-

Troubleshooting & Resolution

Pop-up must be enabled  

The import process requires pop-ups to be enabled. When importing a CSV file, a notification will be shown in the browser if pop-ups are disabled. 

Resolution  

Navigate to the browser settings and enable pop-ups for the masking engine page. 

Referenced object not available on the Masking Engine 

If a referenced object (such as Environment, Rule Set, Table, Column, Domain, Algorithm) is not available on the Masking Engine a warning is shown in the pop-up. 

Note

Note:

  1. Correct spelling and case are required to reference objects.
  2. The reference to algorithms is the Algorithm Name as shown in the UI.

Examples of these warnings can be seen below:

Import Inventory CSV Warnings - example.png

Masking Logs 

Errors in the import process are logged in the masking log (some object warnings are currently not logged). Below are some examples related to the warning shown in the pop-up above.

Reading each line of the csv imported Inventory: Environment,RS,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Environment Environment does not exist

Reading each line of the csv imported Inventory: Env,RuleSet,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Rule Set RuleSet does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Column,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Env Env, Rule Set RS_BUG : Column Column does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Col,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,XXXX@MM!dd
Input date format:XXXX@MM!dd Valid:false

Resolution 

Depending on what object is throwing the warning, please check the object on the Masking Engine UI and correct the reference to the object, the spelling, the case, or for the date the format. 

To get an example of the CSV file, please Export a file from the UI.

Incorrect number of columns in the CSV file 

If the CSV file is missing a comma or column(s), the following error will be thrown in the UI "Error occurred during importing inventory. Please refer application logs.", as shown below.
Import Inventory CSV Error.png

Checking the masking logs provides more information:

Proceeding to reading and checking that csv has proper formatting
...
Reading each line of the csv imported Inventory: Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-
Exception :
java.lang.RuntimeException: Unexpected number of fields on line 19 of imported Inventory: expected 13, found 9
...

Resolution 

Check the masking logs, the row and the error. The error shows how many columns are expected and the failed row is shown above. 

From the example above:

Error: 
  * Unexpected number of fields
  * on line 19 
  * expected 13, found 9
  
Row:
  * Is      : "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-"
  * Expected: "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-,true,Auto,All Row,-"
    

No (invalid) header fields (Invalid File Format) 

If the header is missing or has incorrect fields, the following error is shown: "Invalid File Format". The pop-up will look like this:
Import Inventory CSV Invalid File Format.png

The logs will not show any imported rows (and no error message):

User has requested to import a csv file to Inventory
csv file upload successful to path: /var/delphix/dmsuite/import/delphix_admin/ImportInventory
...
If there is no error and inventory does not get updated, make sure pop-up is enabled.

Resolution  

The best way to resolve this issue is the export a Rule Set (Inventory) and copy the header from this file and paste it into the failed import file.

Invalid characters in the CSV file 

The characters need to match the objects in the Rule Set and algorithm names. Invalid characters in the CSV file can and is likely to cause undesired behaviors which are not easy to detect.

Example of issues:

  • Trailing and leading spaces.
  • Incorrect end-of-line character(s).
  • Invalid character in the row with data.

Resolution 

The best resolution here is to be diligent in editing the CSV file. Open it in a text editor, save it to a standard ANSII character set and check for invalid characters.