Skip to main content
Delphix

Inventory Import and Export (KBA1815)

 

KBA

KBA#1815

 

This document describes the Inventory Export and Import process.  Masked columns can be changed in multiple environments, Rule Sets, and Tables using the CSV import process. 

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases

5.3

5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

4.3

4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0

4.2

4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1

4.1

4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

At a Glance 

Description:  Export and Import of CSV file in order to review and quickly modify Inventory configurations.
Location: UI: Environments - [Environment] - Inventory 
Exported:
Selected Environment Name, Rule Set.
All All Rule Set properties for all other fields. 
Imported:
Required Environment Name, Rule Set, Table Name, Column Name.
Optional For column properties where these have changed: 
Domain, Algorithm, Is Masked, ID Method, Row Type, Date Format.
Character 
Encoding:
For multi-byte encoding, the CSV file needs to be imported using UTF-8 (no BOM).

 

Note

Note:

Only the modified data needs to be imported. For example: 

  • If a column is to be masked. 
  • If a masked column is not to be masked anymore.
  • If the masked column configuration has changed.

 

User Interface: Export, Import, and Pop-up 

Export Inventory 

To export the Inventory of a Rule Set:

  1. Navigate to Environments > Inventory
  2. Select the Rule Set to export which in this example is 'RuleSet'.
  3. Click Export.
    This opens the Export Inventory dialog.
  4. If desired, change the File Name.
  5. Click Save.
    This will open a pop-up indicating that the CSV file is being generated.
  6. After the file has been created, the popup will change. This may take some time, depending on the size of the inventory.
  7. Click the Download File link to download.

UI Inventory Page and Export Pop-up .png

Import Inventory 

To import changes to the Inventory:

  1. Navigate to Environments > Inventory
  2. Click Import.
    This opens the Import Inventory dialog.
  3. Click Select and browse to the file for import.
  4. Click Save.
    It takes a few minutes to display the pop-up which reports import status. The size of the inventory uploaded impacts the time you need to wait. It can be significant.

UI Inventory Import w popup .png

Pop-up 

The stages and the final status message are displayed in a browser pop-up (as in the two examples above).

Note

Note:

 Pop-ups need to be enabled on the browser for the engine URL.

See Troubleshooting and Resolution for examples of warnings and errors. 

Data Format 

The file format for the exported and imported CSV file is detailed below. 

CSV File - Export Description

Only the selected Rule Set will be exported. 

+--------------------+----------------+----------+-------------------------------------------------------------------------+
| CSV Field          | Value          | Export   | Export comment                                                          |
+--------------------+----------------+----------+-------------------------------------------------------------------------+
| Environment Name   | As in UI       | Selected | Only the selected Environment as per Rule Set.                          |
| Rule Set           | As in UI       | Selected | Only the selected Rule Set.                                             |
+--------------------+----------------+----------+-------------------------------------------------------------------------+
| Table Name         | As in UI       | All      | All tables in the Rule Set.                                             |
| Type               | -              | -        | Normally not used - default '-'.                                        |
| Parent Column Name | -              | -        | Normally not used - default '-'.                                        |
| Column Name        | As in UI       | All      | All columns in the Rule Set.                                            |
| Data Type          | As in UI       | All      | The Data Type for specified column.                                     |
| Domain             | As in UI       | All      | The assigned Domain for specified column.                               |
| Algorithm          | As in UI       | All      | The assigned algorithm for specified column.                            |
| Is Masked          | 'TRUE'/'FALSE' | All      | Set to TRUE if specified column is masked, otherwise FALSE.             |
| ID Method          | 'Auto'/'User'  | All      | Set to User if Domain/Algorithm is locked. Auto if profiler can change. |
| Row Type           | 'All Row'      | All      | Normally not used - default 'All Row'.                                  |
| Date Format        | As in UI       | All      | The assigned Date Format for specified column.                          |
+--------------------+----------------+----------+-------------------------------------------------------------------------+

CSV File - Import Description

+--------------------+----------------+-----------+----------------------------------------------------+
| CSV Field          | Value          | Import    | Import comment                                     |
+--------------------+----------------+-----------+----------------------------------------------------+
| Environment Name   | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
| Rule Set           | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
| Table Name         | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
| Type               | -              | -         | Normally not used.                                 |
| Parent Column Name | -              | -         | Normally not used.                                 |
| Column Name        | As in UI       | Required  | Need to match existing. Multiple entries accepted. |
+--------------------+----------------+-----------+----------------------------------------------------+
| Data Type          | -              | -         | Not used.                                          |
+--------------------+----------------+-----------+----------------------------------------------------+
| Domain             | As in UI       | Optional* | Import will set this configuration.                |
| Algorithm          | As in UI       | Optional* | Import will set this configuration.                |
| Is Masked          | 'TRUE'/'FALSE' | Optional* | Import will set this configuration.                |
| ID Method          | 'Auto'/'User'  | Optional* | Import will set this configuration.                |
| Row Type           | 'All Row'      | Optional* | Import will set this configuration.                |
| Date Format        | As in UI       | Optional* | Import will set this configuration.                |
+--------------------+----------------+-----------+----------------------------------------------------+

* Optional - these rows are only needed for columns where the configuration needs to be changed. All other rows can be excluded. This reduces import and processing time on large Rule Sets. 

Example

Environment Name,Rule Set,Table Name,Type,Parent Column Name,Column Name,Data Type,Domain,Algorithm,Is Masked,ID Method,Row Type,Date Format
Environment,RuleSet,Table,-,-,Column,NVARCHAR2 (50),LAST_NAME,LAST NAME SL,TRUE,Auto,All Row,-

Troubleshooting & Resolution

Performance and reliability

The process of exporting to or importing from the CSV file can take significant time depending on the size of the inventory involved.  Later versions of the product have improvements to make the process as efficient as possible, but it can still take many minutes to read and store millions of columns worth of inventory.  In both export and import scenarios, you have to wait for the pop-up which shows the process to be complete.  If you logout or idle out before the process has completed, it will be terminated and will not complete.  You should always receive a popup which shows the process succeeded or the process failed.  If there is no popup, the process is still running or the browser logged out / idled out and the process failed by default.  We are continuing to improve this process to make it more usable and future versions should continue to improve on this functionality.

Pop-up must be enabled  

The import process requires pop-ups to be enabled. When importing a CSV file, a notification will be shown in the browser if pop-ups are disabled. 

Resolution  

Navigate to the browser settings and enable pop-ups for the masking engine page. 

Referenced object not available on the Masking Engine 

If a referenced object (such as Environment, Rule Set, Table, Column, Domain, Algorithm) is not available on the Masking Engine a warning is shown in the pop-up. 

Note

Note:

  1. Correct spelling and case are required to reference objects.
  2. The reference to algorithms is the Algorithm Name as shown in the UI.

Examples of these warnings can be seen below:

Import Inventory CSV Warnings - example.png

Masking Logs 

Errors in the import process are logged in the masking log (some object warnings are currently not logged). Below are some examples related to the warning shown in the pop-up above.

Reading each line of the csv imported Inventory: Environment,RS,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Environment Environment does not exist

Reading each line of the csv imported Inventory: Env,RuleSet,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Rule Set RuleSet does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Column,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,-
Env Env, Rule Set RS_BUG : Column Column does not exist

Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Col,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,XXXX@MM!dd
Input date format:XXXX@MM!dd Valid:false

Resolution 

Depending on what object is throwing the warning, please check the object on the Masking Engine UI and correct the reference to the object, the spelling, the case, or for the date the format. 

To get an example of the CSV file, please Export a file from the UI.

Incorrect number of columns in the CSV file 

If the CSV file is missing a comma or column(s), the following error will be thrown in the UI "Error occurred during importing inventory. Please refer application logs.", as shown below.
Import Inventory CSV Error.png

Checking the masking logs provides more information:

Proceeding to reading and checking that csv has proper formatting
...
Reading each line of the csv imported Inventory: Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-
Exception :
java.lang.RuntimeException: Unexpected number of fields on line 19 of imported Inventory: expected 13, found 9
...

Resolution 

Check the masking logs, the row and the error. The error shows how many columns are expected and the failed row is shown above. 

From the example above:

Error: 
  * Unexpected number of fields
  * on line 19 
  * expected 13, found 9
  
Row:
  * Is      : "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-"
  * Expected: "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-,true,Auto,All Row,-"
    

No (invalid) header fields (Invalid File Format) 

If the header is missing or has incorrect fields, the following error is shown: "Invalid File Format". The pop-up will look like this:
Import Inventory CSV Invalid File Format.png

The logs will not show any imported rows (and no error message):

User has requested to import a csv file to Inventory
csv file upload successful to path: /var/delphix/dmsuite/import/delphix_admin/ImportInventory
...
If there is no error and inventory does not get updated, make sure pop-up is enabled.

Resolution  

The best way to resolve this issue is the export a Rule Set (Inventory) and copy the header from this file and paste it into the failed import file.

Invalid characters in the CSV file 

The characters need to match the objects in the Rule Set and algorithm names. Invalid characters in the CSV file can and is likely to cause undesired behaviors which are not easy to detect.

Example of issues:

  • Trailing and leading spaces.
  • Incorrect end-of-line character(s).
  • Invalid character in the row with data.
  • A column name that includes a comma (any column names that include a comma will need to have their algorithm set manually in the GUI)

Resolution 

The best resolution here is to be diligent in editing the CSV file. Open it in a text editor, save it to a standard ANSII character set and check for invalid characters.