Inventory Import and Export (KBA1815)
KBA
KBA#1815
This document describes the Inventory Export and Import process. Masked columns can be changed in multiple environments, Rule Sets, and Tables using the CSV import process.
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 5.3
5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0 5.2
5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1
5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0
5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
4.3
4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0
4.2
4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1
4.1
4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0
At a Glance
| Description: | Export and Import of CSV file in order to review and quickly modify Inventory configurations. | ||||
|---|---|---|---|---|---|
| Location: | UI: Environments - [Environment] - Inventory | ||||
| Exported: |
|
||||
| Imported: |
|
||||
| Updates: | The CSV Format was updated in version 6.0.9.0. Three fields added ('Require to Apply Change).
|
User Interface: Export, Import, and Pop-up
Export Inventory
To export the Inventory of a Rule Set:
- Navigate to Environments > Inventory.
- Select the Rule Set to export which in this example is 'RuleSet'.
- Click Export.
This opens the Export Inventory dialog. - If desired, change the File Name.
- Click Save.
This will open a pop-up indicating that the CSV file is being generated. - After the file has been created, the popup will change. This may take some time, depending on the size of the inventory.
- Click the Download File link to download.
Import Inventory
To import changes to the Inventory:
- Navigate to Environments > Inventory.
- Click Import.
This opens the Import Inventory dialog. - Click Select and browse to the file for import.
- Click Save.
It takes a few minutes to display the pop-up which reports import status. The size of the inventory uploaded impacts the time you need to wait. It can be significant.
Pop-up
The stages and the final status message are displayed in a browser pop-up (as in the two examples above).
See Troubleshooting and Resolution for examples of warnings and errors.
Data Format
The file format for the exported and imported CSV file is detailed below.
CSV File - Export Description
Only the selected Rule Set will be exported.
+-----------------------------+----------------+----------+----------------------------------------------------------------+ | CSV Field | Value | Export | Export comment | +-----------------------------+----------------+----------+----------------------------------------------------------------+ | Environment Name | As in UI | Selected | Only the selected Environment as per Rule Set. | | Rule Set | As in UI | Selected | Only the selected Rule Set. | +-----------------------------+----------------+----------+----------------------------------------------------------------+ | Table Name | As in UI | All | All tables in the Rule Set. | | Type | - | - | Normally not used - default '-'. | | Parent Column Name | - | - | Normally not used - default '-'. | | Column Name | As in UI | All | All columns in the Rule Set. | | Data Type | As in UI | All | The Data Type for specified column. | | Domain | As in UI | All | The assigned Domain for specified column. | | Algorithm | As in UI | All | The assigned algorithm for specified column. | | Is Masked | 'TRUE'/'FALSE' | All | TRUE = specified column is masked. FALSE = Not Masked. | | ID Method | 'Auto'/'User' | All | User = Domain/Algorithm is locked. Auto = profiler can change. | | Row Type | 'All Row' | All | Normally not used - default 'All Row'. | | Date Format | As in UI | All | The assigned Date Format for specified column. | | Notes* | As in UI | All | To record notes. | | Multi-Column Logical Field* | As in UI | All | To record notes. | | Group Number* | As in UI | All | To record notes. | +-----------------------------+----------------+----------+----------------------------------------------------------------+
* These columns were added in version 6.0.9.0.
CSV File - Import Description
+-----------------------------+----------------+----------+----------------------------------------------------+ | CSV Field | Value | Import | Import comment | +-----------------------------+----------------+----------+----------------------------------------------------+ | Environment Name | As in UI | Required | Need to match existing. Multiple entries accepted. | | Rule Set | As in UI | Required | Need to match existing. Multiple entries accepted. | | Table Name | As in UI | Required | Need to match existing. Multiple entries accepted. | | Type | - | - | Normally not used. | | Parent Column Name | - | - | Normally not used. | | Column Name | As in UI | Required | Need to match existing. Multiple entries accepted. | +-----------------------------+----------------+----------+----------------------------------------------------+ | Data Type | - | - | No effect. | +-----------------------------+----------------+----------+----------------------------------------------------+ | Domain | As in UI | Optional | Import will set this configuration. | | Algorithm | As in UI | Optional | Import will set this configuration. | | Is Masked | 'TRUE'/'FALSE' | Optional | Import will set this configuration. | | ID Method | 'Auto'/'User' | Optional | Import will set this configuration. | | Row Type | 'All Row' | Optional | Import will set this configuration. | | Date Format | As in UI | Optional | Import will set this configuration. | | Notes* | As in UI | Optional | Import will set this configuration. | | Multi-Column Logical Field* | As in UI | Optional | Import will set this configuration. | | Group Number* | As in UI | Optional | Import will set this configuration. | +-----------------------------+----------------+----------+----------------------------------------------------+
"Optional" - these rows are only needed for columns where the configuration needs to be changed. All other rows can be excluded. This reduces import and processing time on large Rule Sets.
* These columns were added in version 6.0.9.0.
Example
Environment Name,Rule Set,Table Name,Type,Parent Column Name,Column Name,Data Type,Domain,Algorithm,Is Masked,ID Method,Row Type,Date Format MyEnv,MyRS,MyTable,-,-,MyColumn,NVARCHAR2 (50),LAST_NAME,LAST NAME SL,TRUE,Auto,All Row,-
Troubleshooting & Resolution
Performance and reliability
The process of exporting to or importing from the CSV file can take significant time depending on the size of the inventory involved. Later versions of the product have improvements to make the process as efficient as possible, but it can still take many minutes to read and store millions of columns worth of inventory. In both export and import scenarios, you have to wait for the pop-up which shows the process to be complete. If you log out or the session expires before the process has completed, it will be terminated and will not complete. You should always receive a popup that shows the process succeeded or the process failed. If there is no popup, the process is still running or the browser logged out / idled out and the process failed by default. We are continuing to improve this process to make it more usable and future versions should continue to improve on this functionality.
Pop-up must be enabled
The import process requires pop-ups to be enabled. When importing a CSV file, a notification will be shown in the browser if pop-ups are disabled.
Resolution
Navigate to the browser settings and enable pop-ups for the masking engine page.
Referenced object not available on the Masking Engine
If a referenced object (such as Environment, Rule Set, Table, Column, Domain, Algorithm) is not available on the Masking Engine a warning is shown in the pop-up.
Examples of these warnings can be seen below:
Masking Logs
Errors in the import process are logged in the masking log (some object warnings are currently not logged). Below are some examples related to the warning shown in the pop-up above.
Reading each line of the csv imported Inventory: Environment,RS,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,- Environment Environment does not exist Reading each line of the csv imported Inventory: Env,RuleSet,table_1,-,-,Col_1,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,- Rule Set RuleSet does not exist Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Column,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,- Env Env, Rule Set RS_BUG : Column Column does not exist Reading each line of the csv imported Inventory: Env,RS,table_1,-,-,Col,varchar (20),FIRST_NAME,LAST NAME SL,true,Auto,All Row,XXXX@MM!dd Input date format:XXXX@MM!dd Valid:false
Resolution
Depending on what object is throwing the warning, please check the object on the Masking Engine UI and correct the reference to the object, the spelling, the case, or for the date the format.
To get an example of the CSV file, please Export a file from the UI.
Incorrect number of columns in the CSV file
If the CSV file is missing a comma or column(s), the following error will be thrown in the UI "Error occurred during importing inventory. Please refer application logs.", as shown below.
Checking the masking logs provides more information:
Proceeding to reading and checking that csv has proper formatting ... Reading each line of the csv imported Inventory: Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,- Exception : java.lang.RuntimeException: Unexpected number of fields on line 19 of imported Inventory: expected 13, found 9 ...
Resolution
Check the masking logs, the row and the error. The error shows how many columns are expected and the failed row is shown above.
From the example above:
Error:
* Unexpected number of fields
* on line 19
* expected 13, found 9
Row:
* Is : "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-"
* Expected: "Environment,RuleSet,Table,-,-,Column,NUMBER (0),-,-,true,Auto,All Row,-"
No (invalid) header fields (Invalid File Format)
If the header is missing or has incorrect fields, the following error is shown: "Invalid File Format". The pop-up will look like this:
The logs will not show any imported rows (and no error message):
User has requested to import a csv file to Inventory csv file upload successful to path: /var/delphix/dmsuite/import/delphix_admin/ImportInventory ... If there is no error and inventory does not get updated, make sure pop-up is enabled.
Resolution
The best way to resolve this issue is the export a Rule Set (Inventory) and copy the header from this file and paste it into the failed import file.
Invalid characters in the CSV file
The characters need to match the objects in the Rule Set and algorithm names. Invalid characters in the CSV file can and is likely to cause undesired behaviors which are not easy to detect.
Example of issues:
- Trailing and leading spaces.
- Incorrect end-of-line character(s).
- Invalid character in the row with data.
- A column name that includes a comma (any column names that include a comma will need to have their algorithm set manually in the GUI)
Resolution
The best resolution here is to be diligent in editing the CSV file. Open it in a text editor, save it to a standard ANSII character set and check for invalid characters.