Using Curl to Collect Masking Job Reports (KBA1400)
Issue
Audit procedures within some customer environments require collecting detailed reports about masking jobs. In environments with excessive (>100000) numbers of tables such as SAP/PeopleSoft collecting these reports is simply not feasible.
Example of Monitor Page in the Masking UI
Resolution
The solution proposed in this document is to use curl from the command line. Since the result of the curl query exposes data in json format, this method is in many situations more convenient than the standard PDF report, because the json formatted data can be queried and parsed.
Commands to collect the Report
To collect the report use the three curl statements below. Please ensure to:
- Change YOURSERVER to your masking engine
- Change PASSWORD to the delphix_admin (or other user - if so please also change user)
- Change the JOBNM to the Job Number of the job you want the report from
curl can be run on any system where curl binaries are installed. The sole requirement is that the client system has HTTP access to the engine.
# 1. Initiate the cookie jar
curl --cookie-jar cjar --output /dev/null http://YOURSERVER:8282/dmsuite/login.do
# 2. Login
curl --cookie cjar --cookie-jar cjar --data 'userName=delphix_admin' --data 'password=PASSWORD' --output /dev/null http://YOURSERVER:8282/dmsuite/login.do
# 3. Get the data - please change the jobNm to the job data you want to collect the info from
curl --cookie cjar http://YOURSERVER:8282/dmsuite/monitorJobsDBCompleted.do?jobNm=JOBNM | grep "{ID"
Example
Example of an Execution
To help highlight what to expect from the data, the example chosen below has been collected from a job run against a minimal environment.
$ # 1. Initiate the cookie jar
$ curl --cookie-jar cjar --output /dev/null http://ak50build.dcenter.delphix.com:8282/dmsuite/login.do
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4349 100 4349 0 0 5550 0 --:--:-- --:--:-- --:--:-- 5554
$ # 2. Login - please change userName and password accordingly
$ curl --cookie cjar --cookie-jar cjar --data 'userName=delphix_admin' --data 'password=Delphix_123' --output /dev/null http://ak50build.dcenter.delphix.com:8282/dmsuite/login.do
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 43 0 0 100 43 0 108 --:--:-- --:--:-- --:--:-- 108
$ # 3. Get the data - please change the jobNm to the job data you want to collect the info from
$ curl --cookie cjar http://ak50build.dcenter.delphix.com:8282/dmsuite/monitorJobsDBCompleted.do?jobNm=25 | grep "{ID"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 20075 0 20075 0 0 27488 0 --:--:-- --:--:-- --:--:-- 27500
{ID :'0',Name :'DEPT',DisplayName :'DEPT',NameLink :'',Progress :'100',StatusIcon :'status-icon-success.png',Status :'Success',Time :'0d 0h 0m',RowsPerMin :'240',TransformationCode :'25_113_DEPT.txt',TransformationReport :'2016/07/01 02:08:29 - DEPT - Loading transformation from XML file [/var/delphix/dmsuite//output/Env/DMSApplicator/RuleSet/25/KETTLE_MASK_XML_25_DEPT_113.xml]<br/>2016/07/01 02:08:30 - DEPT - Dispatching started for transformation [KETTLE_MASK_XML_25_DEPT_113]<br/>2016/07/01 02:08:30 - Table input.0 - Finished reading query, closing connection.<br/>2016/07/01 02:08:30 - Table input.0 - Finished processing (I=4, O=0, R=0, W=4, U=0, E=0)<br/>2016/07/01 02:08:30 - Select values.0 - Finished processing (I=0, O=0, R=4, W=4, U=0, E=0)<br/>2016/07/01 02:08:31 - Get All Lookups Values.0 - Finished processing (I=0, O=0, R=4, W=4, U=0, E=0)<br/>2016/07/01 02:08:31 - Col0_dataFromXML.0 - Finished processing (I=5002, O=0, R=0, W=5002, U=0, E=0)<br/>2016/07/01 02:08:31 - Col0_decryptData.0 - Finished processing (I=0, O=0, R=5002, W=5002, U=0, E=0)<br/>2016/07/01 02:08:31 - Col0_lookUp.0 - Finished processing (I=0, O=0, R=5006, W=4, U=0, E=0)<br/>2016/07/01 02:08:31 - SelectValues_MetaData.0 - Finished processing (I=0, O=0, R=4, W=4, U=0, E=0)<br/>2016/07/01 02:08:31 - String Cut.0 - Finished processing (I=0, O=0, R=4, W=4, U=0, E=0)<br/>2016/07/01 02:08:31 - Update.0 - Finished processing (I=4, O=0, R=4, W=4, U=4, E=0)<br/>',ReportHeading :'Success Report',RowsMasked :'4',RowsRemaining :'0'}
The result can be piped to a file and used as required.
Example of Collected Data
The salient key-value pairs in this particular (and small) masking job are:
| json | Key | Value |
|---|---|---|
|
ID | 0 |
|
Name | DEPT |
|
Progress | 100 |
|
Status | Success |
|
Time | 0d 0h 0m |
|
RowsPerMin | 240 |
|
ReportHeading | Success Report |
| RowsMasked :'4' | RowsMasked | 4 |
| RowsRemaining :'0' | RowsRemaining | 0 |