This document is provided for information purposes only. This document is subject to the information classification set out on each page. If no information classification has been included, this document must be treated as UNCLASSIFIED, SENSITIVE and must not be disclosed other than with the consent of nbn co. The recipient (including third parties) must make and rely on their own inquiries as to the currency, accuracy and completeness of the information contained herein and must not use this document other than with the consent of nbn co. © 2015 nbn co ltd. ‘nbn’, ‘bring it on’, and the Aurora device are trademarks of nbn co ltd | ABN 86 136 533 741

IBM InfoSphere DataStage DemoPhuoc Kha & Chintan ParekhDocument ID // Rev 0.1 // 15/04/2016

UNCLASSIFIED

2© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Agenda

1. Logging DataStage Director (10 mins) Operations Console (10 mins) Reporting Requirements (10 mins)

2. Processing Time (LocOps) PL/SQL LocOps Operation (10 min) Datastage Execution Analysis (10 min) Recommendation (10 min)

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

3© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Prototype DataStage Job

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

4© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

DataStage Director

• Logging of Job flow, success and Failure is done inside DataStage Director logs. These logs are available to application support to access the success and failure of the physical job run, and provide Error, Warning and informational messages about the jobs execution

• Requires IBM InfoSphere client to be installed

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

5© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

DataStage Director

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

6© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

DataStage Director

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

7© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

DataStage Director

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

8© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Operations Console

• Provides engine-wide information about job runs, system resources, workload management queues, and engine status

• Browser-base• Need to ensure EngMonApp process is running

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

9© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Operations Console

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

10© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Operations Console

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

11© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Operations Console

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

12© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Prototype DataStage Job: Write Reject Log File

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

13© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Rejects output to file

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

14© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Reject Log File

Document ID // Rev 0.1 // 15/04/2016 UNCLASSIFIED

15© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Job/Process Log vs Telstra Data Reports

• Why you need process log?• Why you need 7 Telstra Data Reports?

1. Polygon Wash Progressive Report2. Merge Progressive Report3. Street PID Lookup Progressive Report4. Pologon Wash Detail Output Report5. Merge Detail Output Report6. Street PID Lookup Detail Output Report7. Street Matching Data Report

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

16© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Current Tasks of LocOps (addAddressAlias)

1. Determine the location type (e.g. base address or subaddress)

2. Find existing location address by using FindAddress function

3. Create base address and base address location

4. Adding audit information

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

17© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Current Tasks of LocOps (editAddressAlias)

1. Determine the location type (e.g. base address or subaddress)

2. Find existing location address by using FindAddress function

3. End Date existing address

4. Create new alias with new details.

5. Add Audit Information

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

18© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Execution Analysis

Based on our Analysis of LocOps and Datastage jobs, we found following key behaviour

1.LocOps addAddress alias took 36 hours to complete 25,000 records

2.When we disable the findAddress function call and compared DataStage jobs with LocOps below are the timings of execution

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

Task LocOps Time Datastage Time

Insert 3.5 hrs/100,000 25mins/100,000Update 13 hrs/100,000 40mins/100,000

19© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Datastage Log

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

20© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Datastage Log

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

21© 2015 nbn co ltd. ‘nbn’ is a trade mark of nbn co ltd

Recommendations

Document ID // Rev 0.1 // 10/02/2016 UNCLASSIFIED

Step LocOps DataStage1. Determine Loc Type Disable Create reusable module 2. Find Existing Address Disable Create reusable module 3. Insert/Update/Retire Use LocOps Call LocOps