1 / 01 JUNE 2008 / EDS

Not So Risky Business With DI StudioAngelena JasnyEDS Australia

2 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

My first day with DI Studio…

3 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Overview

• SAS Banking Detailed Data Store (DDS) & Credit Risk Management System (CRMS)

• SAS applications used by our system

• Notes about metadata

• Environment structure

• DI Studio and the Extract, Transform and Load layers

• Deployment and Execution

Why are we here?

4 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Not so Risky Business

• Allowed the business to more accurately calculate and understand their risk

•Better understanding of their data» Term Deposits

• Removed a number of variables from the development process

•Time savings with generated code

•Focus on functionality

•Potential cost savings

The SAS Banking DDS and CRMS

6 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

The SAS Banking DDS

DDS (Detailed Data Store)• a database

• keeps history

• SAS business solutions draw data from it

CRMS is the Credit Risk Management solution provided by SAS• Calculates the amount of capital to set aside to cover risk

7 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

The Applications

8 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Applications Used

A complete system requires a number of Applications

• Data Integration Studio 2.4 (DIS)

• SAS Management Console (SMC)

• Process Flow Manager 3.0

• SAS 9 for Windows (SAS 9.1.3)

9 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

The Big Four

10 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

User End Applications

11 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Notes on Metadata

12 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

The Meta-Physical

DI Studio & SMC are interfaces to Metadata• Different Roles – some overlap

“Metadata Repository”= A bunch of SAS tables

SAS 9 for Windows is the interface to the Physical data• Browse, query, create and modify

13 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Environments & Structure

14 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Environments

Environments to support • Development

• Testing

• Production

SAS recommended naming convention• Lev3

• Lev2

• Lev1

Maintenance Phase• three environments not enough

15 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

File Structure

Quiz…• Metadata Server

• Object Spawner (listener)

• Connect Server

• Workspace Server

• Share Server

• Stored Process Server

• OLAP Server

• Scheduling Server

• Batch Server

SASCode• Deployed

• CRMSDeployed

SASFormats

SASMacro

16 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Code Versioning & Change Management

Third party tool for change management• Each environment has own Change Mgmt repository

• Subversion, WinMerge

DI Studio mechanisms• Job Notes

• Extended Attributes

• The Project Repository – personal space

• Check-out / Check-in – locking only

17 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

The Application Layers

18 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

ETL - Extract, Transform and Load

Extract Layer• Turns source system data into a SAS usable form

Transform Layer• Contains the business logic

• Requires understanding of the data and how it relates

Load Layer• Loads the data store

• Can be data the client will want to see, for example, via E-Guide

19 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Extract Layer

20 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Extract Job

Source File objects• User written

–Code control

–Create from code

Table Loader• Replace the target table

Validation & Formatting• Leave it – many fields

21 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

New Objects

22 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Tables

Table from metadata• View data on missing table

Metadata from table• Run the table creation code (DI Studio Program Editor or SAS DM)• Ensure table in a library accessible in metadata• Use DI Studio or SMC to register the table in metadata

23 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Example - Import the table in SMC

24 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Transform Layer

25 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Transform Job Example

26 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Extract Transformations

Tips

…only one…

USE IT ALWAYS!!!

• Keep only necessary fields

• Naming fields

• Extract distinct values and filter

• Expressions, Formatting

• FLEXIBILITY – Design for Change!

27 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Lookup Transformation

Multiple inputs• Avoids overly complex SQL joins and long chains to visit multiple

inputs

Error and exception tables• Multiple lookups - use exception table

• Specify fields that help investigation

Tips• Name temporary input tables

• Field type & length the same

• Must pass a field through

• Cannot use the result of Lookup as lookup key in another lookup in the same transform

• CAN use the result of a lookup in expressions

28 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Other transformsSQL Joins• Name temporary input tables

• Expressions – prefix field names with table name

Eg W5OFI07.AccountNumber

• Keep it simple

Data Validation• Error tables not in metadata

• Lookup tables hidden

Loaders• Localisation of Expressions

• Update/Insert – use indexes– 1 minute vs 6 hours

Splitter• Can use two Extracts instead

Append, sort etc

29 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Load Layer

30 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

DDS Load Job

Business Data to Detailed Data Store• Translate codes into values usable by solutions

Maintains history of the data• SCD Type 2 Loader (Slowly Changing Dimensions)

31 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Deployment and execution

32 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Deploying Code & Creating Job Flows

Putting it all together• Deploy

– Replacement vs Update of jobs

• Add job to flow in SMC

• Manage Dependencies– Triggers

• Schedule or Reschedule the Flow

• Execute the Flow via Process Flow Manager

Recording of Dependencies• Don’t just rely on Process Flow Manager

33 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Job flows & Process Flow Manager

34 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Conclusion

35 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Just the beginning

DDS & SAS Solutions – A Platform for Intelligent Business

Metadata – Nature and Purpose

Organisation - Manage Environments & Code

When writing ETL and using DI Studio, Design for Change

36 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI

Question Time

37 / 01 JUNE 2008 / EDS

EDS Australia229 Elizabeth StreetBrisbane, QLD 4000angelena.jasny@eds.com or eds.com

EDS and the EDS logo are registered trademarks of Electronic Data Systems Corporation. EDS is an equal opportunity employer and values the diversity of its people. © 2007 Electronic Data Systems Corporation. All rights reserved.