3 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Overview
• SAS Banking Detailed Data Store (DDS) & Credit Risk Management System (CRMS)
• SAS applications used by our system
• Notes about metadata
• Environment structure
• DI Studio and the Extract, Transform and Load layers
• Deployment and Execution
Why are we here?
4 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Not so Risky Business
• Allowed the business to more accurately calculate and understand their risk
•Better understanding of their data» Term Deposits
• Removed a number of variables from the development process
•Time savings with generated code
•Focus on functionality
•Potential cost savings
6 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
The SAS Banking DDS
DDS (Detailed Data Store)• a database
• keeps history
• SAS business solutions draw data from it
CRMS is the Credit Risk Management solution provided by SAS• Calculates the amount of capital to set aside to cover risk
8 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Applications Used
A complete system requires a number of Applications
• Data Integration Studio 2.4 (DIS)
• SAS Management Console (SMC)
• Process Flow Manager 3.0
• SAS 9 for Windows (SAS 9.1.3)
12 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
The Meta-Physical
DI Studio & SMC are interfaces to Metadata• Different Roles – some overlap
“Metadata Repository”= A bunch of SAS tables
SAS 9 for Windows is the interface to the Physical data• Browse, query, create and modify
14 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Environments
Environments to support • Development
• Testing
• Production
SAS recommended naming convention• Lev3
• Lev2
• Lev1
Maintenance Phase• three environments not enough
15 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
File Structure
Quiz…• Metadata Server
• Object Spawner (listener)
• Connect Server
• Workspace Server
• Share Server
• Stored Process Server
• OLAP Server
• Scheduling Server
• Batch Server
SASCode• Deployed
• CRMSDeployed
SASFormats
SASMacro
16 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Code Versioning & Change Management
Third party tool for change management• Each environment has own Change Mgmt repository
• Subversion, WinMerge
DI Studio mechanisms• Job Notes
• Extended Attributes
• The Project Repository – personal space
• Check-out / Check-in – locking only
18 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
ETL - Extract, Transform and Load
Extract Layer• Turns source system data into a SAS usable form
Transform Layer• Contains the business logic
• Requires understanding of the data and how it relates
Load Layer• Loads the data store
• Can be data the client will want to see, for example, via E-Guide
20 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Extract Job
Source File objects• User written
–Code control
–Create from code
Table Loader• Replace the target table
Validation & Formatting• Leave it – many fields
22 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Tables
Table from metadata• View data on missing table
Metadata from table• Run the table creation code (DI Studio Program Editor or SAS DM)• Ensure table in a library accessible in metadata• Use DI Studio or SMC to register the table in metadata
26 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Extract Transformations
Tips
…only one…
USE IT ALWAYS!!!
• Keep only necessary fields
• Naming fields
• Extract distinct values and filter
• Expressions, Formatting
• FLEXIBILITY – Design for Change!
27 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Lookup Transformation
Multiple inputs• Avoids overly complex SQL joins and long chains to visit multiple
inputs
Error and exception tables• Multiple lookups - use exception table
• Specify fields that help investigation
Tips• Name temporary input tables
• Field type & length the same
• Must pass a field through
• Cannot use the result of Lookup as lookup key in another lookup in the same transform
• CAN use the result of a lookup in expressions
28 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Other transformsSQL Joins• Name temporary input tables
• Expressions – prefix field names with table name
Eg W5OFI07.AccountNumber
• Keep it simple
Data Validation• Error tables not in metadata
• Lookup tables hidden
Loaders• Localisation of Expressions
• Update/Insert – use indexes– 1 minute vs 6 hours
Splitter• Can use two Extracts instead
Append, sort etc
30 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
DDS Load Job
Business Data to Detailed Data Store• Translate codes into values usable by solutions
Maintains history of the data• SCD Type 2 Loader (Slowly Changing Dimensions)
32 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Deploying Code & Creating Job Flows
Putting it all together• Deploy
– Replacement vs Update of jobs
• Add job to flow in SMC
• Manage Dependencies– Triggers
• Schedule or Reschedule the Flow
• Execute the Flow via Process Flow Manager
Recording of Dependencies• Don’t just rely on Process Flow Manager
35 / 01 JUNE 2008 / EDSNot So Risky Business – Doing IT with DI
Just the beginning
DDS & SAS Solutions – A Platform for Intelligent Business
Metadata – Nature and Purpose
Organisation - Manage Environments & Code
When writing ETL and using DI Studio, Design for Change
37 / 01 JUNE 2008 / EDS
EDS Australia229 Elizabeth StreetBrisbane, QLD [email protected] or eds.com
EDS and the EDS logo are registered trademarks of Electronic Data Systems Corporation. EDS is an equal opportunity employer and values the diversity of its people. © 2007 Electronic Data Systems Corporation. All rights reserved.
Top Related