Edward L. Kimball - Spencer W. Kimball and the Revelation on Priesthood
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16
-
Upload
michael-rainey -
Category
Data & Analytics
-
view
406 -
download
0
Transcript of A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16
![Page 1: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/1.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Michael Rainey | Collaborate 16
A Walk Through the Kimball ETL Subsystems with Oracle Data Integration
1
![Page 2: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/2.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Introduction
2
• Michael Rainey - Data Integration Practice Lead - America- Oracle Data Integration expertise - Blog: http://ritt.md/mRainey - Oracle ACE
@mRainey
![Page 3: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/3.jpg)
[email protected] www.rittmanmead.com @rittmanmead
About Rittman Mead
3
•World’s leading specialist partner for technical excellence, solutions delivery and innovation in Oracle Data Integration, Business Intelligence, Analytics and Big Data
•Providing our customers targeted expertise; we are a company that doesn’t try to do everything… only what we excel at
•70+ consultants worldwide including 1 Oracle ACE Director and 3 Oracle ACEs, offering training courses, global services, and consulting
•Founded on the values of collaboration, learning, integrity and getting things done
Unlock the potential of your organization’s data
•Comprehensive service portfolio designed to support the full lifecycle of any analytics solution
![Page 4: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/4.jpg)
[email protected] www.rittmanmead.com @rittmanmead 4
Visual Redesign Business User Training
Ongoing SupportEngagement Toolkit
Average user adoption for BI platforms is below 25%
Rittman Mead’s User Engagement Service can help
More info: http://ritt.md/ue
![Page 5: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/5.jpg)
[email protected] www.rittmanmead.com @rittmanmead
What’s Most Important for YOU in Data Integration?
5
• Big data?
![Page 6: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/6.jpg)
[email protected] www.rittmanmead.com @rittmanmead
What’s Most Important for YOU in Data Integration?
5
• Big data?• Cloud?
![Page 7: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/7.jpg)
[email protected] www.rittmanmead.com @rittmanmead
What’s Most Important for YOU in Data Integration?
5
• Big data?• Cloud?
• Financial Reporting on “one version of the truth”?
![Page 9: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/9.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Wait! What are Kimball ETL Subsystems?Do you all know of Ralph Kimball?
7
www.kimballgroup.com
![Page 10: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/10.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Wait! What are Kimball ETL Subsystems?Do you all know of Ralph Kimball?
7
www.kimballgroup.com
Ralph Kimball founded the Kimball Group. Since the mid-1980s, he has been the DW/BI industry’s thought leader on the dimensional approach and trained more than 20,000 students. Prior to working at Metaphor and founding Red Brick Systems, Ralph co-invented the first commercially-available workstation with a graphical user interface at Xerox’s Palo Alto Research Center (PARC). Ralph has his Ph.D. in Electrical Engineering from Stanford University.
![Page 11: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/11.jpg)
[email protected] www.rittmanmead.com @rittmanmead
The Kimball GroupDo you all know of Ralph Kimball?
8
![Page 12: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/12.jpg)
[email protected] www.rittmanmead.com @rittmanmead
The Kimball 34 Subsystems of ETL
9
• Extracting Data - Data Profiling - Change Data Capture System - Extract System
![Page 13: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/13.jpg)
[email protected] www.rittmanmead.com @rittmanmead
The Kimball 34 Subsystems of ETL
10
• Cleaning and Conforming Data - Data Cleansing System - Error Event Schema - Audit Dimension Assembler - Deduplication System - Conforming System
![Page 14: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/14.jpg)
[email protected] www.rittmanmead.com @rittmanmead
The Kimball 34 Subsystems of ETL
11
• Delivering Data for Presentation - Slowly Changing Dimension
Manager - Surrogate Key Generator - Hierarchy Manager - Special Dimensions Manager - Fact Table Builders - Surrogate Key Pipeline - Late Arriving Data Handler
- Multi-Valued Dimension Bridge Table Builder
- Dimension Manager System - Fact Provider System - Aggregate Builder - OLAP Cube Builder - Data Propagation Manager
![Page 15: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/15.jpg)
[email protected] www.rittmanmead.com @rittmanmead
The Kimball 34 Subsystems of ETL
12
• Managing the ETL Environment - Job Scheduler - Backup System - Recovery and Restart System - Version Control System - Version Migration System - Workflow Monitor - Sorting System
- Lineage & Dependency Analyzer
- Problem Escalation System - Parallelizing / Pipelining System - Security System - Compliance Manager - Metadata Repository Manager
![Page 16: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/16.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Oracle Data Integration Solutions
13
Copyright*©*2015,*Oracle*and/or*its*affiliates.*All*rights*reserved.**|* Oracle*Open*World*2015* 1*
NoETL*Engine*100%*NaEve*Data*TransformaEon*
Data$Integrator$
Big$Data$Prepara/on$
GoldenGate$
Data$Quality$
Data$Service$Integrator$
Metadata$Management$
NonIinvasive*CDC,*RealEme*streaming*
data*delivery*
Profile,*Cleanse,*Match,*and*
Remediate*Data*
Prepare,*Secure,*Enrich*and*Publish*Unstructured*Data*
Catalog,*Trace*and*View*Models*across*
the*Enterprise*
Federate*Data*Across*DBs,*Services*and*ApplicaEons*
![Page 17: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/17.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Now let’s take a walk through the ETL Subsystems
14
![Page 22: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/22.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data
17
• Data Profiling- Oracle Enterprise Data Quality • Change Data Capture System• Extract System- Oracle Data Integrator - Oracle GoldenGate
![Page 23: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/23.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data
17
• Data Profiling- Oracle Enterprise Data Quality • Change Data Capture System• Extract System- Oracle Data Integrator - Oracle GoldenGate
![Page 30: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/30.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Data Profiling with EDQ
19
• Small dataset due to sampling percentage
![Page 31: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/31.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Data Profiling with EDQ
19
• Small dataset due to sampling percentage
• _projectid looks like a primary key
![Page 32: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/32.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Data Profiling with EDQ
19
• Small dataset due to sampling percentage
• _projectid looks like a primary key
• Investigate school_district blanks
![Page 33: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/33.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Data Profiling with EDQ
19
• Small dataset due to sampling percentage
• _projectid looks like a primary key
• Investigate school_district blanks
![Page 34: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/34.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Oracle Data Integrator
20
• Extract from many different systems? Yes!
- Multiple technologies OOTB - Custom technologies can be added • Data Server - connection to the
data source- Physical Schema - Logical Schema
![Page 35: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/35.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Oracle Data Integrator
20
• Extract from many different systems? Yes!
- Multiple technologies OOTB - Custom technologies can be added • Data Server - connection to the
data source- Physical Schema - Logical Schema
![Page 36: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/36.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Oracle Data Integrator
21
• Models - Based on a single data
source • Datastores- Logically represent a
table, file, XML, etc - Reverse engineer or
build manually
![Page 37: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/37.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Oracle Data Integrator
21
• Models - Based on a single data
source • Datastores- Logically represent a
table, file, XML, etc - Reverse engineer or
build manually
![Page 38: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/38.jpg)
[email protected] www.rittmanmead.com @rittmanmead
City
Extracting Data - Oracle Data Integrator
21
• Models - Based on a single data
source • Datastores- Logically represent a
table, file, XML, etc - Reverse engineer or
build manually
![Page 39: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/39.jpg)
[email protected] www.rittmanmead.com @rittmanmead
City
Extracting Data - Oracle Data Integrator
21
• Models - Based on a single data
source • Datastores- Logically represent a
table, file, XML, etc - Reverse engineer or
build manually
State
![Page 40: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/40.jpg)
[email protected] www.rittmanmead.com @rittmanmead
City
Extracting Data - Oracle Data Integrator
21
• Models - Based on a single data
source • Datastores- Logically represent a
table, file, XML, etc - Reverse engineer or
build manually
StateZip Code
![Page 44: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/44.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Changed Data Only
23
• Change Data Capture- Extract only the changed data since the last ETL extract • Methods- Audit columns - Timed extract - Full “diff compare” - Database log scraping
![Page 45: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/45.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data - Changed Data Only
23
• Change Data Capture- Extract only the changed data since the last ETL extract • Methods- Audit columns - Timed extract - Full “diff compare” - Database log scraping
![Page 51: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/51.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data
26
• Data Profiling- Oracle Enterprise Data Quality • Change Data Capture System• Extract System- Oracle Data Integrator - Oracle GoldenGate
![Page 52: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/52.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Extracting Data
26
• Data Profiling- Oracle Enterprise Data Quality • Change Data Capture System• Extract System- Oracle Data Integrator - Oracle GoldenGate
![Page 53: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/53.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Cleaning and Conforming Data
27
• Data Cleansing System- ODI & EDQ • Error Event Schema- Built on ODI E$ tables • Audit Dimension Assembler• Deduplication System- EDQ • Conforming System
![Page 54: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/54.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Cleaning and Conforming - Data Cleansing System
28
• ODI - Check Knowledge Module- Check logical constraints - “Bad” data moves to error table • EDQ- Data cleansing audit processors
![Page 71: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/71.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Cleaning and Conforming - Error Event Schema
32
Image From: Data Warehouse Lifecycle Toolkit (Wiley Publishing, Inc: 2008).
![Page 72: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/72.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Cleaning and Conforming - Error Event Schema
32
SNP_CONDSNP_KEYSNP_JOIN
E$ Tables
Image From: Data Warehouse Lifecycle Toolkit (Wiley Publishing, Inc: 2008).
SNP_LPI_RUN
![Page 73: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/73.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Cleaning and Conforming - Deduplication System
33
![Page 74: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/74.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Cleaning and Conforming - Deduplication System
33
![Page 75: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/75.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Cleaning and Conforming Data
34
• Data Cleansing System- ODI & EDQ • Error Event Schema- Built on ODI E$ tables • Audit Dimension Assembler• Deduplication System- EDQ • Conforming System
![Page 76: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/76.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data
35
• Slowly Changing Dimension Manager
• Surrogate Key Generator• Hierarchy Manager• Special Dimensions Manager• Fact Table Builders• Surrogate Key Pipeline• Late Arriving Data Handler
•Multi-Valued Dimension Bridge Table Builder•Dimension Manager System•Fact Provider System•Aggregate Builder•OLAP Cube Builder•Data Propagation Manager
![Page 77: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/77.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data
36
• Slowly Changing Dimension Manager- ODI Integration Knowledge Module - Set SCD behavior type for each
target column • Surrogate Key Generator- Database Sequence objects and ODI Sequences • Fact Table Builder- Lookups in ODI
![Page 78: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/78.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data - Slowly Changing Dimension in ODI
37
![Page 79: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/79.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data - Slowly Changing Dimension in ODI
37
![Page 80: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/80.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data - Slowly Changing Dimension in ODI
37
![Page 83: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/83.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data - SCD in ODI - Surrogate Keys
38
Additional audit columns
![Page 84: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/84.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data - SCD in ODI - Surrogate Keys
38
Additional audit columns
![Page 87: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/87.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Delivering Data
40
• Slowly Changing Dimension Manager- ODI Integration Knowledge Module - Set SCD behavior type for each
target column • Surrogate Key Generator- Database Sequence objects and ODI Sequences • Fact Table Builder- Lookups in ODI
![Page 88: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/88.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment
41
• Job Scheduler• Backup System• Recovery and Restart System• Version Control System• Version Migration System• Workflow Monitor• Sorting System
• Lineage & Dependency Analyzer
• Problem Escalation System• Parallelizing / Pipelining
System• Security System• Compliance Manager• Metadata Repository Manager
![Page 89: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/89.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment - Job Scheduler
42
• Create ODI schedule on execution object
- Tied to an agent and context
• Limited flexibility- Custom Fiscal Month end,
for example
![Page 90: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/90.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment - Job Scheduler
43
• Alternative to ODI scheduler - external scheduling tool- ODI Scenarios and Load Plans can be executed via command
line script or web service
./startloadplan.sh LOAD_EDW GLOBAL 6 -AGENT_URL=http://localhost:20910/oraclediagent
![Page 91: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/91.jpg)
[email protected] www.rittmanmead.com @rittmanmead
12.2.1
Managing the ETL Environment - Version Control/Migration
44
• ODI 12.2.1 Lifecycle Management- Integrated with Subversion - Deployment Archives for code
migration between environments
![Page 92: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/92.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment - Workflow Monitor
45
![Page 93: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/93.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment - Workflow Monitor
45
![Page 94: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/94.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment - Workflow Monitor
45
![Page 95: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/95.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment - Workflow Monitor
45
Drilldown from ODI Session to SQL detailed activity report
Obtain real-time and historical agent statistics
![Page 96: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/96.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Managing the ETL Environment
46
• Job Scheduler• Backup System• Recovery and Restart System• Version Control System• Version Migration System• Workflow Monitor• Sorting System
• Lineage & Dependency Analyzer
• Problem Escalation System• Parallelizing / Pipelining
System• Security System• Compliance Manager• Metadata Repository Manager
![Page 97: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/97.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Where did we end up?
47
• The Kimball ETL Subsystems will guide your data warehouse program
• Oracle Data Integration can help you fully implement the ETL Subsystems
- Extract, Load, Transform with ODI and GoldenGate
- Profile and cleanse data with Enterprise Data Quality
![Page 98: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/98.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Where did we end up? One version of the truth…
48
![Page 100: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/100.jpg)
[email protected] www.rittmanmead.com @rittmanmead
Questions?
50
• Websites- kimballgroup.com - rittmanmead.com/blog • Contact- [email protected] - [email protected] • Twitter- @rittmanmead - @mRainey
![Page 101: A Walk Through the Kimball ETL Subsystems with Oracle Data Integration - Collaborate16](https://reader031.fdocuments.in/reader031/viewer/2022030314/5886ad091a28ab6d0e8b4a1d/html5/thumbnails/101.jpg)
[email protected] www.rittmanmead.com @rittmanmead 51