The Operational Data Store - Buckeye DAMA€¦ · Operational Data Store Information Workshop Meta...

Copyright © 2003 Intelligent Solutions, Inc. All Rights Reserved

The Operational Data StoreTactical Analysis at Your Fingertips

Claudia Imhoff, PhDIntelligent Solutions, [email protected]

www.IntelSols.com

1

2Copyright © 2003 Intelligent Solutions, Inc. All Rights Reserved

Agenda

Architectural Differences Between the ODS and the Data WarehouseClasses of the Operational Data StoreODS Interfaces–What Comes in and What Goes Out!Which do You Build First – ODS or Data Warehouse?


Corporate Information FactoryInformation Workshop

Meta Data Management

Operation & Administration

Library & Toolbox Workbench

Change Management

Service Management

Data Acquisition Management

Systems Management

Data Acquisition

CIF Data Management

Data Delivery

Information Feedback

API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems

OperationalData Store

Data Warehouse

Exploration Warehouse

Data Mining Warehouse

OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other


Challenges

New data requirements that do not appear to support traditional analytical BI

Membership number?Customer name, address, and phone number?Current status of problem?

Increasing demands for data freshness, currencyDailyHourlyUp-to-the-second


ChallengesIncreasing demands for availability

24 X 365!99.999% up time stats!

Increasing demands for query responseSub second

New requirement to manage data in the data warehouse that was produced by the data martAn old user with a new look asking unusual questions


CIF - Some Unusual Requests

I need hourly reportingon call center performance.

I need to be notified immediately if fraudis detected againsta customer’s account.

I need access to the status and results of previous contacts

with the customer.

I need to know whatorders from the 30 daybacklog are scheduledto be shipped today?

I need the DataMart rebuilt daily!

I need to know whatproducts belong to acustomer.

I need the customer’sname and address.

I need the DataWarehouse updateddaily!

I need to be able to assignand manage campaigns.


CIF - Some Unusual RequestsI need access to the status and results of previous contacts with the customer.

I need the DataMart rebuilt daily! I need hourly reporting

on call center performance.

I need to know whatproducts belong to acustomer.

I need to be notified immediately if fraudis detected againsta customer’s account. I need the customer’s

name and address.I need to know whatorders from the 30 daybacklog are scheduledto be shipped today?

I need the DataWarehouse updateddaily!

I need to be able to assignand manage campaigns.


Major Business FunctionsInformation Workshop




Change Management

Service Management


Systems Management

Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other

Business Operations

Business Intelligence

Business Management


Operational Data StoreInformation Workshop




Change Management

Service Management


Systems Management

Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other

The Operational Data Store is a subject-oriented, integrated, current, volatile collection of data used to support the tactical decision-making process for the enterprise.


Operational Data Store

... is a:

subject-orientedintegratedcurrentvolatile

… collection of data used to support the tactical decision making process for the enterprise.

as defined by W. H. Inmon, Claudia Imhoff and Greg Battasin Building the Operational Data Store


Operational Data StoreRole

Is the central point of data integration for Tactical AnalysisDelivers a common view of enterprise dataCurrent dataOLTP capabilities

ObservationsSupports actions resulting from Business IntelligenceactivitiesRelatively simple to deploy but expect more difficulty as data currency demands growProvide access from anywhere in the corporation via common messaging interface


Operational Data Store

Organizations are building Operational Data Stores to:Provide operational data integration or

consolidation to facilitate the sharing of critical data (e.g., integrated customer or product data)Provide integrated operational reporting across

the organization (daily batch reporting)Provide integrated operational level tactical

analysis that does not require time-series data.


Operational Data Store is NOT…Corporate

Information FactoryArchitecture

The ODS is NOT used for strategic analysisThe ODS is NOT the lowest level of detail in the data warehouse architectureThe ODS is NOT a staging area for the data warehouseThe ODS is NOT a department-specific application


CorporateInformation Factory

ArchitectureSummary

The ODS is a different structure than a data warehouse.

The ODS is not a staging area for the data warehouse.

The ODS is subject oriented, integrated, volatile and updated.


Agenda

Architectural Differences Between the ODS and the Data WarehouseClasses of the Operational Data StoreODS Interfaces – What Comes in and What Goes Out!Which do You Build First – ODS or Data Warehouse?


Classes ofODS

Classes of Operational Data Stores

Frequency of updateSynchronous with source systemsAsynchronous with source systems

Degree of integration and transformationDegree of summarization

Class I

Class II

Class III

Class IV

NOTE: Managing users frequency expectations can be difficult!


Classes ofODS

Class I - CharacteristicsClass I

Updated synchronouslyUpdates appear within 2 to 3 seconds after entered into source systemUses messaging middleware or enterprise application interface (EAI)Little – if any – integration and transformationHigh-performance, transaction dominated environmentLimited on instantaneous summarization


Classes ofODS

Class I - Example

Airline Flight Information ODS: A flight delay updated in the flight scheduling system in New York shows up both in San Francisco airport’s gate management system and on the airline’s web site within secondsSophisticated middleware or EAI makes this feasibleResource intensive High maintenance overheadDifficult to get initially synchronizedDifficult and complex to maintainAn expensive system!


Classes ofODS

Class II - Characteristics

Class IIData is stored and forwarded laterRefreshment multiple times a dayAnywhere from 15 minutes to several hours

Not the immediacy of Class I ODS

Some integration and transformation can occur as the data flows into the Class II ODS


Classes ofODS

Class II - ExampleBank Consolidated Accounts ODS: Consolidated

information on bank’s corporate customers with numerous dispersed accounts

The customer has many accounts with a nation-wide bank

Multiple branches

The bank has disparate source systemsCustomer information everywhere Consolidation is needed!

ODS will contain relatively simple data structures and integrationLess load on network than a Class I ODSLess expensive


Classes ofODS

Class III - Characteristics

Class IIIMost asynchronous class of ODSOvernight data movement to the ODS

Once a day and ONLY once a dayData is trapped in the operational environment and updated into the ODS

Some source data may be a snapshot at end of the dayUsually updated in batch modeSignificantly more integration and transformation possible


Classes ofODS

Class III - ExampleIntegrated Customer – Product Profile ODS: Computer peripherals manufacturer needs customer product profile information on a national basis for better customer support and enhanced sales opportunities

ODS is fully integrated and complex transformations are possible allowing integrationODS has complex data structuresEasiest to maintain


Classes ofODS

Class IV - Characteristics

Class IVAnalysis (profiling) completed in the data warehouse

Via Feedback loop knowledge is applied to current list on the ODS

Data movement at regular/irregular intervalsA Class I, II, or III can become a Class IVRequires the data warehouse to be in place


Other ODS RenditionsRefresh daily

Truncate tables and reloadEasy to buildNo integrationData starts aging immediatelyShort life

Combining a Class I, II, III and/or IVVery powerfulDifficult to constructChallenging to maintainFrequency of updates can be global

Classes ofODS


Classes ofODS

Class Comparisons Class I Class II Class III Class IV

Refresh Frequency

Seconds to Minutes

Minutes to Hours

Daily

W henever

Technologies Real-tim e m essaging m iddleware and ETL products

Store and forward m iddleware and ETL products

Batch oriented, store and forward m iddleware and ETL products

ETL

Degree of enterprise integration and transformation

None to Low Low to Medium Medium to High High

Cost High Moderate to High

Moderate Minim al

Sum marization None * Very Little ** Yes Yes


Classes ofODS

Summary

Classes of ODSDifferent technologies and purposesBased on different refresh needs and strategies

Class I is instantaneous updatesClass II has a frequency of 15 minutes to several hoursClass III is updated once a dayClass IV feeds information from the data warehouse to any class of ODS


Agenda



Operational Systems to ODS –ODS to Operational Systems

Information Workshop




Change Management

Service Management


Systems Management

Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other

Data Acquisition is the set of processes that capture, integrate, transform, cleanse, and load source data into the data warehouse and operational data store.


ODSInterfaces

Messaging Products

IBM MQSeries

Tibco

Neon

Tuxedo


ODSInterfaces

EAI VendorsBizTalk –MicrosoftData Junction –Integration ArchitectiWay – IBISeeBeyondVirtuoso – OpenLinkVitriaWebMethodsWebSphere - IBM


ODSInterfaces

ETL Tools and the ODSCoglin Mill – RodinData Mirror - ConstellarAb InitioEmbarcadero –DT/StudioMicrosoft – DTSHummingbird – GenioBusiness Objects –Data IntegratorSAS

IBM – Visual Warehouse, Data Propagator, Data JoinerCA – Data MoverData JunctionEtiInformatica – PowerCenterAscential – DataStageOracle – Warehouse Builder (Carleton)Sagent – Group 1


ODSInterfaces

EAI and ETL TogetherETL can use the logical goodness of EAI to pull dataReal-time integration and consolidation for the ODSETL becomes the ‘heavy-lifter’EAI becomes the connectivity

NOTE: Will these technologies merge into one?


ODS to Oper-MartInformation Workshop




Change Management

Service Management


Systems Management

Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other

The Oper-Mart is a subset of data derived from of the operational data store used in tactical analysis and usually stored in a multidimensional manner (star schema or hypercube). They may be created in a temporary manner and dismantled when no longer needed.


ODSInterfaces

Oper-Marts or ODS “Data Marts”Reporting cubes (OLAP), summary tables, small star schemasNot synchronous or dynamic

Rebuilt often

Reflect the data as of a point in timeWill likely lag in currency from the rest of the ODS


ODSInterfaces

Cubes and the ODS

Drill thru to Operational systemsDigital DashboardsPlug in ODS, Oper Mart with a portalVendors

Cognos – PowerPlaySQLServer – Analytic ServicesHyperion – EssbasePackaged Analytics


Managed Query Tools and the ODS

Business ObjectsCognos - ImpromptuCrystal Reports (Business Objects)Brio Query (Hyperion)SASCA – Forest and TreesCA – Info BeaconOracle ReportsMicrostrategies


ODS to DW and DW or Data Mart to ODS

ODSInterfaces





Change Management

Service Management


Systems Management

Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other


ODSInterfaces

Transactional Interface


Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other





Change Management

Service Management


Systems Management

Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other

The Transactional Interface is an easy-to-use and intuitive interface for the end user to access and manipulate data in the operational data store.


Transactional Interface Programming Tools and the ODS

ODSInterfaces

Microsoft - Visual BasicSybase – PowerbuilderJAVAC, C++COBOLOracle Forms


ODSInterfaces

Decision Support InterfaceInformation Workshop




Change Management

Service Management


Systems Management

Data Acquisition

CIF Data Management

Data Delivery


API

API

API

API DSI

DSI

TrI

DSI

DSI

Operational Systems


Data Warehouse



OLAP Data Mart

Oper Mart

External

ERP

Internet

Legacy

Other

The Decision Support Interface is an easy-to-use, intuitive tool to enable end user capabilities such as exploration, data mining, OLAP, query, and reporting to distill information from data.


ODSInterfaces

SummaryInterfaces will need to be created:

From the operational systems to the ODS (and back)From the ODS to the oper-martFrom the ODS to the data warehouseFrom the data warehouse or data mart to the ODSTransactional interface

Middleware software – a very useful where frequency of update is extremely importantETL tools (with meta data) – a large part of work effort


Agenda



ODS BestPractices

Reasons to Build an ODS

Integration of corporate data

Tactical decision support

Data sharing and accessibility

Reporting to off-load the operational system


ODS BestPractices

ODS First

Complex operational systems

Multiple merges and acquisition

CRM – current customer informationAttention to customer touch processes

Objective: Integrated, high-quality, current accessible data


Reasons to Build a Data Warehouse

Need for historical analysis – trends, patterns

No clear picture of profitability

Losing customers and market share but don’t know why


ODS BestPractices

Data Warehouse First

Strategic marketing analysisAttract and retain a customer

NeedsWantsDesires

Determine customer buying habitsDetermine customer profitabilityPerform demographic profiling

Assumption: well integrated operational systems


Hybrid Environment?


Hybrid Environment

Can’t I build a hybrid environment?

The answer is - Of course you can.

A hybrid environment (mixing tactical OLTP and DSSwith strategic DSS) CAN be implemented.

However, just because you CAN do it doesn't meanyou SHOULD.

The massive differences in these two constructs shouldmake it clear that they should NEVER be combined.


Hybrid Environment

ODS and Data Warehouse differ greatly in:Table space allocationsReferential integrity strategyRow level lockingAmount of detailed data (historical versus current)Currency of the dataBackup and recovery strategiesIndexing schemesDisaster recovery strategyReorganization of the database


Hybrid Environment

Implications of mixing the two together:Architectural objectives will not be met.

Environment developed may not meet ANY client requirements (for either OLTP or OLAP).

Credibility and validity of analytical information may be compromised.Performance becomes a big issue.The cost in terms of technology and resources to maintain and monitor this hybrid are great.


ODS BestPractices

Summary

An organization builds an ODS for various reasons, one of the most important is an integrated customer database.The ODS plays an important role in CRM for customer integration.An integrated corporate operational data store can become crucial for day to day business in every industry.


Questions?


Recommended Reading


RecommendedReading

BooksMastering Data Warehouse Design - Relational and Dimensional Techniques, by Claudia Imhoff, Nicholas Galemmo, and Jonathan G. Geiger (John Wiley & Sons, 2003)

Claudia Imhoff, Nicholas Galemmo, and Jonathan GeigerJohn Wiley & Sons – ISBN 0-471-32421-3

Corporate Information FactoryW. H. Inmon, Claudia Imhoff and Ryan SousaJohn Wiley & Sons - ISBN 0-471-19733-5

Building the Customer Centric Enterprise: Data Warehousing Techniques for Supporting Customer Relationship Management

Claudia Imhoff, Lisa Loftis, and Jonathan G. GeigerJohn Wiley & Sons - ISBN 0-471-31981-3


RecommendedReading

BooksData Warehousing for e-Business

W. H. Inmon, R. H. Terdeman, Joyce Norris-Montanari, Dan MeersJohn Wiley & Sons – ISBN 0-471-41579-0

Building the Data WarehouseW. H. InmonJohn Wiley & Sons - ISBN 0-471-14161-5

Building the Operational Data StoreW. H. Inmon, Claudia Imhoff and Greg BattasJohn Wiley & Sons - ISBN 0-471-12822-8


RecommendedReading

BooksThe Data Warehouse Lifecycle Toolkit

Ralph Kimball, L. Reeves, M. Ross, W. ThornthwaiteJohn Wiley & Sons - ISBN 0-471-25547-5

Mastering Data MiningMichael J. A. Berry, Gordon LinoffJohn Wiley & Sons – ISBN 0-471-33123-6

Data Warehouse PerformanceW. H. Inmon, Ken Rudin, Christopher K. Buss, Ryan SousaJohn Wiley & Sons – ISBN 0-471-29808-5

Building and Managing the Meta Data RepositoryDavid MarcoJohn Wiley & Sons – ISBN 0-471-35523-2


RecommendedReading

BooksThe Data Model Resource Book (Volumes 1 and 2)

Len SilverstonJohn Wiley & Sons – ISBN 0-471-38023-7 & 0-471-35348-5

Improving Data Warehouse and Business Information QualityLarry P. EnglishJohn Wiley & Sons – ISBN 0-471-25383-9

Data Warehouse Management HandbookRichard KachurPrentice Hall – ISBN 0-130-83346-0

Data Warehouse Project ManagementSid Adelman and Larissa T. MossAddison Wesley – ISBN 0-201-61635-1


Articles“The Devil You Know: Integrating Old Reports into New BI Systems” by Claudia Imhoff and Mike Biensen (DM Review, June 2003)“The Staging Area” by Joyce Norris-Montanari (TDWI Flashpoint, May 2003)“Just Plug in the Data Appliance and GO! By Claudia Imhoff (DM review, May 2003)“Keep your friends close, and your enemies closer” by Claudia Imhoff (DM Review, April 2003)“All Parallelism Is NOT Created Equal!” by Joyce Norris-Montanari (TDWI Flashpoint, March 2003)“End Users – Use ‘em or Lose ‘em – Round Two” by Claudia Imhoff (DM Review, March 2003)


Articles“Growing Pains” by Claudia Imhoff and David Imhoff (DM Review, February 2003)“Take a Trip and Never Leave the Farm” by Claudia Imhoff (DM Review, January 2003)“We are fam-i-ly – Managing Corporate Relationships for Better CRM” by Claudia Imhoff & Lisa Loftis (DM Review, December 2002)“Financial Analytics: Delivering Metrics That Matter” by ClaudiaImhoff and Raymond Pettit (DM Review, November 2002)“Crystal Clear Customers – The Role of the Operational Data Store” by Claudia Imhoff (DM Review, October 2002)“Want to Maximize Data Quality? Make Data an Enterprise Asset” by Claudia Imhoff and Jonathan G. Geiger (DM Review, September 2002)


Articles“CRM Analytics” by Claudia Imhoff and Lisa Loftis (DM Review, August 2002)“Where’s the Brain? The Role of the Program Management Office” by Claudia Imhoff (DM Review July 2002)“Analytical Applications – The New Kids on the Block” by Claudia Imhoff (DM Review June 2002)“The Oper Mart Application – Continuing the Story” by Claudia Imhoff and Joyce Norris-Montanari (DM Review May 2002)“Sharing the Wealth: Putting it all together in the Corporate Information Factory” by Claudia Imhoff (DM Review April 2002)


Articles“The Corporate Information Factory Comes of Age” by Claudia Imhoff and the Intelligent Solutions Team (DM Review March 2002)“The CRM Maturity Scale” by Claudia Imhoff (DM Review, February 2002)“I’ve Got a Secret – Preserving Customer Trust and the Role of Privacy” by Claudia Imhoff and Jonathan Geiger (DM Review, January 2002)“Making CRM Technologies Work – Hard! The Interplay of the Information Factory Components” by Claudia Imhoff (DM Review, December 2001)


Articles“Instant CRM – Just add vendors!” by Claudia Imhoff and Lisa Loftis (DM Review, November 2001)“Do It My Way or the Highway…Evolving from Personalization to Customization” by Claudia Imhoff (DM Review, October 2001)“Subject-Orientation in a Data Warehouse” by Jonathan G. Geiger (The Data Warehousing Institute Flashpoint, September 26, 2001)“Oper-Marts – An Evolution of the Operational Data Store” by Claudia Imhoff (DM Review, September 2001)“Getting to Know You, Getting to Know All About You” (DM Review, August 2001)“All Mach – No Vector” by Claudia Imhoff (DM Review, July 2001)“CRM ROI – Oxymoron or Management Mandate?” by Claudia Imhoff (e-Business Advisor, July 2001)


Articles“CRM – Technology is not Enough” by Claudia Imhoff and Lisa Loftis (DM Review, June 2001)“The Good, the Quick and the Easy” by Claudia Imhoff (DM Review, May 2001)“Consultant Prescription: Augmenting your IT blood bank for a BI project” by Claudia Imhoff (DM Review, May 2001 Consultant’s Guide)“Create an Enterprise Portal Infrastructure” by Claudia Imhoff (eBusiness Advisor, May 2001)“Mass Customization – The Next Technological Advance for Business Intelligence” by Claudia Imhoff (DM Review, April 2001)“Creating a Truly Customer-Centric Enterprise - The Role of Analytical CRM” by Claudia Imhoff (Microsoft Executive Circle, Q2 2001)


Articles“Quality Relationships Begin With Quality Data” by Claudia Imhoff and Jonathan G. Geiger (e-Business Advisor March 2001)“Howdy Pard’ner” by Claudia Imhoff and Jonathan G. Geiger (DM Review March 2001)“My, How Times Change” by Claudia Imhoff and Jonathan G. Geiger (DM Review, February 2001)

The Operational Data Store - Buckeye DAMA€¦ · Operational Data Store Information Workshop Meta...

Documents

Transcript of The Operational Data Store - Buckeye DAMA€¦ · Operational Data Store Information Workshop Meta...