Decision Support Technology. DSS Reference Architecture Language System Problem Processing System...
-
Upload
luis-hough -
Category
Documents
-
view
225 -
download
0
Transcript of Decision Support Technology. DSS Reference Architecture Language System Problem Processing System...
![Page 1: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/1.jpg)
Decision Support Technology
![Page 2: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/2.jpg)
DSS Reference Architecture
LanguageSystem Problem Processing
SystemKnowledgeSystem
PresentationSystem
![Page 3: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/3.jpg)
Outline
• Knowledge system technology– data management– data warehousing– Data marts– online analytic processing (OLAP)
• ROLAP, MOLAP, WOLAP
![Page 4: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/4.jpg)
Evolution of DSS• Transaction Processing Systems (TPS)
– Operational data stores and OLTP– Batch reports, hard to find and analyze
information, inflexible and expensive, reprogram every new request (circa 60’s)
• MIS– Management reporting from transactions in TPS– Still inflexible, not integrated with desktop tools
(circa 70’s)• DSS
– Combine data with analytic models or expert rules– Integration with desktop tools (80’s)
![Page 5: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/5.jpg)
Evolution of DSS• Data Warehousing
– Data integrated after (cleaning and scrubbing) from multiple sources (both internal and external to the organization)
– OLAP is the technology used to study the data in terms of operations on a multi-dimensional data set
– Data warehousing also supports processing of data by analytic methods and permits data mining (90’s)
![Page 6: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/6.jpg)
Applications
Retail - inventory management, promotions Manufacturing - order shipment Insurance – policy and claims tracking Telecommunications - call analysis Financial – account tracking CRM/eCRM – customer profiling, clickstream
analysis Healthcare – disease management, patient and
physician profiling
![Page 7: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/7.jpg)
Disease Management Programs
Address health quality of population by identifying at-risk members and applying prevention programs
Challenge: identifying at-risk members Data warehouses can help with this effort Aetna U.S. Healthcare
Members with certain ailments are flagged using an algorithm that examines member’s diagnoses, procedures, laboratories, and pharmaceuticals
Data gathered from medical and pharmaceutical claims, member and provider profiles (nearly 3 terabytes)
Results: reduction in frequency of acute asthmatic episodes, improvements in vaccination rates
![Page 8: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/8.jpg)
Data Model• A representation scheme with which to describe data: data
relationships, data semantics, and consistency constraints.
• Examples– ER (entity relationship model)
– Relational model
– Object-oriented model
• References:– http://www.smartdraw.com/resources/centers/software/erd.htm (ER
model)– http://www.fgcu.edu/support/office2000/index.html (MS
Office product tutorials)– http://mis.bus.sfu.ca/tutorials/MSAccess/tutorials.html (MS
Access tutorial)
![Page 9: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/9.jpg)
E-R schema: A simple example
physician patientevaluatesm n
-Modeling the structure of data, not the processing of it
![Page 10: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/10.jpg)
ER Schema: a detailed example
![Page 11: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/11.jpg)
Relational model
• Primary model used today for data-processing applications
• Database systems like Oracle, Sybase, Informix, MS SQL server etc. support this model
• Based on a well understood
theoretical model
![Page 12: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/12.jpg)
Example of a relational schema:
Physician(doc_id, d_name, specialty)
Patient(p_id, p_name)
Evaluates(doc_id, p_id, date, diagnosis)
An Example
![Page 13: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/13.jpg)
Essential features of the
relational model
• A relational model schema consists of relations or tables
• Each table has a set of fields (columns) that are related to one another
• One or more fields whose values determine the value of other fields are called keys
• Tables are normalized in order to remove redundancies
![Page 14: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/14.jpg)
New Data Types
• Text, images, video, audio, time series, spatial, ….
• Other, more exotic data types like fingerprints (an IBM Extender) and face recognition (an Informix DataBlade).
• Offered by IBM, Informix, Oracle
![Page 15: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/15.jpg)
Object-oriented Databases
• Objects belong to object classes determined by the structure (variables) and behavior (methods) of the object - not easy to represent in a relational database
• Methods that impact the state of the variables of the object are encapsulated within the object and facilitate interaction between objects
• eg. Object “bank account” modifies “balance” by “amount”
• Potentially reduced development and maintenance time
![Page 16: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/16.jpg)
From Databases to Transaction Processing
Change
Reality Abstraction
Transaction
Que
ry
AnswerDB'
DB
The real state is represented by an abstraction, called the database, and the transformation of the real state is mirrored by the execution of a program, called a transaction, that transforms the database.
Source: Jim Gray,MSFT
![Page 17: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/17.jpg)
Examples
• Point of sale systems– credit card transactions
• ATM machines– all withdrawals and deposits
• E-commerce web sites
• Health Care– medical records, billing
![Page 18: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/18.jpg)
Databases for Decision Support
• Transaction Processing systems are optimized for performance
• Data they capture are too detailed to be of use for decision support purposes
• Online Analytical Processing (OLAP) imposes very different demands on databases than does Online Transaction Processing (OLTP)
![Page 19: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/19.jpg)
Data Analysis: An Example
Reimbursements
Net Profits
1997 1998
45.4 Mill 52.4 Mill
3.2 Mill 5.2 Mill
% Inc.
15.41
62.50
![Page 20: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/20.jpg)
Reimbursements
Net Profits
1997 1998
45.4 Mill 52.4 Mill
3.2 Mill 5.2 Mill
% Inc.
15.41
62.50
North
South
East
West
Reimb. by region
7.2 Mill
13.4 Mill
18.4 Mill
6.4 Mill
7.5 Mill
18.4 Mill
17.4 Mill
9.1 Mill
4.17
37.31
-5.43
42.18
Data Analysis: A Hospital Example
![Page 21: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/21.jpg)
Specialty A
Specialty B
Specialty C
Specialty D
Reimb. - South by specialty
2.65 Mill
6.45 Mill
3.1 Mill
1.2 Mill
2.70 Mill
7.10 Mill
4.0 Mill
4.60 Mill
1.90
10.08
29.03
283.33
North
South
East
West
Reimb. By region
7.2 Mill
13.4 Mill
18.4 Mill
6.4 Mill
7.5 Mill
18.4 Mill
17.4 Mill
9.1 Mill
4.17
37.31
-5.43
42.18
Data Analysis: An Example
![Page 22: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/22.jpg)
Reimbursements
Net Profits
1997 1998
45.4 Mill 52.4 Mill
3.2 Mill 5.2 Mill
% Inc.
15.41
62.50
North
South
East
West
Profits by region
0.88 Mill
1.12 Mill
1.1 Mill
0.1 Mill
0.50 Mill
2.60 Mill
0.13 Mill
1.97 Mill
-43.18
132.1
-88.18
1870
Data Analysis: An Example
![Page 23: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/23.jpg)
Reimbursements
Net Profits
1997 1998
45.4 Mill 52.4 Mill
3.2 Mill 5.2 Mill
% Inc.
15.41
62.50
Specialty A
Specialty B
Specialty C
Specialty D
Profits-South by specialty
0.20 Mill
0.60 Mill
0.22 Mill
0.10 Mill
0.10 Mill
1.20 Mill
0.40 Mill
0.90 Mill
-50.00
100.0
81.82
800.0
Data Analysis: An Example
![Page 24: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/24.jpg)
Integration System
• Collects and combines information from disparate sources
• Provides integrated view, and a uniform user interface
• Supports sharing of data between entities
WorldWideWeb
Digital Libraries Scientific Databases
PersonalDatabases
Heterogeneous Database Integration
![Page 25: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/25.jpg)
Why look at data in this way?
– What would be the demand for services (forecasting)?– Who are our key customers/patients, and
– What are the margins/outcomes? (profitable customers/satisfied patients)
– How do we market to them/treat them?– What pricing/treatment strategy is desirable?– What are their preferences?– What type of customer/patient services are required?– What services when packaged result in higher/better
sales/revenues/margins/outcomes, efficient workflow?– Which promotion/patient education/counseling works or does
not work and why?– What is the inventory/patient turnover?– Which channel/technology is more effective/profitable?– Why do margins/outcomes differ from one place to another or
one patient to another?
![Page 26: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/26.jpg)
Customer Relationship Management
ANALYZE
TAKE ACTION
DISCOVER
Source: Pilot Software
Customer baseProfitabilityBuying patternSupport patternProductivity
Policies & ProceduresMarketing policiesSupport procedures
Trends in marketSelling opportunitiesOpp. For improvements
![Page 27: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/27.jpg)
Business IntelligenceSoftware applications, technologies, and
analytical methodologies that perform data analysis
Often used as a broad term that includes OLAP, data mining, query, reporting tools and technologies
![Page 28: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/28.jpg)
Business Intelligence Loop
Business Strategist
OLAP Data Mining Reports
Data Storage
Extraction, Transformation, & Cleansing
CRM Clinical IS Pharmacy Lab
DataWarehouse
DecisionSupport
![Page 29: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/29.jpg)
Data Warehouse• A decision support database that is maintained
separately from the organization’s operational databases.
• A data warehouse is a
– subject-oriented,
– integrated,
– time-varying,
– non-volatile collection of data that is used primarily in
organizational decision making (W.H. Inmon)
![Page 30: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/30.jpg)
Why Separate Data Warehouse?• Performance
Operational databases are designed & tuned for known transactions & workloads.
Complex OLAP queries would degrade performance for transactions.
Special data organization, access & implementation methods needed for multidimensional views & queries
Function– decision support requires historical data (up to 5 to 10
years of data)– consolidation of data from many operational systems and
external sources– data quality considerations (semantic and measurement
issues are resolved)
![Page 31: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/31.jpg)
Local vs Central Data Repository
Local Central
2 3
1 4
Data WarehouseRepository
KnowledgeGeneration
Local Rules Base
Local Data
KnowledgeGenerationHIS
Data
OtherData
Sources
Central Data
FeedbackOption
Forward Data
![Page 32: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/32.jpg)
Warehouse Database Schema
• ER design techniques not appropriate
• Design should reflect multidimensional view– Star Schema– Snowflake Schema– Fact Constellation Schema
![Page 33: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/33.jpg)
Multidimensional Data Model
• Database is a set of facts (points) in a multidimensional space• A fact has a measure dimension
– quantity that is analyzed, e.g., sale, budget• A set of dimensions on which data is analyzed
– e.g. , store, product, date associated with a sale amount• Dimensions form a sparsely populated coordinate system• Each dimension has a set of attributes
– e.g., owner, city and county of store• Attributes of a dimension may be related by partial order
– Hierarchy: e.g., street > county >city– Lattice: e.g., date> month>year, date>week>year
![Page 34: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/34.jpg)
Example: Patient profiling
A healthcare organization needed a longitudinal view of patients, including trends of services to patients
Model Facts include Healthcare (e.g., diagnosis, procedure),
Financial (e.g., amount billed, number of claims), Resources (e.g., number of bed-days, inpatient and outpatient visits)
Dimensions include Time, Provider, Claim Type, Demographics, Encounter Type, Diagnosis and Procedure, Person, Organization
Questions answered by system: Which individuals are eligible for services but not obtaining
them? Which individuals are registered for services, but not receiving
preventive healthcare?
![Page 35: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/35.jpg)
Example of a Star Schema
Order NoOrder No
Order DateOrder Date
Customer NoCustomer No
Customer Customer NameName
Customer Customer AddressAddress
CityCity
SalespersonIDSalespersonID
SalespersonNaSalespersonNameme
CityCity
QuotaQuota
OrderNOOrderNO
SalespersonIDSalespersonID
CustomerNOCustomerNO
ProdNoProdNo
DateKeyDateKey
CityNameCityName
QuantityQuantity
Total Price
ProductNOProductNO
ProdNameProdName
ProdDescrProdDescr
CategoryCategory
CategoryDescriptionCategoryDescription
UnitPriceUnitPrice
DateKeyDateKey
DateDate
CityNameCityName
StateState
CountryCountry
OrderOrder
CustomerCustomer
SalespersoSalespersonn
CityCity
DateDate
ProductProduct
Fact TableFact Table
![Page 36: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/36.jpg)
Star Schema
• A single fact table and a single table for each dimension
• Every fact points to one tuple in each of the dimensions and has additional attributes
• Does not capture hierarchies directly• Straightforward means of capturing a multiple
dimension data model using relations
![Page 37: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/37.jpg)
Example of a Snowflake Schema
Order NoOrder No
Order DateOrder Date
Customer NoCustomer No
Customer Customer NameName
Customer Customer AddressAddress
CityCity
SalespersonIDSalespersonID
SalespersonNaSalespersonNameme
CityCity
QuotaQuota
OrderNOOrderNO
SalespersonIDSalespersonID
CustomerNOCustomerNO
ProdNoProdNo
DateKeyDateKey
CityNameCityName
QuantityQuantity
Total Price
ProductNOProductNO
ProdNameProdName
ProdDescrProdDescr
CategoryCategory
CategoryCategory
UnitPriceUnitPrice
DateKeyDateKey
DateDate
MonthMonth
CityNameCityName
StateState
CountryCountry
OrderOrder
CustomerCustomer
SalespersoSalespersonn
CityCity
DateDate
ProductProduct
Fact TableFact Table
CategoryNaCategoryNameme
CategoryDeCategoryDescrscr
MontMonthh
YearYear YearYear
StateNameStateName
CountryCountry
CategoryCategory
StateState
MonthMonth
YearYear
![Page 38: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/38.jpg)
Snowflake Schema
• Represent dimensional hierarchy directly by normalizing the dimension tables
• Easy to maintain
• Saves storage, but may reduce effectiveness of browsing (Kimball)
![Page 39: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/39.jpg)
Fact Constellation
Store Key
Product Key
Period Key
Units
Price
Store Dimension
Product Dimension
SalesFact Table
Store Key
Store Name
City
State
Region
Product Key
Product Desc
Shipper Key
Store Key
Product Key
Period Key
Units
Price
ShippingFact Table
![Page 40: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/40.jpg)
Fact Constellation
• Multiple fact tables share dimension tables.
• This schema is viewed as collection of stars hence called galaxy schema or fact constellation.
• Sophisticated applications require such schema.
![Page 41: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/41.jpg)
Data Warehouse vs. Data Marts• Enterprise warehouse: collects all
information about subjects (customers, products, sales, assets, personnel) that span the entire organization.– Requires extensive business modeling– May take years to design and build
• Data Marts: departmental subsets that focus on selected subjects: Marketing data mart: customer, products, sales.– Faster roll out, but complex integration in
the long run.
![Page 42: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/42.jpg)
Virtual Warehouse
• Views over operational databases– Materialize some summary views for efficient
query processing, easier to build, requisite excess capacity on operational database servers
![Page 43: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/43.jpg)
Data Warehouse Architecture
ClinicalSystemClinicalSystem
PayrollSystemPayrollSystem
Billing SystemBilling System
ExternalData
ExternalData
Other Internal
Data
Other Internal
Data
(e.g.,AS400)
OracleFinancialson HP 9000
Access,Files (Industry Reports)
TransformationIntegration
DataWarehouse
DataWarehouse
Meta-Data
Excel Web Other
DataMining Tools
OLAP OLAP serversservers
![Page 44: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/44.jpg)
Extraction, Transformation, & Load (ETL)
ETL is a set of tools and techniques used to populate a data warehouse
Extraction Extract data from sources (e.g., operational
DBMSs, file systems, Web pages) Transformation
Clean data Convert from legacy/host format to
warehouse format (e.g., convert “surname” to “last name”)
![Page 45: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/45.jpg)
Extraction, Transformation, & Load (ETL)
Load Sort, summarize, consolidate, compute views, check
integrity, build indexes, partition Huge volumes of data to be loaded, yet small time window
(usually at night) when the warehouse can be taken off-line Techniques: batch, sequential load often too slow;
incremental, parallel loading techniques may be used
Refresh Propagate updates from sources to the warehouse When to refresh - on every update, periodically (e.g., every
24 hours), or after “significant” events How to refresh – full extract from base tables vs. incremental
techniques
![Page 46: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/46.jpg)
Metadata
Types of Metadata Administrative metadata
source databases and their contents gateway descriptions warehouse schema, view & derived data definitions dimensions, hierarchies pre-defined queries and reports data mart locations and contents data partitions data extraction, cleansing, transformation rules, defaults data refresh and purging rules user profiles, user groups security: user authorization, access control
![Page 47: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/47.jpg)
Metadata …
Types (continued) Business data
business terms and definitions ownership of data charging policies
Operational metadata data lineage: history of migrated data and sequence of
transformations applied currency of data: active, archived, purged monitoring information: warehouse usage statistics, error
reports, audit trails
Tools: Platinum Repository (Computer Associates) Meta Directory (Information Builders)…
![Page 48: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/48.jpg)
The Complete Decision Support System (Source: Franconi)
Information Sources Data Warehouse Server(Tier 1)
OLAP Servers(Tier 2)
Clients(Tier 3)
OperationalDB’s
SemistructuredSources
extracttransformloadrefreshetc.
Data Marts
DataWarehouse
e.g., MOLAP
e.g., ROLAP
serve
Analysis
Query/Reporting
Data Mining
serve
serve
![Page 49: Decision Support Technology. DSS Reference Architecture Language System Problem Processing System Knowledge System Presentation System.](https://reader036.fdocuments.in/reader036/viewer/2022062417/551a49ed550346a4248b5e3b/html5/thumbnails/49.jpg)
Three-Tier Architecture• Warehouse database server
– Almost always a relational DBMS; rarely flat files
• OLAP servers– Relational OLAP (ROLAP): extended relational DBMS that
maps operations on multidimensional data to standard relational operations.
– Multidimensional OLAP (MOLAP): special purpose server that directly implements multidimensional data and operations.
• Clients– Query and reporting tools.– Analysis tools (excel)– Data mining tools (e.g., trend analysis, prediction)