Step towards Business Intelligence › Steps-Towards... · Data is kept in a specific business line...
Transcript of Step towards Business Intelligence › Steps-Towards... · Data is kept in a specific business line...
Steps Towards Business Intelligence
Ahsan Kabir ,MVPChapter Leader “techforum PASS”
What is BI
Business Intelligence is an umbrella term thatincludes the applications, infrastructure andtools, and best practices that enable decisionmakers to make proper decisions.
• What happened?
• What is happening?
• Why did it happen?
• What will happen?
Past
Present
Future
“Understand the pulse of the
Organization”
Why BI
How BI ?
Microsoft BI Technologies
What is DW?
“…is designed specifically to be a central repository for all data in a company separated from transactional systems.”
“…is designed to be the source of analysis and reports.”
“But it’s not a copy of a source database.”
Why DW
• Central Repository
• Reduce extra load
• sources unaffected
• Empower Business Users
• Improve data quality
• Single version of the truth
1) Data volumes,
2) Real-time data,
3) New sources and types of data, and
4) Cloud-born data
But …..
Now
• The data warehouse is unable to keep up with explosive volumes.
• The data warehouse is falling behind the velocity of real-time performance requirements.
• The data warehouse is slower than desired in adopting a variety of new data sources, slowing time–to-value
• The platform costs more, while performance lags.
Planning
1. Analytical and Report requirement
2. Business Process
3. Prioritization
4. Identify Source Data
5. Dimensional Model
6. Documentation
7. Design Data Warehouse
Data Warehouse vs.Data Mart
Data Warehouse
Enterprise-wide
Union of all data marts
Data Mart
Departmental or Business line
Single business process
Kimball
Bottom-Up
Data marts
Logical data warehouse
Decentralized
Quick results, iterative approach
Inmon
Top-Down:
Enterprise data model
Centralized
Later create data marts
More upfront work but less redo
Kimball vs. InmonMethodology
Data Model
OLAP cube /Multidimensional modeling :
“…is based on the OLAP cube and is fitted with measures and dimensions”
In-memory tabular model:
“…is based on a new In-memory engine for tables “
OLAP cube/Multidimensional modeling
Fact or measure
“… are numeric and additive values “
Foreign keys
Dimension
“…Descriptive information”
Surrogate key.
Business key.
SSAS Loaded into in-Memory engine called xVelocity in-memory.
Tabular modeling allows you to create a table-based model from existing data in Warehouse and and create a relationship between models.
Data Analysis Expression (DAX) is an expression language for SSAS Tabular, which helps you create calculations and measures based on existing columns and relationships.
Tabular Model
Schema Design
The layout indicate the relationship between facts and dimensions is called a schema.
Star Schema :
For each fact entity join with single level of dimension entities.
Snowflake Schema :
If there are dimensions with large numbers of attributes, it might be necessary to break the dimensions down into sub dimension entities
Star Schema
Snowflake Schema
DEMO
Analysis Services (SSIS)
1. Develop Cubes and 2. Create dimensions and measures.3. Creating hierarchies4. MDX queries will be compiled,
parsed, and executed in the SSAS engine
ETL
“…is a program that periodically runs.”
Extract
Fetching data from the source
relational databases, web services, and
SharePoint lists.
Transform
“..Cleansing the data and converting to a
OLAP-friendly data model”
Load
“..loading data into the data warehouse
as fact and dimensions”
Demo
Data is kept in a
specific business
line wise.
Before enter into warehouse
Data is processed
(cleansed and transformed)
Warehouse Data Marts
Users query
the data
warehouse
“…staging area is an area where we fetch data from different sources exactly as it is into our integrated database. “
Staging
Data Quality Services
Data quality issues can be divided into the following categories:
Uniqueness
Validity
Accuracy
Standardization
Completeness
Name Address City House No DoB State Country
Ahsan CDA Avenue CTG 181/1 05/11/1978 BD
Kabir RB Avn CTG 41/6 23/04/1991 DHK Bangladesh
Before
After
Accuracy Consistency Completeness Conformity
Name Address City House
No
DoB Stat
e
Country
Ahsan CDA Avenue CTG 181/1 05/11/1978 CT Bangladesh
Kabir RB Avenue DHK 41/6 23/04/1991 DHK Bangladesh
Start DQS
Knowledge Base Management
Knowledge Base Management is where you can create and manage Knowledge Base, domains, and domain rules
Data quality projects
projects apply the Knowledge Base and matching rules on an existing dataset and provide results.
Administration
Configuration and administration tasks can be performed here
Components in DQS
1. Cleansing,
Cleansing is about cleaning data based on a Knowledge Base and domains.
2. Matching,
Matching would match data based on the similarity rules and threshold defined in a Knowledge Base.
3. Monitoring
Monitoring will show the status of records during the cleansing and matching projects.
4. Profiling.
Profiling will help in creating business rules or changing the domain rules and Knowledge Base from what the existing data profiling results are.
Demo
Technology
SSDT
“…is the integrated IDE for SSIS, SSRS, and SSAS. SSDT was formerly known as Business Intelligence Development Studio (BIDS). “
SSIS
SSIS was released with this name for the first time in 2005, but prior to that, it was named Data Transformation Services (DTS). DTS was available even in SQL Server 2000
SSRS
“is a data Visualization tools for
developing and publishing reports”
ReportServer DB
Report definition,
Snapshot,
Execution log etc.
ReportServer TempDB
Session and
Cached information.
Report Server web application
Report Manager web application
Reporting Services Configuration Manager.
Master Data Service (MDS)
“…is data shared across computer systems in the enterprise.”
“… is the dimension or hierarchy data in data warehouses and transactional systems”
“… is core business objects shared by applications across an enterprise
-The processes and technology to produce and maintain a single clean copy of master data
Customer
ABC
PQR
XYZ
Country
Europe
Norway
Sweden
Features
Domain management
models, entities, attributes, and hierarchies.
Business rules
Data validation is also provided.
Import and export master data
Data cleanup
Architecture
SQL Server database for storing data and metadata.
MDS engine read and write information to that database by : WebUI and Excel Add-ins.
MDS uses subscription views to export information from MDS to other systems
Staging mechanism to import data from other systems, which is called entity-based staging.
Demo
Resources:Data Warehouse Architecture – Kimball and Inmon methodologies: http://bit.ly/SrzNHy
SQL Server 2012: Multidimensional vs tabular: http://bit.ly/SrzX1x
Data Warehouse vs Data Mart: http://bit.ly/SrAi4p
Fast Track Data Warehouse Reference Guide for SQL Server 2012: http://bit.ly/SrAwsj
Complex reporting off a SSAS cube: http://bit.ly/SrAEYw
Surrogate Keys: http://bit.ly/SrAIrp
Normalizing Your Database: http://bit.ly/SrAHnc
Difference between ETL and ELT: http://bit.ly/SrAKQa
Microsoft’s Data Warehouse offerings: http://bit.ly/xAZy9h
Microsoft SQL Server Reference Architecture and Appliances: http://bit.ly/y7bXY5
Methods for populating a data warehouse: http://bit.ly/SrARuZ
Great white paper: Microsoft EDW Architecture, Guidance and Deployment Best Practices:
http://bit.ly/SrAZug
End-User Microsoft BI Tools – Clearing up the confusion: http://bit.ly/SrBMLT
Microsoft Appliances: http://bit.ly/YQIXzM
Thanks