Post on 14-Dec-2015
BW Basic Architecture
Klaus MajenzSAP – Product Line BI
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 2
Overview
complete DW & BI product, comprising ... ETL tools (extractors, transformation, monitoring,
scheduling, ...) OLAP engine data mining engine repository analytical front-end (web- or Excel-based, agents, GIS, ...) prepacked models, built by SAP application departments
client-server architecture SAP web application servers database server: 7 commercial RDBMS platforms
supported (Oracle, MS, 4IBM, SAP)
part of SAP Netweaver™ SAP's open integration and application platform
more details: http://www.sap.com/solutions/netweaver/
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 3
Overview
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 4
Characteristics Key Figures
Infoobjects
Year Month Day City Region Country Sales Pers. Division Distr. Ch. Sales Org. Product Product Grp. Quantity in PC Profit in $1998 011998 19980101 BERLIN DE-NORTH DE JIM SOFTWARE INTERNET AMERICAS PAINT OFFICE 730 67631998 011998 19980101 BERLIN DE-NORTH DE MARISA SOFTWARE INTERNET AMERICAS PAINT OFFICE 390 26141998 011998 19980102 BERLIN DE-NORTH DE JACK SOFTWARE AGENT EUROPE PAINT OFFICE 780 38661998 011998 19980103 BERLIN DE-NORTH DE MANDY SOFTWARE RETAIL EUROPE WIN-OS OS 970 -37341998 011998 19980120 MILAN IT-NORTH IT MONICA SOFTWARE AGENT EUROPE WIN-OS OS 190 13551998 011998 19980121 MILAN IT-NORTH IT MONICA SOFTWARE INTERNET EUROPE WIN-OS OS 810 75651998 011998 19980122 MILAN IT-NORTH IT BILL HARDWARE AGENT AMERICAS PC-3 PC 250 -8611998 011998 19980123 MILAN IT-NORTH IT JOE HARDWARE RETAIL AMERICAS PC-3 PC 40 -441998 011998 19980124 MILAN IT-NORTH IT JIM HARDWARE AGENT AMERICAS PC-3 PC 160 -5031998 011998 19980124 FRANKFURT DE-SOUTH DE KIM HARDWARE INTERNET EUROPE PC-3 PC 50 181998 011998 19980125 FRANKFURT DE-SOUTH DE BILL HARDWARE INTERNET AMERICAS PC-3 PC 990 6468
Scenario (1)
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 5
Scenario (2)
Dimension Time Day Month Year
Dimension Region City Region Country
Dimension Sales Org Sales Person Division Distribution Channel Sales Organization
Dimension Product Product
Product Group
Key Figures Quantity (in pieces)
Profit (in US$)
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 6
An adequate BW Infocube IUSALES
Dimension IUSALEST 0CALDAY
0CALMONTH
0CALYEAR
Dimension IUSALES1 IUCITY
IUREGION
IUCOUNTRY
Dimension IUSALES2 IUSALPER
IUDIV
IUDCHAN
IUSALORG
Dimension IUSALES3 IUPROD
IUPRODGRP
Key Figures IUQUAN
IUPROFIT
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 7
Data Flow in BW
Source System (e.g. R/3, other DB, File, ...)
Persistent Staging Area (PSA)
Operational Data Store (ODS)
Infocube: F fact table
Infocube: E fact table
Extraction
Compression
Infocube Upload (from
PSA)
Infocube Upload (from ODS)
ODS Upload
Aggregate
Initial Fill, Roll-Up
ODS Activate
BW
Qu
ery
Cube Query
ODS Query
V.P. Query
V.P. Query
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 8
Data Flow in BW – what we will look at
Source System (e.g. R/3, other DB, File, ...)
Persistent Staging Area (PSA)
Operational Data Store (ODS)
Infocube: F fact table
Infocube: E fact table
Extraction
Compression
Infocube Upload (from
PSA)
ODS Upload
Aggregate
Initial Fill, Roll-Up
ODS Activate
BW
Qu
ery
Cube Query
Info
cu
be
OD
SP
SA
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 9
PSA
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 10
PSA table
huge number of individual INSERTs no UPDATE SELECT * FROM … WHERE "REQUEST" = … mass deleteion: DELETE … WHERE "PARTNO" = … / DROP PARTITION …
requestpackage
(within request)partition no.
record no.(within package)
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 11
ODS
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 12
ODS object = 3 tables
active data : /BIC/AOIUSALES00 modified data ("activation queue"): /BIC/AOIUSALES40 delta data ("change log") : /BIC/B0008215000
(PSA)
1. ODS upload: INSERT INTO "/BIC/AOIUSALES40"
2. ODS data activation: UPSERT "/BIC/AOIUSALES00" delta records: INSERT INTO "/BIC/B0008215000" (mass) DELETE FROM "/BIC/AOIUSALES40"
3. infocube delta upload from ODS: SELECT * FROM "/BIC/B0008215000"
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 13
ODS tables: /BIC/AOIUSALES00, /BIC/AOIUSALES40acti
ve d
ata
mod
ified
data
same as in PSA table
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 14
ODS Object (BW 3.0)
Active data
Staging EngineStaging Engine
Req2
Req3
Req1
Activation queue
Req.ID I Pack.ID I Rec.No
Change log
Doc.No I Value
ODSRx I P 1 I Rec.1I4711I 104711 I 10
Activation
Activation During activation the data is sorted by the key
fields of active data plus key fields of Activation queue.
This guaranties the correct sequence of the records and allows inserts instead of table locks .
REQU1 I P 1 I Rec.1I4711I 10REQU2 I P 1 I Rec.1I4711I 30
Upload to Activation queue Data from different requests are uploaded in
parallel to the activation queue
ODSRy I P 1 I Rec.1I4711I-10 ODSRy I P 1 I Rec.2I4711I+30
4711 I 30
Before- and After Image Request ID in activation queue and change log differ
from each other. After update, data in the activation queue is deleted.
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 15
Infocube
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 16
InfoCube: Star Schema
F, E DS
Y
X
(1) Fact Table
(2) Dimension
(3) time-independent-SID time-dependent-SID master SID Char
(4) SID AttrS
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 17
Infocube IUSALES
X (City)S (Population)
Facttable
Dimension 1
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 18
Infocube Indexing (1) – Oracle
X (City)S (Population)
Facttable
Dimension 1
Bitmap IndexB-Tree (unique)
B-Tree (non-unique)
line item dimension
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 19
Infocube Indexing (2) – MS SQL Server
X (City)S (Population)
Facttable
Dimension 1
B-tree Index (nonunique, nonclustered)B-Tree (unique, clustered)
line item dimension
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 20
Infocube Indexing (3) – Oracle
Bitmap IndexB-Tree (unique)
B-Tree (non-unique)
F Facttable
• single column indexes support queries
• P-index: compress
• additional bitmap index on part. column
E Facttable
"P-index"
partitioning column(for E facttable)
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 21
Infocube Indexing (4) – MS SQL Server
F Facttable
• single column indexes support queries
• P-index: compress
E Facttable
"P-index"
B-tree Index (nonunique, nonclustered)
B-Tree (unique, nonclustered)
Does not exist on MS-SQL
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 22
Infocube Operations (1)
INSERT: only F facttable array INSERT if array INSERT fails: UPSERT logic
DELETE request (mass deletion): only F facttable DELETE FROM "/BIC/FIUSALES" WHERE KEY_IUSALESP = … alternatively: DROP PARTITION
DELETE specified data DELETE FROM … WHERE …
UPSERT: only E facttable infocube compression (separate slide)
SELECT separate slide
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 23
Infocube Compression (ex.: request 3)b
efo
reafte
r
UPDATE
INSERT
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 24
Infocube Compression (2)
Oracle (via stored procedure; on DB server)
loop over rows for request REQ in F facttable attempt UPDATE of E facttable if UPDATE fails then INSERT rowid into temporary table INS
do mass INSERT INTO E facttable using INS DROP PARTITION corresponding to REQ in F facttable
MS SQL-Server (via ABAP; via application server)
loop over rows for request REQ in F facttable attempt UPDATE of E facttable if UPDATE fails then attempt INSERT
DELETE FROM F facttable WHERE requestid = REQ
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 25
Aggregate Fill
INSERT INTO [/BIC/E100010] SELECT [D1].[SID_IUCITY] AS [KEY_1000101], [D2].[SID_IUSALPER] AS [KEY_1000102], 0 AS [KEY_100010P], SUM ([F].[/BIC/IUPROFIT]), SUM ([F].[/BIC/IUQUAN]), COUNT(*) AS [FACTCOUNT] FROM [/BIC/FIUSALES] [F], [/BIC/DIUSALES1] [D1], [/BIC/DIUSALES2] [D2], [/BIC/DIUSALESP] [DP] WHERE [F].[KEY_IUSALES1] = [D1].[DIMID] AND [F].[KEY_IUSALES2] = [D2].[DIMID] AND [F].[KEY_IUSALESP] = [DP].[DIMID] AND [DP].[SID_0CHNGID] = 0 AND ( [F].[KEY_IUSALESP] = 0 OR [F].[KEY_IUSALESP] = 2 ) AND [DP].[SID_0REQUID] BETWEEN 0 AND 40 GROUP BY [D1].[SID_IUCITY], [D2].[SID_IUSALPER]
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 26
Aggregate Roll-Up
INSERT INTO [/BIC/F100011] SELECT [D1].[SID_IUCITY] AS [KEY_1000111], [D3].[SID_IUPROD] AS [KEY_1000112], 7 AS [KEY_100011P], SUM ([F].[/BIC/IUPROFIT]), SUM ([F].[/BIC/IUQUAN]), COUNT(*) AS [FACTCOUNT] FROM [/BIC/FIUSALES] [F], [/BIC/DIUSALES1] [D1], [/BIC/DIUSALES3] [D3], [/BIC/DIUSALESP] [DP] WHERE [F].[KEY_IUSALES1] = [D1].[DIMID] AND [F].[KEY_IUSALES3] = [D3].[DIMID] AND [F].[KEY_IUSALESP] = [DP].[DIMID] AND [DP].[SID_0CHNGID] = 0 AND [F].[KEY_IUSALESP] = 5 AND [DP].[SID_0REQUID] = 498 GROUP BY [D1].[SID_IUCITY], [D3].[SID_IUPROD]
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 27
Infocube Query Example: Infocube IUSALES
(1) Fact Table
(2) Dimensions
(3) Characteristics (simplified)
month
year
day
city regioncountry
product product group
sales person
division
distribution channel
sales organization
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 28
Query Example & Processing (under Oracle)
month
year= [98-99]
regioncountry = 'US'
product group
(1) Fact Table
(2) Dimensions
(3) Characteristics (simplified)
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 29
Step 1: Restrictions Master Data Dimensions
month
year= [98-99]
regioncountry = 'US'
product group
(1) Fact Table
(2) Dimensions
(3) Characteristics (simplified)
Typical Query Processing
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 30
Step 2: Restrictions Dimensions Fact Table
product group
bitmap indexbitmap index
(1) Fact Table
(2) Dimensions
(3) Characteristics (simplified)
Typical Query Processing
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 31
Step 3: Assemble Result
product group
(1) Fact Table
(2) Dimensions
(3) Characteristics (simplified)
small subset of facttable
Typical Query Processing
month
year= [98-99]
regioncountry = 'US'
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 32
Query Example (1) – simple
SELECT "DT"."SID_0CALMONTH" AS "S____081" ,"DT"."SID_0CALYEAR" AS "S____083" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"D3"."SID_IUPRODGRP" AS "S____524" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN" FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALES3" "D3" , "/BIC/DIUSALESP" "DP" WHERE "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALES3" = "D3"."DIMID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND ( "DT"."SID_0CALMONTH" = 200007 AND "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 ) GROUP BY "DT"."SID_0CALMONTH", "DT"."SID_0CALYEAR", "D1"."SID_IUCOUNTRY", "D3"."SID_IUPRODGRP"
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 33
Query Example (2) – navigational attribute
SELECT "DT"."SID_0CALMONTH" AS "S____081" ,"DT"."SID_0CALYEAR" AS "S____083" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"X1"."S__IUCOLOR" AS "S____530" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN"FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALES3" "D3" , "/BIC/XIUPROD" "X1" , "/BIC/DIUSALESP" "DP"WHERE "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALES3" = "D3"."DIMID" AND "D3"."SID_IUPROD" = "X1"."SID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND ( "DT"."SID_0CALMONTH" = 200007 AND "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 AND "X1"."OBJVERS" = 'A' )GROUP BY "DT"."SID_0CALMONTH", "DT"."SID_0CALYEAR", "D1"."SID_IUCOUNTRY", "X1"."S__IUCOLOR"
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 34
Query Example (3) – external hierarchy
SELECT "DT"."SID_0CALYEAR" AS "S____083" ,"DT"."SID_0CALMONTH" AS "S____081" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"H1"."PRED" AS "S____524" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN"FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALES3" "D3" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALESP" "DP" , "/BI0/0300148611" "H1" /* This is a (UNION) view! */WHERE "F"."KEY_IUSALES3" = "D3"."DIMID" AND "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND "D3"."SID_IUPRODGRP" = "H1"."SUCC" AND ( "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 AND "H1"."SUCC" <> 2000008999 )GROUP BY "H1"."PRED", "DT"."SID_0CALYEAR", "DT"."SID_0CALMONTH", "D1"."SID_IUCOUNTRY"
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 35
Examples of Conceptual Modeling
in SAP BW
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 36
Examples
Reveal why pure RDBMS technology ...
sometimes requires an additional conceptual layer on top,
is not sufficient is some cases,
has no chance in some situations because it has to be more general than necessary.
Examples
example 1: infoproviders in SAP BW uniform view on differing physical layouts
example 2: non-cumulative key figures in SAP BW semantic relationship between table columns
example 3: aggregates in SAP BW could be implemented by using materialized views (or equivalent) but they have proved to be inferior
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 37
Example 1: Infoprovider (1)
An infoprovider in SAP BW ...
comprises a reporting scenario,
is the entity on which a query is defined,
combines (aggregated or non-aggregated) operational data with master data (e.g. product, customer, ... data),
or constitutes a master data entity
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 38
Example 1: Infoprovider (2) – Examples
Example A:
a cube is an infoprovider
fact table holds operational data on certain granularity
dimensions hold master data
Example B:
customer master data can be an infoprovider
same UI as for other infoproviders
selections, projections, summaries using attributes (e.g. address, customer category, region, ...)
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 39
Example 1: Infoprovider (3) -- Overview (SAP BW 3.x)
Infoprovider
Multi-provider
UNION
Infoset
JOIN
Infocube
multi-dim.
analytical
sales cube
ODS-Object
flat
operational
POS data
Characteristic(master data)
flat
master data
product data
Virtual Infoprovider
APIe.g. remote
access
real time
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 40
Example 2: Non-Cumulative Key Figures (1)
also: "semi-additive measures"
example: account balance
conceptually:
physically:
account day balance delta
A 29-Sep 100 € 10 €
B 29-Sep 500 €
A 30-Sep 110 €
B 30-Sep 500 € - 100 €
A 1-Oct 110 €
B 1-Oct 400 €
A 2-Oct 110 €
B 2-Oct 400 €
A 3-Oct 110 € - 60 €
B 3-Oct 400 €
A 4-Oct 50 €
B 4-Oct 400 €
account day ref point delta
A 29-Sep no 10 €
B 30-Sep no - 100 €
A 3-Oct no - 60 €
A 4-Oct yes 50 €
B 4-Oct yes 400 €
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 41
Example 2: Non-Cumulative Key Figures (2)
non-cumulative key figures / semi-additive measures balance can be reconstructed for any moment in the past
that information has not to be physically stored advantages
significantly reduced data volumes better performance more flexibility
however: algorithms are required for reconstruction read insertion load
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 42
Example 3: Aggregates in SAP BW
SAP BW constraints:
only SUM, MIN, MAX aggregations are materialized
uploaded data (in an infocube) can still be identified
delta roll-ups are simple
Materialized or Indexed Views / Automatic Summary Tables
could be used in theory
however: maintenance is considerably slower
... due to expensive tracking and logging mechanisms that are necessary if the general case has to be covered
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 43
Summary
SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 44
Summary
brief introduction to SAP BW
three examples:1. an additional conceptual layer on top of the relational one
2. a semantical pattern that is frequently used in business
3. an object that might suffer from the generic approach
Do the examples reveal shortcomings of RDBMS or are they application domain specific ?