BW Basic Architecture Klaus Majenz SAP – Product Line BI.

BW Basic Architecture

Klaus MajenzSAP – Product Line BI

SAP-AG 2005,BW Basicarchitecture, Klaus Majenz 2

Overview

complete DW & BI product, comprising ... ETL tools (extractors, transformation, monitoring,

scheduling, ...) OLAP engine data mining engine repository analytical front-end (web- or Excel-based, agents, GIS, ...) prepacked models, built by SAP application departments

client-server architecture SAP web application servers database server: 7 commercial RDBMS platforms

supported (Oracle, MS, 4IBM, SAP)

part of SAP Netweaver™ SAP's open integration and application platform

more details: http://www.sap.com/solutions/netweaver/


Overview


Characteristics Key Figures

Infoobjects

Year Month Day City Region Country Sales Pers. Division Distr. Ch. Sales Org. Product Product Grp. Quantity in PC Profit in $1998 011998 19980101 BERLIN DE-NORTH DE JIM SOFTWARE INTERNET AMERICAS PAINT OFFICE 730 67631998 011998 19980101 BERLIN DE-NORTH DE MARISA SOFTWARE INTERNET AMERICAS PAINT OFFICE 390 26141998 011998 19980102 BERLIN DE-NORTH DE JACK SOFTWARE AGENT EUROPE PAINT OFFICE 780 38661998 011998 19980103 BERLIN DE-NORTH DE MANDY SOFTWARE RETAIL EUROPE WIN-OS OS 970 -37341998 011998 19980120 MILAN IT-NORTH IT MONICA SOFTWARE AGENT EUROPE WIN-OS OS 190 13551998 011998 19980121 MILAN IT-NORTH IT MONICA SOFTWARE INTERNET EUROPE WIN-OS OS 810 75651998 011998 19980122 MILAN IT-NORTH IT BILL HARDWARE AGENT AMERICAS PC-3 PC 250 -8611998 011998 19980123 MILAN IT-NORTH IT JOE HARDWARE RETAIL AMERICAS PC-3 PC 40 -441998 011998 19980124 MILAN IT-NORTH IT JIM HARDWARE AGENT AMERICAS PC-3 PC 160 -5031998 011998 19980124 FRANKFURT DE-SOUTH DE KIM HARDWARE INTERNET EUROPE PC-3 PC 50 181998 011998 19980125 FRANKFURT DE-SOUTH DE BILL HARDWARE INTERNET AMERICAS PC-3 PC 990 6468

Scenario (1)


Scenario (2)

Dimension Time Day Month Year

Dimension Region City Region Country

Dimension Sales Org Sales Person Division Distribution Channel Sales Organization

Dimension Product Product

Product Group

Key Figures Quantity (in pieces)

Profit (in US$)


An adequate BW Infocube IUSALES

Dimension IUSALEST 0CALDAY

0CALMONTH

0CALYEAR

Dimension IUSALES1 IUCITY

IUREGION

IUCOUNTRY

Dimension IUSALES2 IUSALPER

IUDIV

IUDCHAN

IUSALORG

Dimension IUSALES3 IUPROD

IUPRODGRP

Key Figures IUQUAN

IUPROFIT


Data Flow in BW

Source System (e.g. R/3, other DB, File, ...)

Persistent Staging Area (PSA)

Operational Data Store (ODS)

Infocube: F fact table

Infocube: E fact table

Extraction

Compression

Infocube Upload (from

PSA)

Infocube Upload (from ODS)

ODS Upload

Aggregate

Initial Fill, Roll-Up

ODS Activate

BW

Qu

ery

Cube Query

ODS Query

V.P. Query

V.P. Query


Data Flow in BW – what we will look at

Source System (e.g. R/3, other DB, File, ...)

Persistent Staging Area (PSA)

Operational Data Store (ODS)

Infocube: F fact table

Infocube: E fact table

Extraction

Compression

Infocube Upload (from

PSA)

ODS Upload

Aggregate

Initial Fill, Roll-Up

ODS Activate

BW

Qu

ery

Cube Query

Info

cu

be

OD

SP

SA


PSA


PSA table

huge number of individual INSERTs no UPDATE SELECT * FROM … WHERE "REQUEST" = … mass deleteion: DELETE … WHERE "PARTNO" = … / DROP PARTITION …

requestpackage

(within request)partition no.

record no.(within package)


ODS


ODS object = 3 tables

active data : /BIC/AOIUSALES00 modified data ("activation queue"): /BIC/AOIUSALES40 delta data ("change log") : /BIC/B0008215000

(PSA)

1. ODS upload: INSERT INTO "/BIC/AOIUSALES40"

2. ODS data activation: UPSERT "/BIC/AOIUSALES00" delta records: INSERT INTO "/BIC/B0008215000" (mass) DELETE FROM "/BIC/AOIUSALES40"

3. infocube delta upload from ODS: SELECT * FROM "/BIC/B0008215000"


ODS tables: /BIC/AOIUSALES00, /BIC/AOIUSALES40acti

ve d

ata

mod

ified

data

same as in PSA table


ODS Object (BW 3.0)

Active data

Staging EngineStaging Engine

Req2

Req3

Req1

Activation queue

Req.ID I Pack.ID I Rec.No

Change log

Doc.No I Value

ODSRx I P 1 I Rec.1I4711I 104711 I 10

Activation

Activation During activation the data is sorted by the key

fields of active data plus key fields of Activation queue.

This guaranties the correct sequence of the records and allows inserts instead of table locks .

REQU1 I P 1 I Rec.1I4711I 10REQU2 I P 1 I Rec.1I4711I 30

Upload to Activation queue Data from different requests are uploaded in

parallel to the activation queue

ODSRy I P 1 I Rec.1I4711I-10 ODSRy I P 1 I Rec.2I4711I+30

4711 I 30

Before- and After Image Request ID in activation queue and change log differ

from each other. After update, data in the activation queue is deleted.


Infocube


InfoCube: Star Schema

F, E DS

Y

X

(1) Fact Table

(2) Dimension

(3) time-independent-SID time-dependent-SID master SID Char

(4) SID AttrS


Infocube IUSALES

X (City)S (Population)

Facttable

Dimension 1


Infocube Indexing (1) – Oracle


Facttable

Dimension 1

Bitmap IndexB-Tree (unique)

B-Tree (non-unique)

line item dimension


Infocube Indexing (2) – MS SQL Server


Facttable

Dimension 1

B-tree Index (nonunique, nonclustered)B-Tree (unique, clustered)

line item dimension


Infocube Indexing (3) – Oracle

Bitmap IndexB-Tree (unique)

B-Tree (non-unique)

F Facttable

• single column indexes support queries

• P-index: compress

• additional bitmap index on part. column

E Facttable

"P-index"

partitioning column(for E facttable)


Infocube Indexing (4) – MS SQL Server

F Facttable

• single column indexes support queries

• P-index: compress

E Facttable

"P-index"

B-tree Index (nonunique, nonclustered)

B-Tree (unique, nonclustered)

Does not exist on MS-SQL


Infocube Operations (1)

INSERT: only F facttable array INSERT if array INSERT fails: UPSERT logic

DELETE request (mass deletion): only F facttable DELETE FROM "/BIC/FIUSALES" WHERE KEY_IUSALESP = … alternatively: DROP PARTITION

DELETE specified data DELETE FROM … WHERE …

UPSERT: only E facttable infocube compression (separate slide)

SELECT separate slide


Infocube Compression (ex.: request 3)b

efo

reafte

r

UPDATE

INSERT


Infocube Compression (2)

Oracle (via stored procedure; on DB server)

loop over rows for request REQ in F facttable attempt UPDATE of E facttable if UPDATE fails then INSERT rowid into temporary table INS

do mass INSERT INTO E facttable using INS DROP PARTITION corresponding to REQ in F facttable

MS SQL-Server (via ABAP; via application server)

loop over rows for request REQ in F facttable attempt UPDATE of E facttable if UPDATE fails then attempt INSERT

DELETE FROM F facttable WHERE requestid = REQ


Aggregate Fill

INSERT INTO [/BIC/E100010] SELECT [D1].[SID_IUCITY] AS [KEY_1000101], [D2].[SID_IUSALPER] AS [KEY_1000102], 0 AS [KEY_100010P], SUM ([F].[/BIC/IUPROFIT]), SUM ([F].[/BIC/IUQUAN]), COUNT(*) AS [FACTCOUNT] FROM [/BIC/FIUSALES] [F], [/BIC/DIUSALES1] [D1], [/BIC/DIUSALES2] [D2], [/BIC/DIUSALESP] [DP] WHERE [F].[KEY_IUSALES1] = [D1].[DIMID] AND [F].[KEY_IUSALES2] = [D2].[DIMID] AND [F].[KEY_IUSALESP] = [DP].[DIMID] AND [DP].[SID_0CHNGID] = 0 AND ( [F].[KEY_IUSALESP] = 0 OR [F].[KEY_IUSALESP] = 2 ) AND [DP].[SID_0REQUID] BETWEEN 0 AND 40 GROUP BY [D1].[SID_IUCITY], [D2].[SID_IUSALPER]


Aggregate Roll-Up

INSERT INTO [/BIC/F100011] SELECT [D1].[SID_IUCITY] AS [KEY_1000111], [D3].[SID_IUPROD] AS [KEY_1000112], 7 AS [KEY_100011P], SUM ([F].[/BIC/IUPROFIT]), SUM ([F].[/BIC/IUQUAN]), COUNT(*) AS [FACTCOUNT] FROM [/BIC/FIUSALES] [F], [/BIC/DIUSALES1] [D1], [/BIC/DIUSALES3] [D3], [/BIC/DIUSALESP] [DP] WHERE [F].[KEY_IUSALES1] = [D1].[DIMID] AND [F].[KEY_IUSALES3] = [D3].[DIMID] AND [F].[KEY_IUSALESP] = [DP].[DIMID] AND [DP].[SID_0CHNGID] = 0 AND [F].[KEY_IUSALESP] = 5 AND [DP].[SID_0REQUID] = 498 GROUP BY [D1].[SID_IUCITY], [D3].[SID_IUPROD]


Infocube Query Example: Infocube IUSALES

(1) Fact Table

(2) Dimensions

(3) Characteristics (simplified)

month

year

day

city regioncountry

product product group

sales person

division

distribution channel

sales organization


Query Example & Processing (under Oracle)

month

year= [98-99]

regioncountry = 'US'

product group

(1) Fact Table

(2) Dimensions



Step 1: Restrictions Master Data Dimensions

month

year= [98-99]


product group

(1) Fact Table

(2) Dimensions


Typical Query Processing


Step 2: Restrictions Dimensions Fact Table

product group

bitmap indexbitmap index

(1) Fact Table

(2) Dimensions




Step 3: Assemble Result

product group

(1) Fact Table

(2) Dimensions


small subset of facttable


month

year= [98-99]



Query Example (1) – simple

SELECT "DT"."SID_0CALMONTH" AS "S____081" ,"DT"."SID_0CALYEAR" AS "S____083" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"D3"."SID_IUPRODGRP" AS "S____524" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN" FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALES3" "D3" , "/BIC/DIUSALESP" "DP" WHERE "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALES3" = "D3"."DIMID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND ( "DT"."SID_0CALMONTH" = 200007 AND "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 ) GROUP BY "DT"."SID_0CALMONTH", "DT"."SID_0CALYEAR", "D1"."SID_IUCOUNTRY", "D3"."SID_IUPRODGRP"


Query Example (2) – navigational attribute

SELECT "DT"."SID_0CALMONTH" AS "S____081" ,"DT"."SID_0CALYEAR" AS "S____083" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"X1"."S__IUCOLOR" AS "S____530" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN"FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALES3" "D3" , "/BIC/XIUPROD" "X1" , "/BIC/DIUSALESP" "DP"WHERE "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALES3" = "D3"."DIMID" AND "D3"."SID_IUPROD" = "X1"."SID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND ( "DT"."SID_0CALMONTH" = 200007 AND "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 AND "X1"."OBJVERS" = 'A' )GROUP BY "DT"."SID_0CALMONTH", "DT"."SID_0CALYEAR", "D1"."SID_IUCOUNTRY", "X1"."S__IUCOLOR"


Query Example (3) – external hierarchy

SELECT "DT"."SID_0CALYEAR" AS "S____083" ,"DT"."SID_0CALMONTH" AS "S____081" ,"D1"."SID_IUCOUNTRY" AS "S____520" ,"H1"."PRED" AS "S____524" , COUNT( * ) AS "1ROWCOUNT" , SUM ( "F"."/BIC/IUPROFIT" ) AS "IUPROFIT" , SUM ( "F"."/BIC/IUQUAN" ) AS "IUQUAN"FROM "/BIC/FIUSALES" "F" , "/BIC/DIUSALES3" "D3" , "/BIC/DIUSALEST" "DT" , "/BIC/DIUSALES1" "D1" , "/BIC/DIUSALESP" "DP" , "/BI0/0300148611" "H1" /* This is a (UNION) view! */WHERE "F"."KEY_IUSALES3" = "D3"."DIMID" AND "F"."KEY_IUSALEST" = "DT"."DIMID" AND "F"."KEY_IUSALES1" = "D1"."DIMID" AND "F"."KEY_IUSALESP" = "DP"."DIMID" AND "D3"."SID_IUPRODGRP" = "H1"."SUCC" AND ( "DT"."SID_0CALYEAR" = 2000 AND "DP"."SID_0REQUID" <= 745 AND "H1"."SUCC" <> 2000008999 )GROUP BY "H1"."PRED", "DT"."SID_0CALYEAR", "DT"."SID_0CALMONTH", "D1"."SID_IUCOUNTRY"


Examples of Conceptual Modeling

in SAP BW


Examples

Reveal why pure RDBMS technology ...

sometimes requires an additional conceptual layer on top,

is not sufficient is some cases,

has no chance in some situations because it has to be more general than necessary.

Examples

example 1: infoproviders in SAP BW uniform view on differing physical layouts

example 2: non-cumulative key figures in SAP BW semantic relationship between table columns

example 3: aggregates in SAP BW could be implemented by using materialized views (or equivalent) but they have proved to be inferior


Example 1: Infoprovider (1)

An infoprovider in SAP BW ...

comprises a reporting scenario,

is the entity on which a query is defined,

combines (aggregated or non-aggregated) operational data with master data (e.g. product, customer, ... data),

or constitutes a master data entity


Example 1: Infoprovider (2) – Examples

Example A:

a cube is an infoprovider

fact table holds operational data on certain granularity

dimensions hold master data

Example B:

customer master data can be an infoprovider

same UI as for other infoproviders

selections, projections, summaries using attributes (e.g. address, customer category, region, ...)


Example 1: Infoprovider (3) -- Overview (SAP BW 3.x)

Infoprovider

Multi-provider

UNION

Infoset

JOIN

Infocube

multi-dim.

analytical

sales cube

ODS-Object

flat

operational

POS data

Characteristic(master data)

flat

master data

product data

Virtual Infoprovider

APIe.g. remote

access

real time


Example 2: Non-Cumulative Key Figures (1)

also: "semi-additive measures"

example: account balance

conceptually:

physically:

account day balance delta

A 29-Sep 100 € 10 €

B 29-Sep 500 €

A 30-Sep 110 €

B 30-Sep 500 € - 100 €

A 1-Oct 110 €

B 1-Oct 400 €

A 2-Oct 110 €

B 2-Oct 400 €

A 3-Oct 110 € - 60 €

B 3-Oct 400 €

A 4-Oct 50 €

B 4-Oct 400 €

account day ref point delta

A 29-Sep no 10 €

B 30-Sep no - 100 €

A 3-Oct no - 60 €

A 4-Oct yes 50 €

B 4-Oct yes 400 €


Example 2: Non-Cumulative Key Figures (2)

non-cumulative key figures / semi-additive measures balance can be reconstructed for any moment in the past

that information has not to be physically stored advantages

significantly reduced data volumes better performance more flexibility

however: algorithms are required for reconstruction read insertion load


Example 3: Aggregates in SAP BW

SAP BW constraints:

only SUM, MIN, MAX aggregations are materialized

uploaded data (in an infocube) can still be identified

delta roll-ups are simple

Materialized or Indexed Views / Automatic Summary Tables

could be used in theory

however: maintenance is considerably slower

... due to expensive tracking and logging mechanisms that are necessary if the general case has to be covered


Summary


Summary

brief introduction to SAP BW

three examples:1. an additional conceptual layer on top of the relational one

2. a semantical pattern that is frequently used in business

3. an object that might suffer from the generic approach

Do the examples reveal shortcomings of RDBMS or are they application domain specific ?

BW Basic Architecture Klaus Majenz SAP – Product Line BI.

Documents

Transcript of BW Basic Architecture Klaus Majenz SAP – Product Line BI.