Berg_NW2010_EDW_v4

8/4/2019 Berg_NW2010_EDW_v4

1/50

20010 Wellesley Information Services. All rights reserved.

8/4/2019 Berg_NW2010_EDW_v4

2/50

20010 Wellesley Information Services. All rights reserved.

What you need to know to

get the most out of usingSAP NetWeaver BW as yourenterprise data warehouse

Dr. BjarneBerg

8/4/2019 Berg_NW2010_EDW_v4

3/50

3

What Well Cover

Introduction

The EDW architectural options Federated Data Warehouse Centralized Data Warehouse

Distributed Data Warehouse

Data Integration challenges Masterdata

Transaction data conversion

Data cleansing

Designing for Flexibility

The Support Organization

The top 10 EDW pitfalls

8/4/2019 Berg_NW2010_EDW_v4

4/50

4

We will take a detailed look at the pros and cons of your EDW

architectural options, including federated, centralized, anddistributed EDW models, and explore when each approachis appropriate.

Learn how to interface The Support Organization and how to

consolidate different master and transactional data.Weigh your options for building a centralized or a

decentralized EDW support organization.Examine the top 10 pitfalls companies face when

implementing SAP NetWeaver BW as their EDW and how toovercome them.

.n th is se ssio n

8/4/2019 Berg_NW2010_EDW_v4

5/50

5

A Quick Definition: BI Vs. Data Warehousing

Data warehousing is the act of

extracting, transferring, transforming,storing and retrieval of data forreporting and analytical purposes.

Business Intelligence (BI) is aterminology for applications that usesdata stores for analytical purposes.

BI applications are not required to runon top of data warehouses, but the

majority does

BI applications are not required to runon top of data warehouses, but the

majority does

8/4/2019 Berg_NW2010_EDW_v4

6/50

6

Introduction

The EDW architectural options Federated Data Warehouse Centralized Data Warehouse Distributed Data Warehouse

Data Integration challenges Masterdata Transaction data conversion

Data cleansingDesigning for FlexibilityThe Support OrganizationThe top 10 EDW pitfallsWra -u

8/4/2019 Berg_NW2010_EDW_v4

7/50

A Logical Enterprise DW Architecture

Metadata

DataExtractio

nIntegrati

onand

CleansingProcesses

CustomDeveloped

Applications

DataMining

Statistical

Programs

QueryAccessTools

Data Resource Management and Quality Assurance

Summarize

dData

SegmentedData

Subsets

FunctionalArea

Summation

Marketingand Sales

Purchasin

g

CorporateInformati

on

ProductLine

Location

Purchasing

Systems

Invoicing

Systems

GeneralLedger

ExternalData

Sources

OtherInternalSystems

Translate

Attribute

Calculate

Derive

Summarize

Synchronize

SourceData

Extract

OperationalData Store

Transform

DataWarehouse BI

Applications

Source: Bjarne Berg, Introduction to Data Warehousing,1997

8/4/2019 Berg_NW2010_EDW_v4

8/50

8

Metadata

SAPBOBJOLAP

Universes

-Ad HocWebi

OLAPPioneer

Dashboards

XcelciusBatch

reportsCrystal


Security

Training

UserSupport

Projects

-Ad hoc

Synchronization

IT DevelopedSemantic

Layer

IT Support&

Development

BusinessDriven BI

Applications

IT DrivenData

Warehouses

BExExplorer

SAP BWA

Data

Warehouse(s)

DataWarehouse(s)

DW ODSs

DW

-Starschemas

SAP BW(s)SAP BW(s)

SAP DSOs

SAPBW

InfoCubes

SAPBOBJ

SQLUnivers

esDirect

Connections

Customand

3rdparty

SAPBOBJData

Service

s BPC

ExternalApplicati

ons

FinancialReport

center

EnterprisePortal

SalesReport

centerManufactu

ringReport

centerHR

Reportcenter

Partnerfacing

Reportcenter

-Ad HocReport

center

Customerfacing

Reportcenter

Users

Employees

Customers

Partners

8/4/2019 Berg_NW2010_EDW_v4

9/50

9

Federated Data Warehouses are best in very large

organization where development is separated bygeography, organizational boundaries, or wheremultiple data warehouses exists due to mergers& acquisitions.

To make FDWs successful, there needs to be arapid convergence to standardized technologies.This include:

Same type of databases and support pack levels(costs and compatibility) Same technical platforms Hardware, Backups and

Archiving (costs) Shared Portal and user interface strategy (reduced

training and support) Shared security design and centralized

administration risk mana ement

If the data is federated you gain faster response time to business needs,can execute multiple projects in parallel, and work 24/7 across theglobe. But without any standardization, it can also be very costly.

8/4/2019 Berg_NW2010_EDW_v4

10/50

10

Metadata

SAPBOBJOLAP

Universes

-Ad HocWebi

OLAPPioneer

Dashboards

XcelciusBatch

reportsCrystal


Security

Training

UserSupport

Projects

-Ad hoc

Synchronization


Layer

IT Support&

Development

BusinessDriven BI

Applications

IT DrivenData

Warehouses

BExExplorer

SAP BWA

SAP BWSAP BW

SAP DSOs

SAPBW

InfoCubes

SAPBOBJ

SQLUnivers

es

DirectConnect

ions

Customand

3rdparty

SAPBOBJData

Services

BPC

ExternalApplicati

ons

FinancialReport

center

EnterprisePortal

SalesReport

centerManufactu

ringReport

centerHR

Reportcenter

Partnerfacing

Reportcenter

-Ad HocReport

center

Customerfacing

Reportcenter

Users

Employees

Customers

Partners

OLTP sourcesOLTP sources

SAP ECC,Siebel

JDEOracle

Others

8/4/2019 Berg_NW2010_EDW_v4

11/50

11

Centralized Data Warehouses are great for small

and mid-size data warehouses (less than 15-40Tb). There are great benefits in terms of theease to mange upgrades, support packs,enforcing development standards, transportcontrol, master data management and the

overall total cost of ownership

To make CDWs successful, there needs to be: Adequate funding of hardware, application servers,

database servers Serious consideration should be made to move BI and

reporting to BWA Focus on using the database capacity on storage and

data loads-- not queries No direct reporting from DSOs (takes too much system

resources) Broadcasting , caching and performance tuning is a

If the data is centralized it is faster to develop new solutions for the

business and merging from different data sources are easier

8/4/2019 Berg_NW2010_EDW_v4

12/50

12

Metadata

SAPBOBJOLAP

Universes

-Ad HocWebi

OLAPPioneer

Dashboards

XcelciusBatch

reportsCrystal


Security

Training

UserSupport

Projects

-Ad hoc

Synchronization


Layer

IT Support&

Development

BusinessDriven BI

Applications

IT DrivenData

Warehouses

BExExplorer

SAP BWA

SAP BW(s)SAP BW(s)

SAP DSOs

SAPBW

InfoCubes

SAPBOBJ

SQLUnivers

es

DirectConnect

ions

Customand

3rdparty

SAPBOBJData

Services

BPC

ExternalApplicati

ons

FinancialReport

center

EnterprisePortal

SalesReport

centerManufactu

ringReport

centerHR

Reportcenter

Partnerfacing

Reportcenter

-Ad HocReport

center

Customerfacing

Reportcenter

Users

Employees

Customers

Partners

SAP BW(s)SAP BW(s)

SAP DSOs

SAP

BWInfoCubes

8/4/2019 Berg_NW2010_EDW_v4

13/50

13

A Decentralized Data Warehouses makes sense if

there are logical division between business units,geographies and little shared reporting I.e. in aconglomerate organization with diverse businessunits.

The benefits of DDWs include the flexibility of theFDW with the technology standardization andlower cost of ownership of the CDW. To makeDDWs successful, there needs to be:

A formal Masterdata Management (MDM) strategywith clearly defined standards

A rule based data cleaning and data integration planfor centralized reporting

A shared hardware location to keep costs lowerWith DDWs there is a risk of creating stove-pipe data marts that

cannot be integrated at the corporate level without very high costs.

8/4/2019 Berg_NW2010_EDW_v4

14/50

14

F

B

w

bOrganization

8/4/2019 Berg_NW2010_EDW_v4

15/5015

Introduction




8/4/2019 Berg_NW2010_EDW_v4

16/50

1616

For all data warehouses 60-80% of the effort isto move, store, retrieve and integrate data fromvarious source systems.

From a SAP perspective, Informationmanagement is six distinct efforts. Therefore,

several SAP BI tools exists with differentcapabilities

8/4/2019 Berg_NW2010_EDW_v4

17/50

17

The XI Data Services Architecture

Data integration in an EDW can be done with SAP

BOBJ Data Services. The tool architectural can beillustrated in terms of source data, process andtarget data.

TargetTarget

8/4/2019 Berg_NW2010_EDW_v4

18/50

Applications

1.SAP R/3 & ECC ABAP

BAPI

Applications

1.SAP R/3 & ECC ABAP

BAPI

Extraction and data movement may take 30-50%

of the time in a process chain. Therefore, do notplan to build an EDW with slow non-nativeconnectivity to the source systems.

8/4/2019 Berg_NW2010_EDW_v4

19/50

Reconciliation Between Systems

The majority of timespent on maintaining acomplex EDW is thetime spent on reconciliationof the data

You have to prove that the data in thewarehouse is equal to the data you extracted, oryour financial reporting systems will have no

credibility.

You are also legally required to have areconciliation process that can be tracked, if youuse the warehouse for financial reporting toexternal entities.

8/4/2019 Berg_NW2010_EDW_v4

20/50

Reconciliation Between Systems-Dashboards

Many companies invest in developing manualcontrol queries, while others use reconciliationproducts that are powered by SAP NetWeaver

An example of a reconciliation Dashboard builton SAP BW. In this example:

1.A reconciliation memo was written on Feb. 1st 2.PCA reconciliation between BW and R/3 failed

on Feb. 16th

8/4/2019 Berg_NW2010_EDW_v4

21/50

2121

Using BOBJ Data Services you can

consolidate data from many sourcesystems, cleanse and integrate thembefore you send it to SAP BI. This avoidsmulti-nested DSOs and complex load

logic.Source systems- Oracle- JDE- Peoplesoft- Baan

- Siebel- Custom- Hyperion- .Other
http://www.changing-dimensions.com/Cube.jpg/Cube-full;init:.jpg

8/4/2019 Berg_NW2010_EDW_v4

22/50

2222

Interesting use BOBJ Data Services

Using BOBJ Data Services you integrate,

cleanse and merge data from sourcesystems during

1)ECC implementation projects,

2)Retirement of legacy systems,3)Mergers and Acquisitions.Source systems

- Oracle- JDE- Peoplesoft

- Baan- Siebel- Custom- Hyperion- .Other

8/4/2019 Berg_NW2010_EDW_v4

23/50

23

The Data Profile Tab in BOBJ Data ServicesThis tab in the view data screen contains dataprofile statistics on each column that can help youdecide on the quality of the input data.

The system automatically captures the following

statistics in a profile grid.

1. Column Name2. Number of distinct values in a column3. Number of records with a NULL value in this column

4. Maximum & Minimum value of the column

8/4/2019 Berg_NW2010_EDW_v4

24/50

24

The ValidationValidation allows you to create rules forcleaning data prior to loading it to the system.You can have a pass rule and an 'Action onFailure' that can provide complex logic.

8/4/2019 Berg_NW2010_EDW_v4

25/50

25

The AuditThe Auditingselection allows youto take complexactions when thedata quality is poor.

You can:1.Send an email toan administrator

2.Load the data to atable for latercorrection

3.Modify the datathrough scripts

4.Create customfunctions for yourown processing

logic

8/4/2019 Berg_NW2010_EDW_v4

26/50

:a l D a ta C le a n sin g E xa m p le o f E n h a n ce d P a rty M a st

: ,Source SAP AG 2009

ou can also add new items such as geocodes for. .isualization in SAP BI I e mapsou can add newharacteristicso the data such:s

)1 Legal taxjurisdictions

)2 Census track ID)3 Block group ID

)4 Insurance ratingterritories)5 Tax authority name)6 Tax authority FIPS

codes)7 &Longitude Latitude

)8 City type)9 ...

:REAT FEATURE The Census track IDallows you to analyze your customers and

partners using government censusinformation

8/4/2019 Berg_NW2010_EDW_v4

27/50

Universal Data Cleansing: Customer Aggregating & Discovery

common way to look atustomer data is by Households.nstead of single recordsOBJ DQ allows you to look at customer's

,ddresses and create shared master records, ( . .ustomer mapping keys aggregating data i e),ggregated sales data for the household" - " ,heck no call lists examining churn( - ).pparent customer turn overou can also integrating all master data from" "any records into a single super recordhat contains all the unique master data you.ave about a single customer or partner

8/4/2019 Berg_NW2010_EDW_v4

28/50

: &iversal Data Cleansing Data integration BAS

SAP Data Quality Management has pre-delivered content formany solutions including CRM -> ECC integration, including:

1)Across platform search capabilities2)Automated address correction3)De-Duplication of records

4)Direct system connection (no file extraction)5)Supported for all major releases: R/3 4.6c; ECC 5 and 6; CRM 4 and 5

The Business Address Service (BAS) feature can:

1)Use Postal reference files from 190 countries to clean address, includingsuggestion lists

2)Data scans and searches in SAP for duplicate records using partial userinput.

" at a Qu ality M anag emen t f or SAP p rovi des a pre packa ged nati vent eg r a ti o n o f d at a q ua li t y b e s t pr a ct ic e s wi t h in t he S A P

"nv ir o n me n t u si n g th e B OB J Da t a S er v ic es pl at f or m,SAP AG 2009

Wh t i N i BOBJ D t S i

8/4/2019 Berg_NW2010_EDW_v4

29/50

29

What is New in BOBJ Data Services

Expanded matching capabilities to allow the business

user to select other fields (beyond street name andzip code) within the generation of break keys. An improved method to install the functionality of this

product into your IC WebClient or CRM IC WebClient

environment. To do so, you add a Component Usageto the Component to which you want to add PostalValidation.

If you have purchased the geocoding option for this

product, geocoding allows you to return latitude,longitude, and relevant status information for a U.S.address record

8/4/2019 Berg_NW2010_EDW_v4

30/50

30

Introduction




8/4/2019 Berg_NW2010_EDW_v4

31/50

31

In BW 7.2 a new feature called "Semantic

partitioned object" (SPO) is introduced to helppartition InfoCubes for query performance, andDSOs for load performance.

SPOs can be added to MultiProviders for easy

query administration and to mask complexity

Source: SAPAG, 2010

.W 7 2 p ro vid es W iza rd s toelp you p artition o b jects b y,e a r b u sin e ss u n its o r.roducts

W a lso g e n era teu to m a tica lly a ll n e e d e d D T P

u ch a s tra n sfo rm a tio n ru le sn d filters to loa d th e co rrect.n fo P ro vid e r

aintenance is easier since any remodeling .nly need to change the reference structure

8/4/2019 Berg_NW2010_EDW_v4

32/50

32

With BW 7.2, you can have data inBWA, InfoCube are not required.

Once you exceed a few hundredcritical users and/or 3-4 Tb of datayou should seriously consider BWA

Some of SAP reference clients

BWA is no longer exotic.

Many large SAP-BI customershave already implemented BWA& projects are under way inEurope, Asia and the Americas.

WA is becoming mainstream and enhanced- .n BW 7 2

8/4/2019 Berg_NW2010_EDW_v4

33/50

33

IT cannot hold BI hostage with longdelivery times and slow responsesto changing user demands.

The only way to be successful is toprovide flexible data structures andcleansed, integrated data to the

business and let the business groups take over the BIdevelopment.

So what is needed is a stronger emphasis on scalable, fast ITsolutions and a ramp up of BI capabilities of the business units.

Keeping BI front-end solutions such as Webi, Visual Composer andPioneer in the hands of IT instead of the business will createinflexible systems that are unlikely to succeed.

eparate t e Data Ware ouse rom t e BIsolutions

8/4/2019 Berg_NW2010_EDW_v4

34/50

34

Introduction




8/4/2019 Berg_NW2010_EDW_v4

35/50

BI Support Organization Big Picture

You need to separate the operations of BI

systems from the project workIf there is no support organization, the BI systemquickly becomes an orphan when the project ends

Without asupport org. thereis a risk that

future BI projectsare delayed sincethe project teamhas to supportpreviousprojects

8/4/2019 Berg_NW2010_EDW_v4

36/50

The BI Help Desk Level 1 Support

The first level

support shouldbe done byPower Users inthe organization

You will have totrain these

resources,empower themto makechanges, and

leverage them

Query related support tickets from a centrallocation/Web site should be routed to the

power users in each department.

The power user can escalate the ticket to Level-2 support if he/she is unable to resolve it.

8/4/2019 Berg_NW2010_EDW_v4

37/50

The BI Help Desk Level 2 Support

The second level

support is usedfor issues thatare not related toqueries,

presentations,reports, andformatting

This include dataloads,performance,

security,

Some support ticket types are alwaysrouted to Level 2 support.

It is important to have a generic emailaddress for Level 2 support that is notrelated to an individual. Emails to thisaddress should not be deleted.

Break-Fix - Splitting Projects & Support

8/4/2019 Berg_NW2010_EDW_v4

38/50

TrainingProject Stack

Break fix and Production stack

Break-Fix - Splitting Projects & SupportEnvironments

By Introducing a Break-Fix (BWB) environment, the

support team can correct break-fixes and move codeinto the Testing environment (BWQ) and Productionenvironment (BWP) without impacting the projectteam

Trans orts can be ca tured in the buffer and moved

BWD

BWS

BWT

BWB BWQ BWP

The Break-Fix andproduction stack as well

as the trainingenvironment is ownedby the support team.

The project teams ownthe development and

Sandbox environments(BWS and BWD).

8/4/2019 Berg_NW2010_EDW_v4

39/50

39

Introduction




Pitfall #1: Lack of Reasonable SLA with

8/4/2019 Berg_NW2010_EDW_v4

40/50

Pitfall #1: Lack of Reasonable SLA withEDW Support Team

Some examples of reasonable

performance include:

1.90% of all queries run under 20 seconds

2.System is available 98% of the time

3.Data loads are available at 8am 99% of thetime

4.User support tickets are answered within 30minutes(first response)

5.User support tickets are closed within 48hours 95% of the time.

6.System is never unavailable for more than 72hrs including upgrades, service packs,

8/4/2019 Berg_NW2010_EDW_v4

41/50

More EDW Pitfalls.

Pitfall #2: Jack-of-all-trades Master of

none. BI is complex with many different tools and

technologies. Dont rely on a single personwith no specialized skills.

Make each person responsible for a focusedtechnology/task.

Pitfall #3: An army of Architects whodont understand SAP. Have one architect quality is more

important than quantity

Architecture is technical by nature.PowerPoints only gets you a small part of

8/4/2019 Berg_NW2010_EDW_v4

42/50

More EDW Pitfalls.

Pitfall #4: Not separating the Support Team fromthe Project team

Keeping the lights-on is a core focus area. Many EDWs fail because of lack of training,

production and user support, and by havingnobody around to do continuous

improvements.

Pitfall #5: A Firm Belief in Monolithic DataWarehouses Google runs on over 500,000 servers, why

must your data warehouse run on one? Divide and concur when the performance

becomes a too-large problem. Separate BI onto SAP BWA and use the data

warehouse for data movement and data

8/4/2019 Berg_NW2010_EDW_v4

43/50

More EDW Pitfalls.

Pitfall #6: Analysis Paralysis.

You will never have perfect EDWrequirements get over it. The business will change and so will the BI

system. Change is a sign of success notfailures (people who cares wants to make itbetter).

Not moving forward and keep analyzing is acostly decision

Pitfall #7: A Single User Interface will solve allmy EDW problems..

There are no magic bullets. Most companiesneed 2-3 end user tools.

Start with OLAP (Pioneer) web, then continue

8/4/2019 Berg_NW2010_EDW_v4

44/50

More EDW Pitfalls.

Pitfall #8: Enforce EDW Standards

Standards are not a word document buried ina file cabinet If you allow exceptions the standards

quickly become meaningless.

It costs to keep your house clean, but datamanagement and data integration willbenefit greatly from it. Remember: the road tohell is paved with good intentions - unknown.

Pitfall #9: Keep Your EDW Support Teammotivated

The average application developer stays onthe job for 47 months, the average support

person is only there for 25 months!

i l i f ll

8/4/2019 Berg_NW2010_EDW_v4

45/50

Final EDW Pitfall.

Pitfall #10: Not Creating a BI Technology Advisory

Board for the EDWUse ad-hoc best practice advise from

external experts on an periodic basis.

If you are struggling with something, thereare many others who have cracked thenut already leverage their experiences.

Attend BI conferences, take good notes andleverage the many experts at the booths,

the speakers and the forums.

You are not alone, but your team needs toget plugged into the many ASUG, BIExpert, SDN and SAP BI communities.

8/4/2019 Berg_NW2010_EDW_v4

46/50

46

Introduction




Resources

8/4/2019 Berg_NW2010_EDW_v4

47/50

47

Resources

Support Organizations - ppt

download with more details http://www.comeritinc.com/Downloads.

htm

Implementing Enterprise DataWarehousing: A Guide forExecutives

by Alan Schlukbier

Efficient SAP NetWeaver BIImplementation and Project

Management

7 Key Points to Take Home

8/4/2019 Berg_NW2010_EDW_v4

48/50

48

7 Key Points to Take Home

There are more than one way to architect an

EDW. However, you need to make sure your BIsolution is designed, not evolutionary

Consider FDW and DDWs when data volumesare extremely high or your company just

underwent a merger or acquisitionMake the front-end independent from the

backend

Formalize a data integration strategy with MDMand Reconsolidation as key focus areas

Invest in people, not just technology Greatsupport staff is key to EDW success

SAP BWA should be part of your EDW strategy

Your Turn!

8/4/2019 Berg_NW2010_EDW_v4

49/50

49

Your Turn!

:o w to co n ta ct m e.r B ja rn e B e rg.b e rg @ C o m e ritIn c co m

Disclaimer

8/4/2019 Berg_NW2010_EDW_v4

50/50

Disclaimer

SAP, R/3, mySAP, mySAP.com, SAP NetWeaver, Duet, PartnerEdge, and other SAPproducts and services mentioned herein as well as their respective logos are

trademarks or registered trademarks of SAP AG in Germany and in several othercountries all over the world. All other product and service names mentioned are thetrademarks of their respective companies. Wellesley Information Services is neitherowned nor controlled by SAP.

Berg_NW2010_EDW_v4

Documents

Transcript of Berg_NW2010_EDW_v4