Lessons for Using SAP Data Services to Build a Data Mart That Rapidly Delivers Consistent...

8/11/2019 Lessons for Using SAP Data Services to Build a Data Mart That Rapidly Delivers Consistent Information to Reports, Dashboards, And Mobile Devices

1/44

Copyright 2013Wellesley Information Services, Inc.

All rig hts reserved .

Lessons for Using SAPData Services to Build aData Mart That RapidlyDelivers ConsistentInformation to Reports,Dashboards, and MobileDevices

Don LodenDecision First Technologies


2/44

1

In This Session

We will discuss an end-to-end solution of design, development,

and delivery for a Business Intelligence (BI) data model where

content is delivered to both a laptop dashboard and iPad The focus will be on:

Back-end data design for optimum storage and delivery

Front-end user consumption components Throughout this process, maintaining absolute data

synchronization is at the forefront of criticality while enriching

end-user experience and t imeliness of the data follow suit


3/44

2

What Well Cover

Key learning points

Design approach

Data sources, development, and synchronization

Presentation of dashboard

The security model integration

Best practices for implementation

Wrap-up


4/44

3

Learning Points

Architect data delivery mechanisms to ensure compatibility and

precision between two different user interfaces:

SAP BusinessObjects Dashboards (formerly Xcelsius)

Mobile iPad application

Combine the following:

SAP BusinessObjects Enterprise (BOE) Web service data Complex user security and multi-variable calculations

Provide drill-through functionality with stacked column bar

charts in SAP BusinessObjects Dashboards Use advanced features of universe design, SAP BusinessObjects

Web Intelligence, BOE security to apply row-level security of data

against hierarchical user data


5/44

4

What Well Cover

Key learning points

Design approach





Wrap-up


6/44

5

Design Approach (High Level)

Deliver the sales force sales data from merging a manufacturing

ERP and third-party pricing data to a dimensional data model to

be consumed by SAP BusinessObjects Dashboards Collapse the dimensional data model down and be able to

simplistically store the same data on an iPad

Be able to deliver a fini te list of products to recommend based onspecific business rules


7/44

6

Design Approach (Recommended Products

Mockup)

Recommend Products Report

Semi-Cartesian product of: Product Type, Style, Ranking,

Brand, Back, Price, New Product Indicator, Weight, Color,Density, Texture, Pattern

Summed the Sales Dollars

Sorted based on mandated criteria


8/44

7

Design Approach (Source System Analysis)

Use the Data Services Profi ler to analyze the ERP products, sales,

customers, sales force hierarchy (Source System Analysis [SSA])

Also look into the new Pricing system

Needed to ensure 1:1 match from ERP into Pricing

The Data Services Profiler provides such information as the

number of distinct values, min values, max values, patterns, nulls,and zeros

Output from the SSA provides guidance for final data model

design


9/44

8

Design Approach iPad Dashboard Data

Model Design

Consolidated data model (non-star) to three tables:

Dealer Sales

Recommended Products

Benchmark Inputs (estimates of expected purchases from that

customer)


10/44

9

Design Approach Laptop Dashboard Data

Model Design

Dimensional data model for the report ing data mart

Conducive to dashboard design for dropdowns, prompts, and

inputs (distinct lists from dimensions) Star schema (fact[s] in the center with surrogate keys to

dimensions)

3 Fact Tables: Transactions (sales, prices, etc.)

14 Dimensions: Means to slice (customers, products, etc.)

Util ize the following SAP BusinessObjects technologies:

Universe

Web Intelligence Web Services (BI Services and QaaWS)

Dashboards

Data Services


11/44

10

See SAP Data Services Profil ing in Action!


12/44

11

What Well Cover

Key learning points

Design approach





Wrap-up


13/44

12

Data Sources

Manufacturing ERP

Sales Force

Sales (stated invoices)

Customers (dealers)

Products

Entitlements (who can buy what and lowest available prices)

Pricing System

Agreed upon pricing for customers and products

Different than enti tlements (above) Active Directory

To look up salesman user names and enforce row-level security


14/44

13

Development Process

Dual-layer approach for data delivery to iPad model and

Dashboard model:

Source data, perform transformations along the way, generatesurrogate keys, and place it into a STAGE environment

Heavy lift ing done in the STAGE build

Intense processing and time consuming to rebuild Allowed back-end ETL development to occur and not impact

the Data Mart

Changes to the Data Mart can be pushed out at night

From there, provide data to DATA MART environment

(replication of STAGE tables)

Simultaneously, consolidate the STAGE tables into the three

needed for the iPad model


15/44

14

Development Change Data Capture

Needed a way to determine changes in data

iPad supported three data tables

And one Notification table

Rudimentary Type II slowly changing dimensions in data

warehousing

Needed a way to generate notifications amongst data This record is new, generate a notification for ADD

This record changed, generate a notification for CHANGE

This record isnt in the input dataset, generate a DELETE


16/44

15

Development Change Data Capture (cont.)

Wanted to leverage the transforms within Data Services:

Table Comparison transform: Looks into a table and

determines if a record has changed

If the record that the Table Comparison is looking for is not

found, the input record is marked as INSERT

If the record that the Table Comparison is looking for isfound, the Table Comparison begins to look to see if

anythings changed

If there is no change, the record is discarded

If there is a change, the record is flagged as UPDATE

If the record is not present on the input, but does exist in the

target, the record (in the target) is marked DELETE


17/44

16


Delete all data from the Notification Table

Well see why shortly

The source STAGE tables come together in the Query transform(join on SKEYs, fi lter, etc.)

The Table Comparison looks into the History table:

First execution, the History table is empty (no history) Subsequent executions, there is a historical reference to

compare against


18/44

17


If the record is not found in the History table, its f lagged as

INSERT

A copy also goes down to the Map_Operation transform The Map_Operation transform manipulates these INSERT,

UPDATE, and DELETE statements


19/44

18


If a record is found in the History table, the Table_Comparison

begins to compare the data values (price, style, etc.)

If theres no change, the record is discarded If there is a change, the record is:

UPDATED in the History table (making the History table be

current) The Map_Operation changes the UPDATE to INSERT

(because the Notif ication table is empty each execution)


20/44

19


Each newly inserted record into the Notification table also has the

change code (ADD, CHANGE, DELETE) on the record:

Now we can publish these notifications to the iPad with the data

Op Code Val1 Val2 Val3

ADD Y0 CLR Y

CHANGE Y1 C2 N

CHANGE Y9 BLU NDELETE Y11 RD Y


21/44

20


The data needed to be delivered(and the notif ication) to the iPad

by a certain time each day (5 am EST)


22/44

21

Walk Through the SAP Data Services ETL Code


23/44

22

Development Performance

Large data sets

5M products

450K customers

80M enti tlements (semi-Cartesian product developed)

8M recommended products

Not all of this data will f it on an iPad

Devise sorting and ranking to pick top qualities of products

to recommend for a salesmans customers


24/44

23

Development Performance (cont.)

Make use of Data Services database pushdown capabilities

The database engine can crank through a local data set better

than the ETL engine pulling that set across the network Database I/O is the biggest bottleneck

Ensure use of physical hardware

For ETL engine For database servers (MS SQL 2008R2)

Ensure fresh indexes (as needed)

Index rebuild tasks Can be performed by database server job or in SAP


25/44

24

Development Flow Diagram


26/44

25

Ensuring Data Synchronization

Ensure the iPad model (three tables) matches that of the Data Mart

Since both were built off the same data set (STAGE), place

audits on both models to ensure sales figures and prices werein alignment

Could be done with SAP Data Services Auditing or manually

Manual tables and processes provide more control Do not interfere with performance

Do not update data in the event that there was a mismatch (use

stale data)


27/44

26

What Well Cover

Key learning points

Design approach




Best practices for implementation Wrap-up


28/44

27

Presentation of the Dashboard Opening

Screen


29/44

28

Presentation of the Dashboard

Sales by Construction


30/44

29

Presentation of the Dashboard (cont.)

Some special considerations to support migration between

environments

Ensure to use Lifecycle Manager to manage components withinSAP BusinessObjects Enterprise (BOE)

This wil l maintain the CUIDs for objects


31/44

30

What Well Cover

Key learning points

Design approach






32/44

31

The Security Model Integration

Could pull the salesman ID (e.g., a number) out of the ERP

Lookup to MS Active Directory

Use stored procedure within MS SQL Server called from SAPData Services

Leverage single sign-on (SSO) within SAP BusinessObjects (for

laptop dashboard)


33/44

32

The Security Model Integration (cont.)

Security handled for the iPad elsewhere

Need to provide the user name for the salesman within each of

the iPad tables (for fi ltering) Need to restrict who can see what (e.g., one salesman should not

be able to see anothers sales/pricing, etc.)

Leverage row-level security model Inner join amongst data model

Dimensions held sales force hierarchy (salesman reports to

regional manager who reports to district manager, etc.)

NET_ID = @Variable(BOUSER)


34/44

33

What Well Cover

Key learning points

Design approach






35/44

34

Best Practices for Implementation

Align business requirements with tool capabilities and reduce

complexity in dashboard design

Optimize integration of data sources and data model to enhancedashboard performance

Flattening data to make all of the data available with one read

Aggregating the data to provide a very high level of granularityfor presentation in a dashboard

Data delivery mechanism must ensure compatibility and precision

between various different user interfaces

Audits are important!

Use advanced features of universe design, WebI, and BOE

Administration to apply row-level security of data against

hierarchical user populations


36/44

35

What Well Cover

Key learning points

Design approach






37/44

36

Wrap-Up

The data model(s) should drive the technology being used

SAP BusinessObjects universes work best on a star schema

Consolidating that model into a few tables to be stored on amobile device, the iPad

When dealing with large data sets, ensure that database

pushdown from the ETL is maximized Separate back-end and front-end development tasks (if possible)

Helps to not pull the rug out from underneath the front end

during data reloads

If possible, build in the security model such that a user of such

secure data wouldnt need another housekeeping task of

managing a user name and password

Leverage the use of MS Active Directory and SSO within BOE


38/44

37

Where to Find More Information

http://scn.sap.com/docs/DOC-26314

Bashir Arwan, BusinessObjects 4.0 Single Sign On (SSO) with

WIN AD (SAP Community Network, 2012). Configuration of SSO for BOE

http://help.sap.com/businessobject/product_guides/boexir3/en/xi3

_ds_performance_optimization_guide_en.pdf Data Services Performance Optimization Guide (Business

Objects, 2008).

Documentation with Data Services to ensure that data flows

are achieving the best/most database pushdown


39/44

38

Where to Find More Information (cont.)

http://blog.tech-cats.com/2007/09/querying-active-directory-

through-sql.html

Boyan Kostadinov, Querying Active Directory Through SQLServer (Aaquagua! Blog, 2007).

Technical configuration of MS SQL Server to query AD

https://scn.sap.com/message/9875562 OpenDocument functionality for launching Web Intelligence

reports from dashboards thread on SAP Community Network


40/44

39

7 Key Points to Take Home

Back-End Considerations

Ensure as much database pushdown as possible (for

performance) Make use of default records in all dimension tables

Avoid expensive outer joins

Make every query an inner join! Provide the same base model, STAGE, for both the dashboards

reporting (the Data Mart, which was a replication of the STAGE

model) and the iPads delivery (three consolidated tables)

Single sign-on (SSO) can remove the headache with

maintaining SAP BusinessObjects Enterprise users


41/44

40

7 Key Points to Take Home (cont.)

Front-End Considerations

Push down data aggregation, calculations, formulas to simplify

code, isolate integration components, and facilitate ease ofmaintenance

Excel logic Web Services logic Web Intelligence

Universe Database

Minimize dashboard navigation, dynamic visibi lity, and drill-

through to enhance usabil ity and reduce overall integration

complexity

When conducting Quality Assurance and User AcceptanceTesting, Lifecycle Manager (LCM) effectively migrates SAP

BusinessObjects Enterprise dashboard components by

maintaining adherence to object CUIDs between environments


42/44

41

Your Turn!

How to contact me:

Don [email protected]

@donloden

Please remember to complete your session evaluation


43/44

42

Disclaimer

SAP, R/3, mySAP, mySAP.com, SAP NetWeaver, Duet, PartnerEdge, and other SAP products and services mentioned herein as well as their

respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and

service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP.

Lessons for Using SAP Data Services to Build a Data Mart That Rapidly Delivers Consistent...

Documents

Transcript of Lessons for Using SAP Data Services to Build a Data Mart That Rapidly Delivers Consistent...