20 Tips and Tricks to Improve Data Load Performance.ppt

7/22/2019 20 Tips and Tricks to Improve Data Load Performance.ppt

1/57

2012 Wellesley Information Services. All rights reserved.

20 Tips and Tricks

to Improve DataLoad Performance

Jesper ChristensenCOMERIT


2/57

In This Session

Gain insight into SAP NetWeaverBW data load processes, how

they work, and what tools are available to monitor and optimizetheir performance

Receive best practices to maximize data load performance while

reducing long-term maintenance costs

Understand the benefits of optimized data load processes Find out how to enable version history to track code changes and

how to create reusable ETL logic to improve throughput and

reduce data load time

Get tips on when and how to use customer exits in DataSourcesand variables to manage risk and reduce maintenance costs

Identify the challenges and benefits of semantic partitioning and

the importance of efficient data models

1


3/57

2

What Well Cover

Loading data in SAP NetWeaver BW

Finding performance bottlenecks Optimizing the database

Optimizing the ABAP code

Optimizing the data models

Optimizing the data updates

Wrap-up


4/57

SAP NetWeaver BW Data Load Processing Overview

SAP NetWeaver BW data load processing consists of three main

activities: Extraction = Collecting the data in the source systems and

preparing it before sending it to SAP NetWeaver BW

Transformation = Transforming the data using routines,

lookups, formulas, etc. Load = Updating the data into InfoProviders DataStore Objects

(DSOs), cubes, and master data

3


5/57

Dataflow in SAP NetWeaver BW

4Source: SAP


6/57

Extraction Interface Types

5

Source: help.sap.com


7/576

DataSources Supported by SAP NetWeaver Extraction

SAP NetWeaver BW Service API

Allows data from SAP systems in standardized form to beextracted and accessed directly

These can be SAP application systems or SAP NetWeaver

BW systems

File interface The file interface permits the extraction from and direct access

to files, such as csv files

Web services

Permit you to send data to the SAP NetWeaver BW systemunder external control


8/577

DataSources Supported by SAP NetWeaver Extraction(cont.)

Universal Data (UD) Connect

Permits the extraction from and direct access to relational data Database (DB) Connect

Permits the extraction from and direct access to data located in

tables or views of a database management system

Staging Business Application Programming Interfaces (BAPIs)

Open interfaces that SAP BusinessObjects DataServices and

certified third-party tools can use to extract data from older

systems


9/578

Extraction Time Can Be Split into Two Categories

Extraction time

DB time to select the data to be extracted Logic applied during extraction such as joins, lookups, and

filtering

Middleware and network time

The time used to transfer the data from the source system tothe target SAP NetWeaver BW system

Interface types such as Web services and Universal Data (UD)

Connect are good for small amounts of data and cannot handle

large volumes Fixed format files are larger to transfer but faster to load into

SAP NetWeaver BW

WAN Network time can become a bottleneck during peak hours


10/579

Transformation Types

SAP NetWeaver BW supports the 3.x and the 7.x versions of

transforming the data 3.x is using Transfer rules and Update rules

Two steps of logic to process the dataset

Loads to different targets must be processed together

Used to have better performance than transformations

Old method; no more development or performance

enhancements; do not continue to use

7.x is using transformations

Is using a single step of logic to process the dataset

Loads to different targets can be processed independently

Better performance

Always use this option for new development


11/5710

Loading Data to Information Providers Types

Loading of the data to InfoProviders differs depending on type

DSO Update of the activation queue

Activation of data (update of active table and changelog)

SID determination

Should in general be switched off for DSOs

Master data

Update of master data tables

SID determination

Check duplicate key values

Very time consuming for time-dependent attributes

Attribute change run to activate the master data

Generate navigation data


12/57

Loading Data to Information Providers Types (cont.)

Loading of the data to InfoProviders differs depending on type

(cont.) Cubes

Update of data to the InfoCube star schema

SID determination

Roll up data to aggregates

Update data to SAP NetWeaver BW Accelerator (SAP

NetWeaver BWA)

Performance considerations for loading the data

Ensure that the database parameters are in place

Implement the correct SAP NetWeaver BW settings for your

InfoProviders

11


13/5712

What Well Cover






Wrap-up


14/57

Tip 01: SAP NetWeaver BW 7.x Statistics

SAP NetWeaver BW includes a great statistics tool

It collects information on most SAP NetWeaver BW-specificactivity

Such as data loads and queries

Its delivered as business content

So you must activate it just like all business content

How to Activate Admin Cockpit document on help.sap.com

http://help.sap.com/saphelp_nw04s/helpdata/en/46/f9bd550d4

0537de10000000a1553f6/frameset.htm


15/57

Tip 01: SAP NetWeaver BW 7.x Statistics (cont.)

Define standard measure that can

be monitored on a daily, weekly,and monthly basis to evaluate

data load performance trends

Records processed per minute

or Time to process 1 million

records

Time spent on extraction

Time spent in transformations

Top 10 long running loads Total time spent for Attribute

and Hierarchy change runs

Use the standard queries and

reports as a starting point 14


16/57

Tip 02: See Details About Performance in the Monitor

The load monitor transaction code RSMO gives more details

about the processing steps InfoPackage details

Data Transfer Process (DTP) details

15


17/57

16

Tip 03: Use SE30 to Test Performance

Transaction code SE30 ABAP Runtime Analysis gives a detailedview of performance

Remember to set

the accuracy toLow

Run

transaction

code RSA3

Note: SE30 can also be used for transformations by simulating the DTP run


18/57

Tip 03: Use SE30 to Test Performance (cont.)

Detailed Runtime will show you the bottlenecks

Sort descending based on

Net Time and you will see

your bottleneck on the top


19/57

18

What Well Cover






Wrap-up


20/57

Tip 04: Implement the Correct DB Parameters

Key DB parameters

SAP has recommended some parameter values for SAPNetWeaver BW that usually improve performance

Expect to evaluate these parameter settings frequently, though,

to ensure that the DB operates optimally

See three key SAP Notes: 830576Parameter recommendations for Oracle 10g

387946Use of locally managed tablespaces for

BW systems

1044441Basis parameterization for NW 7.0 BI systems

19


21/57

20

Tip 05: Manage Database Statistics

DB statistics are also crucial forSAP NetWeaver BW performance

The DB will not know the mostoptimal execution path for anSQL statement without DBstatistics

To set up DB statistics: Set up BRCONNECT job

using DB20 to recalculate DBstatistics

Use program RSANAORA toanalyze specific tables

DB statistics can run very slowly under Oracle when

you use SAP NetWeaver BW programs or DB

statistics. Make sure you use BRCONNECT.


22/57


23/57

22

What Well Cover






Wrap-up


24/57

Tip 07: Coding Tips Dynamic Calls

Code the extractor user exits so that they call a dynamic program

per DataSource Isolate the code per DataSource in a self-contained program

Minimize risk that a syntax error in code for one DataSource

impacts extraction from all other DataSources

Example Program name = ZBW +

Form name = DOZBW +

This same technique can be used with customer exit variable

code

23


25/57

24

Tip 07: Coding Tips Dynamic Calls (cont.)

Illustration: Sample dynamic program call


26/57

Tip 08: Coding Tips Field Symbols

Performance consideration: Where possible, use field symbols to

populate fields in the data package The move costs of a LOOP ... INTO statement depend on the

size of a table line

The larger the line size, the longer the move will take

By applying a LOOP... ASSIGNING statement you can attach afield symbol to the table lines and operate directly on the line

contents

This is a much faster way to access the internal table lines

without moving their contents

25


27/57

26

Tip 08: User Exit Field Symbols

Illustration: Sample use of field symbols

User Exit (without field-symbols)

REPORT YBWZDS_AGR_USER.*****************************************************************

* Form called dynamically must start with DOYBW + *****************************************************************

FORM DOYBWZDS_AGR_USER

TABLES C_T_DATA STRUCTURE ZOXBWD0001.

data: l_logsys type logsys.

l_s_data like ZOXBWD0001.

select single logsys from t000

into l_logsys

where mandt = sy-mandt.

loop at c_t_data into l_s_data.

l_s_data-load_dt = sy-datum.

l_s_data-logsys = l_logsys.

modify c_t_data from l_s_data index sy-tabix.

endloop.

ENDFORM.

User Exit (with field-symbols)


* Form called dynamically must start with DOZBW +

*****************************************************************

FORM DOYBWZDS_AGR_USERTABLES C_T_DATA STRUCTURE ZOXBWD0001.data: l_logsys type logsys.field-symbols: like c_t_data.select single logsys from t000into l_logsyswhere mandt = sy-mandt.loop at c_t_data assigning .-load_dt = sy-datum.-logsys = l_logsys.endloop.

ENDFORM.


28/57


29/57

28

Tip 09: User Exit: Read Instead of Loop

Illustration: Sample use of field symbols

User Exit (without read)

REPORT YBW2LIS_13_VDITM.*****************************************************************

* Form called dynamically must start with DOYBW + *****************************************************************

FORM DOYBW2LIS_13_VDITM

TABLES C_T_DATA STRUCTURE ZOXBWD0001.

data: l_logsys type logsys.

l_s_data like ZOXBWD0001.

field-symbols: like c_t_data,

like VBAP.Loop at c_t_data assigning .

Loop at itab assigning

where VBELN = c_t_data-VEBLN.

c_t_data-NETVALUE = c_t_data-NETVALUE + - NETWR.

endloop.

Endloop.

ENDFORM.

User Exit (with read)


* Form called dynamically must start with DOZBW +

*****************************************************************

FORM DOYBWZDS_AGR_USERTABLES C_T_DATA STRUCTURE ZOXBWD0001.data: l_logsys type logsys,

l_idx type sy-tabix.

field-symbols: like c_t_data,

like VBAP.Loop at c_t_data assigning .

READ TABLE ITAB WITH TABLE KEY VBELN = c_t_data-VEBLN

BINARY SEARCH.L_idx = sy-tabix.

Loop at itab assigning FROM INDEX l_idx.

check -VBELN = c_t_data-VEBLN.c_t_data-NETVALUE = c_t_data-NETVALUE + - NETWR.

endloop.

endloop.

ENDFORM.


30/57

29

Tip 10: Delta Enable Generic DataSources

Improve extract performance by creating delta-enabled generic

DataSources Simple:

By date

By timestamp

By sequential number (unique table key) Complex:

PointersABAP techniques can be used to record an array of

pointers to identify new and changed records


31/57

30

Tip 10: Delta Enable Generic DataSources (cont.)

Illustration: Delta enabling a generic DataSource

Ensure that you set the upper or lowerlimits correctly based on the data you are

extracting!


32/57

Tip 11: Lookups

Do not use single selects for lookups!

For better performance: Use start routines to read lookup data to an internal table

Read internal table to populate field values in routines

For best performance:

Add lookup fields to InfoSource

Use start routine and field symbols to populate blank fields for

entire data package at one time (see illustration on slide titled

User Exit Field Symbols)

31


33/57

Tip 12: Program Includes

Use includes for all complex routine logic

Access logic by using perform statements Increase portability of transformation logic

Use same read statements for multiple lookups

Reduce risk of errors in obscure places

Decrease maintenance cost of complex update rules

One place to go to fix/enhance logic

Code is consistent and easier to follow

Enable version management of code

Track changes over time

Compare between systems

Revert to previous versions

32


34/57

33

Tip 12: Program Includes (cont.)

IllustrationSelect into internal table

Start routine

FORM startupTABLES MONITOR STRUCTURE RSMONITOR "user defined monitoring

MONITOR_RECNO STRUCTURE RSMONITORS

DATA_PACKAGE STRUCTURE DATA_PACKAGEUSING RECORD_ALL LIKE SY-TABIX

SOURCE_SYSTEM LIKE RSUPDSIMULH-LOGSYSCHANGING ABORT LIKE SY-SUBRC. "set ABORT 0 to cancel update

**$*$ begin of routine - insert your code only below this line *-*

* fill the internal tables "MONITOR" and/or "MONITOR_RECNO",* to make monitor entriesperform READ_USR02_TO_MEMORY_FOR_0BWTC_C02

TABLES MONITORDATA_PACKAGE

USING RECORD_ALLSOURCE_SYSTEM

CHANGING ABORT.* if abort is not equal zero, the update process will be canceled* ABORT = 0.

*$*$ end of routine - insert your code only before this line *-*

Program include

****************************************************************** INITIALIZATION (ONE-TIME PER DATA PACKET) ********************** TO READ FROM DATABASE (ALL RECORDS FOR DATA PACKAGE) **********

****************************************************************** FORM READ_USR02_TO_MEMORY_FOR_0BWTC_C02*---------------------------------------------------------------*Form READ_USR02_TO_MEMORY_FOR_0BWTC_C02TABLES MONITOR STRUCTURE RSMONITOR

DATA_PACKAGE STRUCTURE /BIC/CS80BWTC_C02

USING RECORD_ALL LIKE SY-TABIXSOURCE_SYSTEM LIKE RSUPDSIMULH-LOGSYS

CHANGING ABORT LIKE SY-SUBRC.

* Refresh the internal table.refresh: GT_USR02.

* Read USR02 user data to memory for this data packageselect * into corresponding fields of table GT_USR02from USR02FOR ALL ENTRIES IN DATA_PACKAGE

where BNAME = DATA_PACKAGE-TCTUSERNMorder by primary key.

* if abort is not equal zero, the update process will be canceledABORT = 0.

ENDFORM. "READ_USR02_TO_MEMORY_FOR_0BWTC_C02


35/57

34

Tip 12: Program Includes (cont.)

IllustrationInclude perform statements

Update routine

FORM compute_key_fieldTABLES MONITOR STRUCTURE RSMONITOR "user defined monitoringUSING COMM_STRUCTURE LIKE /BIC/CS0BWTC_C02

RECORD_NO LIKE SY-TABIXRECORD_ALL LIKE SY-TABIXSOURCE_SYSTEM LIKE RSUPDSIMULH-LOGSYS

CHANGING RESULT LIKE /BI0/V0BWTC_C02T-USERGROUPRETURNCODE LIKE SY-SUBRCABORT LIKE SY-SUBRC. "set ABORT 0 to cancel update

**$*$ begin of routine - insert your code only below this line*-** fill the internal table "MONITOR", to make monitor entriesPERFORM READ_GT_USR02

USING COMM_STRUCTURE-TCTUSERNMRECORD_NORECORD_ALLSOURCE_SYSTEM

CHANGING GS_USR02ABORT.

RESULT = GS_USR02-CLASS.*if abort is not equal zero, the update process will be canceled*$*$ end of routine - insert your code only before this line *-*ENDFORM.

Program include

****************************************************************** RECORD PROCESSING (RUN PER RECORD) ***************************** TO READ FROM MEMORY (ONE RECORD) ***********************************************************************************************

* FORM READ_GT_USR02*---------------------------------------------------------------*

FORM READ_GT_USR02USING TCTUSERNM LIKE USR02-BNAME

RECORD_NO LIKE SY-TABIX

RECORD_ALL LIKE SY-TABIXSOURCE_SYSTEM LIKE RSUPDSIMULH-LOGSYS

CHANGING GS_USR02

ABORT LIKE SY-SUBRC. "ABORT0 cancels updateSTATICS: L_RECORD LIKE SY-TABIX.IF RECORD_NO L_RECORD.L_RECORD = RECORD_NO.CLEAR GS_USR02.

* Read user data from internal table GT_USR02READ TABLE GT_USR02WITH KEY BNAME = TCTUSERNM

INTO GS_USR02.

ENDIF.ENDFORM. "READ_GT_USR02


36/57

Tip 13: Use Start and End Routines

Start routines can be used to process the data efficiently prior to

starting the single records processing The most efficient place to delete records from the data

package prior to spending time on processing them

End routines in SAP NetWeaver 7.x allows for processing of the

data after it has been passed through the transformation It is the most efficient place to copy data records (e.g., for

generating year-to-date figures)

35

C


37/57

36

What Well Cover






Wrap-up

Ti 14 D t M d li D fi i Di i


38/57

37

Tip 14: Data Modeling: Defining Dimensions

Use as many dimensions as possible

Separate common filter characteristics into own dimension Use line-item dimensions for high cardinality characteristics such

as document numbers

Do not set the high cardinality flag!

Define related characteristics in the same dimension

Calculate expected number of dimensional entries

Try not to exceed 10% of expected fact table entries

Verify the dimension design after the first dataloads usingprogram SAP_INFOCUBE_DESIGNS

Add all relevant time characteristics

If 0CALMONTH is lowest granularity, add 0CALMONTH2,0CALQUARTER, 0CALQUART1, 0HALFYEAR, and 0CALYEAR

Provides greatest reporting flexibility without need to reload

Ti 15 I l t S ti P titi i


39/57

38

Tip 15: Implement Semantic Partitioning

What is it?

An architectural design to enable parallel data loading andquery execution

Partitioning criteria: Year, Region, or Actual/Plan

Source: SAP

Ti 15 I l t S ti P titi i ( t )


40/57

39

Tip 15: Implement Semantic Partitioning (cont.)

Benefits of semantic partitioning:

Reduction in SAP NetWeaver BWA footprint (when partitionedby year)

Parallel data loading (when not partitioned by year)

Parallel query execution

Best case when partitioning criterion is set as constant Almost as good to create variables to filter on 0INFOPROV

Archival of a single InfoCube does not impact others

Easier DB maintenance

Performance benefits are so significant

semantic partitioning should be deployedon virtually every data model!


41/57

Wh t W ll C


42/57

41

What Well Cover






Wrap-up

Ti 16 S it h Off SID D t i ti f DSO


43/57

42

Tip 16: Switch Off SID Determination for DSOs

Switch off SID determination for DSOs that are not used in

reporting

SID determination is required only for report DSOs and take up

40-70% of the activation time

Ti 17 A ti t P ll l P i


44/57

43

Tip 17: Activate Parallel Processing

Parallel processing is

possible for most steps

in SAP NetWeaver BW

DTP Parallel

Processing

DSO settings

Transaction code

RSODSO_SETTINGS

Tip 18 Compress Data


45/57

44

Tip 18: Compress Data

Compression of InfoCubes helps with two things in the dataflow:

Makes the tables that are updated smaller and hence faster toupdate

The process variant that drops and recreates the indices during

loading in a process deletes only the indices on the F-fact table

and hence the time to rebuild indices is much faster

Recommendation

Compress data that is older than 2-8 days depending on your

load schedule

Tip 19: Implement Number Range Buffering of DIMs and


46/57

p p g gSIDs

The number range tables (NRIV) are called for every new distinct

record that is loaded to SAP NetWeaver BW as either master data

or dimension in an InfoCube

The NRIV table is accessed with a select for update statement,

which can be quite slow

Buffering should be done as follows:

Determine the large number ranges (Document numbers,

Dimensions with documents or many distinct values)

Goto t-code SNRO and set up buffering

45

Tip 20: Implement SAP NetWeaver BW Accelerator


47/57

46

Tip 20: Implement SAP NetWeaver BW Accelerator

SAP NetWeaver BWA is superior to aggregates when it comes to

improving performance

Aggregates require continuous tuning as the data and query

requirements change over time

SAP NetWeaver BWA requires limited maintenance effort in

comparison

If you can afford it, you should invest in SAP NetWeaver BWA

Tip 20: Implement SAP NetWeaver BW Accelerator (cont )


48/57

47

Tip 20: Implement SAP NetWeaver BW Accelerator (cont.)

Disk speed is growing slower than other hardware components

47

In-memory datastores

Multi-channelUI, high eventvolume, crossindustry valuechains

Application-aware andintelligent data

management

Disk-based datastorage

Simpleconsumption ofapps (fat clientUI, EDI)

General-purpose,application-agnostic

database

1990 2010Architectural DriversImprovement20101990

216Addressable

Memory

2502x50.15MB/$

0.02MB/$

Memory

5066x253.31MIPS/$

0.05

MIPS/$CPU

Technology Drivers

600MBPS

5MBPS

Disk

Data Transfer120x

1000 x100Gbps

100Mbps

NetworkSpeed

264 248x

Source: 1990 numbers SAP AG,

2010 numb ers, Dr. Berg

Physical hard dr ive speeds grew by only 120 t imes

sinc e 1990. Al l o ther hardw are components g rew faster.



49/57

Source: SAP


In this example, the

average query

execution took 58.8seconds; after

SAP NetWeaver

BW Accelerator,

the average query

took 17.9 seconds(295% faster overall)

48



50/57


With SAP NetWeaver BW 7.3,

you can have data in SAP

NetWeaver BW Accelerator;

InfoCubes are not required

This saves the loading time to

the BW cube start schema

You should implement SAP

NetWeaver BWA if you want to

consistently improve query

performance and data load performance

49



51/57


SAP NetWeaver BWA is an appliance, but it does require some

maintenance activities to keep it running smoothly

Monitor SAP NetWeaver BWA utilization to avoid overloading

The rule of thumb is that you should have data that is less

than 50% of the memory size

Overloading SAP NetWeaver BWA will cause performancedegradation

Compress the cubes and rebuild indices on a regular basis

SAP NetWeaver BWA is not a cheap toy. The licensing is

based on blades used. Avoid using more space than needed by dropping and

rebuilding the SAP NetWeaver BWA indices on a regular

basis

50



52/57

51

Gather information about end-userquery requirements and drill-downpatterns

You can suggest aggregates based onquery design

Execute the query multiple times usingrealistic drill-down scenarios

Allow time for users to execute queriesand collect SAP NetWeaver BW statistics

You can suggest aggregates based onSAP NetWeaver BW statistics

Analyze the use of aggregates Modify aggregates for optimization

Before aggregate creation: After aggregate creation:


Avoid aggregates but consider as a back up for SAP NetWeaver

BW Accelerator

They come at a cost

Additional step in data loading

Longer runtime for master data and hierarchy activations

Check that the query is using the aggregate via

RSRT

What Well Cover


53/57

52

What We ll Cover





Optimizing the data updates Wrap-up

Resources


54/57

Resources

Joe Darlak of COMERIT, SAP NetWeaver BI and Portals 2010

conference (Orlando, Florida)

Practical Tips to Improve Data Loading Performance and

Efficiency in SAP NetWeaver by Up to 75%

Training BW360 BWPerformance and Administration class

53

7 Key Points to Take Home


55/57

54

7 Key Points to Take Home

Use the SAP NetWeaver BW statistics to find data loads that

require optimizationtarget to optimize top 5-10 every month

Use SE30 to analyze ABAP runtime for DataSources and

transformations

Review and implement the recommended database parameters for

SAP NetWeaver BW

Ensure that all SQL statements used in the data loading process

are using indices and that statistics are calculated for the tables

Make sure that the ABAP coding used in extraction exits and

transformation is optimized

Review and optimize the data models to avoid unnecessary

processing

Use parallel processing during data loading and updates

Your Turn!


56/57

55

Your Turn!

How to contact me:

Jesper Moselund Christensen

[email protected]

Disclaimer


57/57

Disclaimer

SAP, R/3, mySAP, mySAP.com, SAP NetWeaver, Duet, PartnerEdge, and other SAP products and services mentioned herein as well as their

respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product

and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by

SAP.

20 Tips and Tricks to Improve Data Load Performance.ppt

Documents

Transcript of 20 Tips and Tricks to Improve Data Load Performance.ppt