BOCX Speaker Performance Tuning Steve Bickerton

SAP Business Objects

Tuning Business Analytics Services

April, 2011

© 2011 SAP AG. All rights reserved. 2 Confidential

Agenda

1. How to Properly Size a BI Platform (BOE)

2. Picking the right tool for the job- Tuning Perspective

3. SAP BOE X1 3.1 and SAP NetWeaver BW

4. Understanding and Tuning Webi reports for SAP BW

5. Processing Monitor Techniques


Sizing Overview, Estimating a Peak Load

Sizing Inputs The number of users, and frequency of their activity.

•Type of user activity: heavy (constant requests), active, medium or light user (viewing

output only).

•Application use (Intelligence, Crystal, Xcelsius, SBO Analysis etc) , scheduling vs. refresh

on demand.

•Company requirements: high availability

Three Essential Calculations: •Potential Users: the number of users that can log into the system (e.g. named users) at

any time.

•Concurrent Active Users: users who are logged into the system and interacting with the

system.

•Simultaneous Requests: the number of requests that are being made by concurrent active

users at the same time. This factor has the largest impact on CPU and memory usage.


Business Objects Enterprise Tiers and Services

The Business Objects product suite

can be divided into 4 broad

categories: Client Tier, Application

Tier, Intelligence Tier, and Processing

Tier.

Each Tier is Optimized for Specific

Tasks and Operations

The Application, Intelligence Tier and

Processing Tier are server based and

represent the foundation of Business

Objects Enterprise.

These tiers can be segregated to

ensure optimal memory usage

Tiers can be scaled Horizontally

(new servers) or Vertically

(additional services).

Vertical scale if possible and for

redundancy, then Horizontally

scale.


BusinessObjects Enterprise Architecture


Sizing Information Now Publically Available

BI 4.0 Sizing

• In XI 3.1, sizing output was in CPUs and

RAM. In 4.0 CPUS are deprecated, and

the output with be in SAPS. Advantage of

SAPS:

• Each of our hardware partners

generates the SAPS rating for the

systems that they sell. This means the

SAPS ratings are always current –

there is built in anti-obsolescence.

• SAPS are O/S independent, scalar

units of capacity

• SAP Business Objects BI 4.0 sizing

approach is now consistent with how

SAP provides sizing information for

other products like Business Suite 7

• “T-Shirt sizing” now available from

Service Market Place

• Browse to http://service.sap.com/sizing

Hardware Sizing Sizing Guidelines

Solutions & Platform

BI 3.1 Sizing

• XI 3.1 sizing output is CPU and RAM

http://service.sap.com/sizing


Agenda







BI Approach: Consider the flow of what determines what

BW (or other) data

source

End User

Our only focal point. Nothing is more

important than the end user.

If we fail to address end user needs, the

implementation fails.

Our end user needs determine the

definition of reports and dashboards.

This determines also our technical

landscape and data requirements.

The delivery of our reports and dashboards

determines the sizing and configuration of

the SBO BOE architecture. A mismatch here

will cripple the environment and impact user

experience.

Data structures, optimizations,

aggregations, indexes should be based

on data access needs of the reports,

dashboards and universes, and sized to

support end user needs.

Reports and

Dashboards

SAP Business

Objects Enterprise

BW (or other) data

source

SAP Business

Objects Enterprise

Reports and

Dashboards

End User

Following a bottom-up approach leads to database back-ends that don’t serve

BI data access needs well, a BOE environment not configured to support our BI

solutions, and reports and dashboards that are unlikely to serve end user needs.


Analysis

BEx Analyzer, WAD

Web Intelligence

Explorer

The Right Tool For The Job Providing best fit solutions across the range of BI requirements

Crystal Reports

Full

Dashboards (aka Xcelsius)

Data Exploration

OLAP Analysis

Interactive Reporting

Dashboards & Visualization

Reporting

Interactive Experience, Responsiveness Limited

Professionally

Informed

Technically

Capable

Information

Consumer

Executives,

Managers

Business

Analysts


BI Decision Tree

End User Request

Information

Discovery Objective

Presentation

Operational

Report

Interactive Formatted

Type of

View

Crystal

Reports

Web

Intelligence

Information

Delivery

Casual

Type of

Discovery

Defined Scope

of Analysis

Unbound

Search and

Exploration

Explorer

Type of

Analysis Advanced

Web

Intelligence

OLAP

Source

End

Xcelsius

Non BW SAP BW

SBO Analysis

Office/ for

OLAP

SBO Analysis

for OLAP

End

Business

Analytics/KPIs

© SAP 2009 /


Impact of Incorrect Tool Choice or Lack of Proper Scoping

• Data dumps are not “BI”. Users do not need

millions of hundreds or thousands of rows. These

“reports” lead to memory errors/ stability issues on

our BI platforms.

• Troubleshoot the cause (do you really need a data

dump) not the symptom (tuning BOE for million row

report).

• Better options include:

• Collect proper requirements! If we don’t use phone

books anymore, do we need data dumps? What is

the cost to your organization to supportt this?

• If users really want to search millions of rows of

data to find a relevant data point- position Explorer

with BWA or HANA.

• Use Guided Analysis:

• Drill to detail from summary dashboard, using

open document syntax.

• Report drilling, or report to report passing

parameters.


BI Solution with Guided Navigation QTD Dashboard

Detail QTD Revenue Report by Product Line

Detail QTD Revenue Report

Detail Top Seller % Achievement Report

Detail Total % Achievement Report

Total % Achievement Dashboard

Top Seller % Achievement Dashboard

Performance by Region and

Product Line

Self-service Web

intelligence

Reports

Help users by preparing most likely

dashboard and reports in advance,

with guided navigation into detail

results

Will reduce situation where many

users create very similar ad-hoc

reports

Will save end users time by reducing

the number of steps to get the

information they need

Still provide ad-hoc access to

universe for further analysis. The

Query Panel can also be used to

edit any of the pre-built Web

Intelligence reports for easy

expansion of querying

Put query time and row limits on

your universes!

Workflow an Guided Navigation


Tuning Dashboards- “Dos” and “Don’ts”

Don’t Do

Do not over use Dynamic Visibility.

Do not over engineer.

Use dashboard builder to break your

SWF files apart

Do not trigger all your queries to run at

the same time (very common design

issue!)

Trigger queries logically. Use

Dashboard Builder to segregate

queries as well.

Do not overload data into Xcelsius

model (keep your data set to 500

rows, maximum 5,000 rows)

Bind selection values to trigger other

queries.

Use drill to detail to accomplish greater

detail.

Don’t fetch 1000s of rows for

aggregations

Push Aggregation down to the

RDBMS. Build simple summary tables

in the database.

For large deployments, don’t over rely

on Live Office for caching of data.

XI 3.1 SP 3 Customers should

seriously consider QaaWS caching

capabilities.


Agenda







Leverage the Existing Improvements in the Product

• Significant improvements have been made

in SAP NetWeaver 7.01 SP 6 (BW 7 EHP 1

SP 6).

• OLAP Data Access (ODA) has been

significantly improved. The ODA component

connects to BW's OLAP BAPI. This connection

has now been optimized and streamlined in

order to improve the interoperability.

• MDX automatically sorts the data. WebI does

not leverage this sorting, as users can sort at

the report level. Sorting inside the OLAP BAPI

is redundant and can now be skipped by using

a new UNORDER() function in SAP's MDX.

• Leaner memory consumption during flattening.

This means that the new API has a much

slimmer result structure - thus (a) consuming

less memory and (b) requiring less processing.

This has led to an optimized flattening

algorithm, now on the API side.


BW 7.01 SP 3 vs. BW 7.0 Performance


MDX PERFORMANCE

IMPROVEMENTS

Leaner and dynamic data structure

inside BW analytic engine

processing

Availability:

-SAP note 1241650

- XI 3.1 FP1.2

DATA TRANSFER

Enhanced communication between

ODA

and BW while transferring data

Availability

- BW 7.01 SP6

- XI 3.1 FP1.9

- XI 3.1 SP2 FP2.1

Data Provider Calculator

ODA

Dataset BAPIs

Rowset APIs

Metadata BAPIs

Query Execution controller

Parser Execution Instance

Function library

Analytical Engine

Data Manager

DATA STREAMING

Enhanced communication between

ODA layer and Calculator layer

No longer waiting data full

completion from BW prior to send it

to the Cube.

Availability: FP1.8 (codeline XI

3.1), FP2.1 (codeline XI 3.1 SP2)

QUERY STRIPPING

Only objects needed in the report

will be considered during the

refresh Phase

Availability: SP3

On-Going Improvement Efforts BOE-BW

MDX and BEx Query based access to BW

MD

X E

ngin

e

BO

E-W

ebi

BW

Webi

METADATA RETRIEVAL

OPTIMIZATION

Only active hierarchies information

Retrieved

Availability:

• FP1.8 (codeline XI 3.1),

• FP2.1 (codeline XI 3.1 SP2)

Data Streaming

15% perf gain*

Metadata Retrieval

Optimization

25% perf gain*

Query Stripping

Data Transfer

Gzip compression

New Flattening

format

MDX Performance

improvements

40% perf gains*


Improvements and fixes summary BW queries with large result sets Remove one million cell limit BOE XI 3.1 FP1.2 or SP2

BW 7.01 SP2

MDX data transfer optimization (gzip compression) BOE XI 3.1 FP 1.9 or 2.1

BW 7.01 SP6 (planned)

Query stripping: only objects needed in the report will be considered

during the refresh phase

BOE XI 3.1 SP3

BW Query navigation behaviour (fetch

new data from BW at drill down)

Query stripping: only objects needed in the report will be considered

during the refresh phase

BOE XI 3.1 SP3

BW Queries with hierarchies Metadata retrieval optimization BOE XI 3.1 FP 1.8 or 2.1

BW 7.01 SP4

Hierarchy navigation BOE XI 3.1 SP2

BW Queries larger than Universe /

Web Intelligence Query (especially

with key figures)

BW MDX performance optimizations BW 7.01 SP5

Build queries specifically for reporting needs, and as much as possible

have your universe match your BEx query. Only include dimensions

needed in the report in the Webi queries.

N/A

BW Queries change management Tight change management for BW Queries that are used in OLAP

Universes: BEx queries and Universe always in synch

N/A

Usability Improvements meta data handling BOE XI 3.1 SP2

Universe structure refresh feature with OLAP universes BOE XI 3.1 SP2

BW 7.01 SP3

BW Queries with variables used as

input control in Web Intelligence

Usability Improvements meta data handling BOE XI 3.1 SP2

BW Queries involving navigational

attributes

Use Characteristics instead of Navigational Attributes whenever

possible

N/A

General performance improvements

for all MDX access based use cases

BW MDX and ODA Optimization BOE XI 3.1 FP1.2 or SP2

BW 7.01 SP3

Data Streaming BOE XI 3.1 FP 1.8 or 2.1

BW 7.01 SP4


Important SAP Transactions

• ST05 – To review the Query plan on the BI Database and

trace time spent by the query at different levels i.e. DB,

network etc.

• Mdxtest – Tool to run WEBI Generated MDX statement

outside of Web Intelligence.

• SE11 – Build indexes on the Database to improve query

performance.

• SM66 – Transaction to monitor Performance of SAP BI

Server.

• RSRT – Provides Detailed Information on Operational

aspect of running a query

• ST03 – Workload Analysis Monitor, Provides monitoring

On Work Processes

• RSA1 – Helps to understand structure of the Cube and

displays InfoObject Information


SOFA Logs

SAP Note 1235111 outlines the process for turning on SOFA traces

that help to capture the logs.

MDA.log file captures all the MDX generated by WEBI.

Sample MDX Query

SELECT { [Measures].[4EWRHBHIGZRV4SGPEOWQ9AIAX] } ON COLUMNS , NON EMPTY

CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CRO

SSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJO

IN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( CROSSJOIN( { [ZMATERIAL__ZFGSBCLAS

].[A6G], [ZMATERIAL__ZFGSBCLAS].[ANQ], [ZMATERIAL__ZFGSBCLAS].[Q7R] } ,

[Z_DPLOC].[LEVEL01].MEMBERS ), [ZMATERIAL__Z_SRVCMCD].[LEVEL01].MEMBERS ),

[ZMATERIAL__ZSTOPDATE].[LEVEL01].MEMBERS ), [ZMATERIAL__ZLIFSTAT].[LEVEL01].MEMBERS ),

[ZMATERIAL__ZGROWTH].[LEVEL01].MEMBERS ),

[ZMATERIAL__ZFGSBFMLY].[LEVEL01].MEMBERS ),

[ZMATERIAL__ZFGFAMILY].[LEVEL01].MEMBERS ),

[ZMATERIAL__ZFCSDATE].[LEVEL01].MEMBERS ), [ZMATERIAL__ZEOSDATE].[LEVEL01].MEMBERS ),

[ZMATERIAL__ZEOPDATE].[LEVEL01].MEMBERS ), [ZMATERIAL].[LEVEL01].MEMBERS ),

[ZMATDPLOC__ZSPNDRGRP].[LEVEL01].MEMBERS ),

[ZMATDPLOC__ZRUNRGRP].[LEVEL01].MEMBERS ),

[ZMATDPLOC__ZLSTDTAPD].[LEVEL01].MEMBERS ),

[ZMATDPLOC__ZDPPLNR].[LEVEL01].MEMBERS ), [ZMATDPLOC__ZDPABC].[LEVEL01].MEMBERS ),

[ZMATDPLOC__ZAUTOREL].[LEVEL01].MEMBERS ),

[ZMATDPLOC__ZAPPROVED].[LEVEL01].MEMBERS ),

[ZMATDPLOC__ZAPOREGN].[LEVEL01].MEMBERS ), [ZFISCWKYR].[LEVEL01].MEMBERS )

DIMENSION PROPERTIES MEMBER_NAME, [ZMATERIAL].[2ZMATERIAL],

[ZMATERIAL__ZFGSBCLAS].[2ZMATERIAL__ZFGSBCLAS],

[ZMATERIAL__Z_SRVCMCD].[2ZMATERIAL__Z_SRVCMCD], [Z_DPLOC].[2Z_DPLOC] ON ROWS

FROM [ZSCG_M11/ZSCG_M11_Q02_FUC_3_2] SAP VARIABLES [!V000001] INCLUDING

[ZASOFWKYR].[200910]


Agenda







Runtime Parts for Report Execution

The overall runtime can be split in 5 different parts (Database, OLAP, MDX, RFC and Web Intelligence).

9000 to 9011

3500 to 4999

40000 to 40036

Webi Report stats

less other values


Database runtime in this example is 2.030

Tuning Database Runtime

• Larger result sets requested from Database may impact the Database runtime compared to BEx reporting

• Web Intelligence has different approach of where data is loaded at refresh time as compared to BEx.

• Web Intelligence uses so called “MicroCube” to load data on the BusinessObjects Enterprise Server

• This is comparable to a BEx report in “Query to read all data at once”


Tuning Database Runtime

The following Tuning Measures may help to reduce the Database runtime:

• If database selects (EVENTID 9011) are much higher value than database transfers (EVENTID 9010), building Aggregates can help to reduce the database runtime.

• If there are unselective accesses to the database causing the long runtime in many cases the filter set in the WebIntelligence report is not restrictive enough (e.g. data is requested for millions of Assets, etc.). In this situation redesigning the WebIntelligence report in regards to the Mandatory Filters can help improving the performance.

• If the long database runtime is caused by master data accesses (/BI0/S* or /BIC/S* tables) or navigation attributes (/BI0/X*, /BI0/Y*, /BIC/X* or /BIC/Y* tables) it might help to change the Master Data Read Mode. By default MDX reads only posted values. This can be switched to data in master data table according to SAP Note 1224318 and setting RSADMIN parameter MDX_JOIN_CUBE_DIME=A.


Gathering OLAP Runtime Stats

• Click the header of EVENTID to mark the whole column. Filter on event

range 3500 to 4999.

• The sum of the EVTIME in this example is 20.33 seconds. Another

important measure for the OLAP runtime is the EVCOUNT of the 3200

event(s). This is the number of cells which OLAP had to calculate and can

significantly influence the OLAP performance.


Tuning OLAP Runtime

To improve OLAP runtimes the same techniques as for normal BEx queries can be applied

RSRT should be checked in regards to OLAP cache settings and in regards to the “Use Selection of Structure Elements” flag


Tuning OLAP Runtime

If there is a long runtime on event id 3200 (OLAP: Data transfer to front-end) it might help in some cases to check the mentioned flag as shown in the following illustration.


Tuning OLAP Runtime

• Check if the “Cache Mode” is set e.g. to “Main Memory Cache Without Swapping” so the OLAP cache can be utilized for the WebIntelligence report.

• The EVCOUNT of event 3200 gives a hint on how many cells had to be calculated.

• Calculating millions of cells can significantly impact the performance.

• Root causes for too many calculations in OLAP are:

• Many calculated keyfigures

• Exception aggregation

• Too large datasets (keep in mind the read mode is all at refresh time)

• Suboptimal coding in MDX (always check if you are on the latest SP level and EHP1)

•When using hierarchies with a big amount of unassigned nodes the performance

can improve when suppressing the reporting on the unassigned nodes. This can be

done for single hierarchies in transaction RSH1.


Tuning OLAP Runtime


Tuning OLAP Runtime

Activate the Hierarchy after the change. Reverting to the original settings can be

done without refreshing the Universe. For tuning reports on top of MultiProviders

the MultiProvider hint can be useful for tuning the performance.

For further information on the MultiProvider hint please follow the description in SAP

Note 911939 or check the following link to the SAP Developer Network (SDN ):

How to Create Efficient MultiProvider Queries (NW7.0)

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/b03b7f4c-c270-

2910-a8b8-91e0f6d77096

There is a large variety of other potential tuning opportunities for OLAP runtimes.

http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/b03b7f4c-c270-2910-a8b8-91e0f6d77096










Gathering MDX Runtime Stats

•Click the header of EVENTID to mark

the whole column. Filter on event range

40000 to 40036.

•The sum of the EVTIME in this example

is 8.64 seconds. This is the overall MDX

runtime including the flattening.

•In SAP BW 7.0 systems (prior EHP1

SP3) you won’t find events greater than

40032 because the flattening of the data

is done on the Business Objects XI

server.

•Events 40033 and 40036 is the

flattening done by the BW backend


Tuning MDX- Basic Hints

• A significant long MDX runtime has normally two different reasons.

• Either the result set of the executed MDX statement is very large or

• There is a complex generation of the axis data. In general an MDX statement requests a certain number of axes to be generated. The number of joining columns selected in the MDX Axis will be represented as number of Axis.

• Following is a sample of MDX statement:

SELECT { [Measures].[Keyfigure1], [Measures].[Keyfigure2] }ON

COLUMNS,

CROSSJOIN(

CROSSJOIN(

[0COUNTRY].MEMBERS, [0CITY].MEMBERS),

[0COSTCENTER].MEMBERS) ON ROWS

FROM <BEx Query xyz>


Tuning MDX- General Hints (Contd..)

• Following will be the outcome of the MDX statement:

• Each axis (0COUNTRY, 0CITY and 0COSTCENTER) has to be evaluated separately.

• Adding more axis into the MDX statement leads to a higher number of permutations and decreases the performance.

• Adding / removing elements on the WebIntelligence query panel can indirectly influence the generation of the MDX statement.



• You will encounter the same performance deterioration when you do a CROSSJOIN over two hierarchies on a deep level.

• If you have e.g. a country hierarchy and a product hierarchy which have to be cross joined in MDX you will get all the possible permutations: outcome of the MDX statement:

• Each axis (0COUNTRY, 0CITY and 0COSTCENTER) has to be evaluated separately.

• Adding more axis into the MDX statement leads therefore to a higher number of permutations and decreases the performance.

• You can indirectly influence the generation of the MDX statement by adding / removing elements on the WebIntelligence query panel.



• After the cross join of the 3 members of both the hierarchies you will get 9 lines for the different permutations:

• Use filters on a specific members or hierarchy nodes ( e.g. L01 Country = “Walldorf”) to avoid memory dumps in SAP BW because of the size of the internal table used to create the permutations.


Getting E2E Runtime from WebI Report


Tuning WebIntelligence Runtime- General Hints

• Customizing BI Universe Connection:

• Universe Configuration Parameters for Universe Connections

• Keep the default value- “Connection Pool Mode” = “Keep the connection active for”

• Choosing “Disconnect after each transaction” will slow down the Universe building process and will also impact end-user workflows such as hierarchical list of values.

• Customizing BI Universe Connection :

• Universe Configuration Parameters for Universe Connections

• Change “Pool timeout” from 10 minutes to 1 minute



• Use Query Stripping( Available in SP3)

• BusinessObjects Service Pack 3 introduced a new feature called “Query stripping” Query Stripping enables you to optimize the query generation automatically. With Query Stripping option enabled, Web Intelligence generates a query containing only those objects that effectively contribute to a report. This feature only applies on OLAP universes, not relational universes. Query Stripping needs to be activated in both “Query level” and the “Document level”


Tuning WebIntelligence Runtime- General Hints (Contd..)

• Activate “Query Stripping “ in “Document level”



Remove all Unnecessary

L00 objects

• When a characteristic has no active hierarchy, the L00 node will be All members, and will not provide any reporting value. In this case, it is best to delete all L00 objects in order to reduce complexity for the report designing users.



• Dynamic filtering with WebI filters.

• When filtering on high cardinality characteristics, ensure that any characteristics which are filtered in WebI are filtered on indexed values.

• Reports with high data volume

• The OLAP BAPI (MDX) interface is not suitable to be used for queries which return a high volume of data in a single request. The volume of data returned can be measured by the number of cells returned. In general, it is desirable to reduce this number to the minimum required for the reporting requirement. Position a Data Federator Universe instead.

• Use inclusive member filters rather than exclusive ones:

• Avoid using “Not Equal To”, “Not In”, “Not between”, etc. in the filter pane of the WebI query panel. If this is not possible, consider doing the filtering in the BI query (see Static filtering with BEx Query restrictions)

• Reducing the size by optimizing WebI queries

• If not on XI 3.1 SP3, Remove unused fields from the query. Review the query definition before publishing a document, and remove any fields from the query definition which are not actively used or desired to be made available during analysis. WebI will not optimize the query at run-time to remove those fields which are not required.


Agenda




4. Understanding and Tuning Webi against SAP BW



Monitoring Techniques for UNIX (Solaris) • Monitor Process usage at the OS level.

Leverage the following commands to get a

better understanding of the current CPU,

memory, and disk I/O activity on the server:

• prstat – this is a Solaris command is

useful for monitoring specific process-

related information such as CPU and

memory usage on a per process basis

• prstat -u dboeuser -n30 -s cpu 5 10

> prstat.out – this command returns

the top 30 processes, sorted by CPU

usage, owned by the UNIX id

dboeuser every 5 seconds over 10

iterations (covering a 50 second

timeframe), and writes this output to a

file named prstat.out. This can be a

useful command to run during

performance testing windows in order

to monitor specific XI 3.1 process

activity.


Monitoring Techniques for UNIX (Solaris)

• sar – sar (system activity reporter) is a

useful Solaris utility for displaying the

overall system performance on the

server. It can capture daily utilization

by various activity types in the

environment.

• sar -A -o outfile 5 500 & This

command would run in the

background, retrieve all system activity

every five seconds over 500 iterations,

and write the results to a file named

outfile in the current working directory.

To review the results in this file, we

would need to use sar again, as the

output generated by sar is in binary

format.

• iostatsar – iostat reports terminal and

disk I/O activity and CPU utilization.

• The values to focus on from the iostat

output are:

• Reads/writes per second (r/s , w/s)

• Percentage busy (%b)

• Service time (svc_t)

• mpstat - mpstat reports per-processor

statistics in tabular form. Each row of

the table represents the activity of one

processor. The first table summarizes all

activity since boot; each subsequent

table summarizes activity for the

preceding interval. All values are rates

(events per second) unless otherwise

noted.

BOCX Speaker Performance Tuning Steve Bickerton

Documents

Transcript of BOCX Speaker Performance Tuning Steve Bickerton