Post on 19-Dec-2015
307: BW Performance Tuning - Queries / Data Loads
Ramesh SampathMike Maddox
Suresh KandoorVJ Sudarsan
http://www.codongroup.com
Objectives
To provide a comprehensive understanding of the factors affecting data load and query performance and to discuss the strategies to help identify, monitor and resolve performance issues.
2
• Why Load Performance & Query Run-Times are important ?
• Query Performance• Identifying long running Queries• Query tuning Techniques
• Data Load Performance• Identifying long running loads• Improving Data Load times
• Questions
Agenda
3
Why Query Run-Times ?
• More Reporting Functions moving from R/3 to BW
• User Frustration with Data Warehouse Promise
• Impacts Number of Analysis Performed
• User Productivity is affected by Query Run-Times
• Quick response times, reduces Concurrent Users
• Affects the bottom line !! 4
Query Performance
• Identifying and Isolating the slow Queries – Using BW Statistics– Using SAP Transactions
• Exploring Possible Solutions to achieve better query run-times
5
Identifying long running Queries
Using BW Statistics
Average Run-Time of Query in Seconds
6
Identifying long running Queries
Using BW Statistics
7
Identifying long running Queries
Using BW Statistics
8
Identifying long running Queries
Using SAP Transactions• ST22 – Query Run Time exceeded set Limit
for queries
9
Identifying long running Queries
Using SAP Transactions• Analyze SM50, SM66 to identify real-time query
runs / users affected
10
Why Load Performance & Query Run-Times are important ?
Query Performance Identifying long running Queries Query tuning Techniques
• Data Load Performance• Identifying long running loads• Improving Data Load times
• Questions
Agenda
11
Query Performance
• Query Tuning Techniques– Analyze Query Plan– Identify degenerated indexes or incorrect
statistics– Design Considerations (Data Modeling)– Queries on InfoCube and ODS– Selections on Hierarchies– Do’s and Don’ts in Query Building
12
Query Tuning Techniques
Displays Query Plan
Displays List of Aggregates applicable to query based on selection
Analyze Query Plan (RSRT)
Check Read Mode of Queries
13
Query Tuning Techniques
Specify Database Hints
14
Query Tuning Techniques
Specify Database Hints
Query Cost Estimate
Check the SQL Generated, to Identify Secondary Indexes on Dimension Tables, Master Data Tables
15
Query Tuning Techniques
Full Table Scan on Fact Table – May lead to bad performance
Are the Statistics data Accurate ?
16
Query Tuning Techniques
Identify degenerated indexes or incorrect statistics: (RSRV)
Check DB Parameter Settings
Check Degenerated Indexes
Check Database Statistics 17
Query Tuning Techniques
Transaction: RSRV in BW 3.XCheck DB Parameter Settings
18
Query Tuning Techniques
• Design Considerations for– Dimension Tables– Fact Tables– Aggregates– ODS Objects– Master Data
19
Query Tuning Techniques
• Design Considerations for Dimension Tables:– Smaller Dimension Tables - Dimension Table size
< 5% of Fact Table– Avoid M:N relationship– Logical grouping of objects in dimension– Fewer Dimension Tables per Cube– Avoid Near Line Item Dimensions– For large dimension tables, change index type to B-
tree from Bitmap (default)– Create Secondary Indexes on Dimension tables
20
Query Tuning Techniques
• Design Considerations for Fact Tables:– Partition large fact tables (over 10 million records)– Create Physical Partitioning (one Million records per
partition)– Compression– Aggregates– Avoid Virtual Key Figures– Avoid exception aggregation
21
Query Tuning Techniques
Logical Partitioning of Cubes
Consolidated view of all data
Parallel SELECT Statements
MultiProvider
Europe
2000
America
2001
Asia
2002
The Power of Parallel Processing!
Query against MultiProvider
Sub-queries: Run against small structures (pruning)
Year
Partiton by Region
22
Query Tuning Techniques
• Design Considerations for Aggregates:– Data Selected from DB:Reported > 20:1– Best on Delta Capable Cubes – Use Nav. Attributes in Aggregate (Pro / Con). -
Avoid adding frequently changing attributes– Create Aggregates on Hierarchy nodes– Create Base Aggregate and summarized
Aggregate from Base– Avoid Virtual Key Figures / Virtual Characteristics– Avoid "Before Aggregation" Formulas in Queries
23
Query Tuning Techniques
• Design Considerations for ODS Objects:– Create Secondary Indexes – Physical Partitioning using DB Tools– Locally managed indexes on Partitioned ODS– Use Multi-provider to split the Data– Create summarized cubes – High level analysis on Cubes, Detailed level
analysis run queries or Jump to ODS queries. – Avoid Compounded Info Objects in ODS – Inefficient
filtering by SAP– Index on Master Data P & Q Tables
24
Query Tuning Techniques
• Design Considerations for Master Data:– Additional Over head with Time dependent Master data
and hierarchies– Hierarchy vs. Navigational Attributes. Navigation
attributes perform better than hierarchies.– Create additional index on the attributes of the master
data that are used in query selections. X, Y Tables are used in Cube based Query filters. X, Y, P, Q Tables are used in ODS Based query filters. Create secondary indexes on Master data tables based on its usage
25
Query Tuning Techniques
• Query Selection on Hierarchies– Temporary tables generated by SAP when querying based Hierarchy nodes. – Solution: Flatten the hierarchy nodes as attributes and select on them.
HIERNODE
26
Query Tuning Techniques
• Do’s and Don’ts in Query Building:– Avoid generic queries – Break down queries by summary level & detail level analysis– Always have mandatory variables on time component – Take advantage of Partition Pruning – Limit the Use of hierarchies in Query Selections, use them
extensively in Query Output– Use trailing Wild cards in query selections – Index!– Create indexes on MD, Dimension table objects used in
query selections– Limit the use of ‘Before Aggregation’ Formulas
27
Query Tuning Techniques
• Summary of Query Enhancing Techniques– Aggregates– Compression– Partitioning– Secondary Indexes– Accurate Database Statistics– Multi-Providers– Data Class / Table Space assignment for Cubes /
ODS Objects and Indexes
28
Why Load Performance & Query Run-Times are important ?
Query Performance Identifying long running Queries Query tuning Techniques
Data Load Performance• Identifying long running loads• Improving Data Load times
• Questions
Agenda
29
Why Load Performance ?
• With BW Evolving as Corporate Data Warehouse, more Functional Areas (Logistics / Financials / HR / …) are being serviced by BW
• Small Window to Load Large Amounts of Data
• Shrinking Data Load Batch Windows
• Request for Frequent Data Loads. Middle of day data loads, hourly …
30
Data Load Performance
• Identifying and Isolating the Load Performance:
– Using BW Statistics– Using SAP Standard Transactions
• Exploring Possible Solutions to improve load run times
31
BW Statistics – Load Times By Data Targets
Identifying long running loads
Record Count Load Time
32
Detailed Analysis for Info Cube Loads
Identifying long running loads
Slow Update Rules
Inserts to Cube Slow
# of Records processed
33
BW Statistics – Load Times By InfoSource
Identifying long running loads
Load Time
Record Count
34
Detailed Analysis to identify cause on all Loads (Cube / ODS / Master Data)
Identifying long running loads
Slow R/3 Extractor
Slow Update Rules
Slow Transfer Rules
# of Records Processed
35
Identifying long running loads
SM37 on BW system – ‘BI_ODSA*’ for ODS Activation Times
SM50, SM66 – Process Overview
ST04 – Database Analysis Overview
36
Identifying long running loads
• Summary of factors affecting data loads– Inefficient Extract Programs– Inefficient logic in User Exits– Slow Data Transformation Services
– Transfer Rules– Update Rules
– Slow Data Loading Services– Loads into Info Cubes– Loads into ODS Objects
37
Why Load Performance & Query Run-Times are important ?
Query Performance Identifying long running Queries Query tuning Techniques
Data Load Performance Identifying long running loads Improving Data Load times
• Questions
Agenda
38
Improving Data Load Times
• Strategies to improve Load Performance:– Extractor Performance– Info Cube Data Loads– Data Transformation Services (DTS)– ODS Data Loads– Master data Loads– Flat Files Data Loads
39
Improving Data Load Times
• Strategies to improve Load Performance: (Contd.)
– Perform Run-time Analysis on Extractor to identify Processing Times by ABAP Logic & Database Selects
– Check Select Statements on Large R/3 Tables– Ensure that Selects are based on Primary Keys or
secondary indexes– Check the usage of Run-time ABAP Memory by Internal
Tables.
– Schedule set up job in parallel 40
Improving Data Load Times
• Every thing has been checked, but extract is still slow ?– Does the selection parameters entered at the
Info package facilitate the use of Indexes !
Selection Parameters
41
Improving Data Load Times
• Strategies to improve InfoCube Data Loads:– Load Master Data before Transactions to pre-create SID’s.– Number range Buffering on Large or Near Line item
Dimensions for Large data loads– Packet Size – reduce the number of records per data packet– Secondary Indexes – Drop indexes on complete re-loads or
loading significant amount of data (over 25% of existing cube data)
– ‘Turn-off’ Archival Logs before Large Initial Loads– On complete re-load of cube, do not delete dimension table
entries, if you do not anticipate any changes to dimension entries.
42
Improving Data Load Times
• Strategies to improve InfoCube Data Loads: (Contd.)– Incremental Data Loads
– Do not drop Secondary Indexes on Incremental Loads– Do not select Refresh statistics after load
43
Improving Data Load Times
• Strategies to improve DTS– Eliminate Single Selects on Update Routines &
Transfer Routines– Utilize Start Routines– Avoid Transfer Rules to enhance Array Inserts
Examine Update Routines – Info Object Level
Utilize Start Routines – Data Packet Level
44
Improving Data Load Times
• Strategies to improve ODS Data Loads– ‘Turn-off’ Bex Reporting– Mark line item characteristics viz., doc Number as
Attributes only– Large Initial Loads to ODS
– ‘Turn-off’ Database Archival Logs – Delete Secondary Indexes
– First time load to ODS (ODS data is empty) – Load up to 1 Million Records, but do not exceed it. – Activate After the First load to facilitate Bulk Inserts
45
Improving Data Load Times
Activating data in ODS Object
Setting ins SPRO
46
Improving Data Load Times
• Strategies to improve subsequent data loads to ODS:– Activation – Single Record Inserts vs. Mass Insert when ODS is
empty– Enable Unique records, if true. Bulk Inserts.
– Data Packet Size (<10 Data Packets per Request)– Eliminate unused indexes.– Focus on smaller data loads for fewer records per activation
to reduce the commit interval.
New in BW 3.X
47
Improving Data Load Times
• Why does deletion of an Active Request from ODS takes a long time even for few thousand records ?– Data base partitioning on Active table – No Index on ODS Primary Key on ODS Change Log Table.
Add a Index to Change log Table (SE11)
ODS Primary Key
48
Improving Data Load Times
• Strategies to improve Master Data Loads: – Smaller Data Packet Size to minimize commit
intervals– Convert full loads to delta extract based on date
selection in the extract tables– Create accurate DB statistics– ‘Turn-off’ Consistency Check.
49
Improving Data Load Times
• Strategies to improve Flat File Data Loads:– Use Logical File Names / Application Server Files– Run in Background Batch Mode– Break Large Files into Smaller Files and load them in parallel– Use Fixed Length Files in place of CSV files
50
Improving Data Load Times
• Summary of Data Load Enhancing Techniques– Extractor Performance on Source System– Info Cube Data Loads (Initial / Delta Loads)– Data Transformation Services (Transfer, Update
Rules)– ODS Data Loads (Initial, Subsequent Loads)– Master Data Loads– Flat File Loads
51
Questions ?
Questions
Ramesh Sampath rsampath@codongroup.comVJ Sudharsan vssood@codongroup.comMike Maddox mike.maddox@shell.comSuresh Kandoor suresh@codongroup.com
http://www.codongroup.com
Thank you for attending!
Please remember to complete and return your evaluation form following this session.
Session Code: [307]