Saroj Bagai/SAP Global Product Support Oct 8, 2015 Customer

75
Customer Coffee Corner for SAP IQ Troubleshooting SAP IQ Query Performance Saroj Bagai/SAP Global Product Support Oct 8, 2015 Customer

Transcript of Saroj Bagai/SAP Global Product Support Oct 8, 2015 Customer

Customer Coffee Corner for SAP IQ – Troubleshooting

SAP IQ Query Performance

Saroj Bagai/SAP Global Product Support

Oct 8, 2015 Customer

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 2Customer

Agenda

Troubleshooting SAP IQ Query Performance Presentation

Open Discussion about Query Performance Issues

Common links for information

Product components

BC-SYB-IQ (IQ)

BW-SYS-DB-IQ (IQNLS)

XX-SER-LIKEY-SYB ( License Key)

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 3Customer

Background Information

• Is it generated query?

• Any Schema Changes?

• Any Changes on the OS/HW?

• Are you using any front tool?

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 4Customer

Isolating Performance Issue

• Is performance problem due to OS/HW

- Monitor OS, HW and storage for any bottlenecks or Issues

- Check for high CPU use, high CPU system time, low CPU user time, high wait time.

- Check for IO average service times that are above 10ms.

• Performance problems within SAP Sybase IQ

- Look for missing indexes by enabling index advisor

- Enable query html plan if issue is only with isolated query.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 5Customer

OS Diagnostics Tools

OS Utility Description

Unix Top, TopsProvides runtime look of processor activity in real time. Top:

Available on Solaris, Linux and HP-UX. Topas: AIX

ps Reports process status.

vmstatProvides information about system processes memory, paging,

block Q, traps and CPU activity

iostat -t –d –x 5 20 Provides disk subsystem information.

sar Writes selected OS activity results to standard output.

Windows Task Manager,

Resource Manager

Provides detailed information about windows performance, and

running applications, CPU usage, and other system services.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 6Customer

Common Performance Problems

• Disk Swapping and Paging

• Index and Row fragmentation

• Catalog file growth

• Thrashing and Query Execution

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 7Customer

Diagnostics To Collect

• Monitor debug output

(DBA)> create table iqmondummy (c1 int)

(DBA)> iq utilities main into iqmondummy start monitor '-debug -interval 20'

(DBA)> iq utilities private into iqmondummy start monitor '-debug -interval 20'

(DBA)> iq utilities main into iqmondummy stop monitor

(DBA)> iq utilities private into iqmondummy stop monitor

(DBA)> drop table iqmondummy

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 8Customer

Diagnostics To Collect

• Query html plan

set temporary option query_plan = 'ON';

set temporary option query_detail = 'ON';

set temporary option query_plan_as_html ='ON' ;

set temporary option query_plan_after_run='on';

set temporary option query_timing='on';

set temporary option DML_OPTIONS10='on';

set temporary option FORCE_NO_SCROLL_CURSORS ='on' ;

set temporary option index_advisor='on' ;

Exec query

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 9Customer

Common Causes For Poor Query Performance

• Inaccurate estimates:

- Row Count

- Distinct Value Counts

• Schema Design:

- Normalized: specific better performance is seen with selective normalization e.g like

conditions

- Denormalized: selective denormalization is good for situations with skewed correlations

between joined tables.

• Query expressions less than ideal

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 10Customer

Query HTML Plans

• Query html plans components:

- IQ Version: SAP Sybase IQ version used to generate query plan

- DFO Tree: Tree representation of the query execution plan

- Query Timing Chart: Each operation has row providing details about the time being spent on that

operator during various query execution phases.

- Thread diagram: Provides information about thread usage for executing query

- CPU utilization diagram: Provides information about the CPU usage throughout the lifecycle of

query. Red represents System CPU usage and Green represents User CPU usage.

- Query Text: The Query SQL text.

- Query Plan Detailed Nodes: A HTML table for every query node including individual details for each

query plan node.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 11Customer

Query Timing Chart

Depicts time expended at each node

Wall clock

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 12Customer

Thread Utilization Diagram

Displays threads used over the life of the query

Thread usage corresponds to timing Chart

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 13Customer

CPU Usage Diagram

• CPU utilization diagram: Provides information about the CPU usage throughout the lifecycle

of query. Red represents System CPU usage and Green represents User CPU usage.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 14Customer

Query SQL TEXT

• Query Text: The Query SQL text.

Load into table

– create table fact (name varchar(255), minute timestamp, metric float) ;

– create table timedimension (minute timestamp) ;

– LOAD TABLE fact ("name" ',', "minute" ',', "metric") FROM '/work/data/fact.csv'

DELIMITED BY ',' ROW DELIMITED BY '\x0a' QUOTES ON ESCAPES OFF ;

– commit ;

"DBA"."timedimension"( "minute" ) using file '/work/data/timedimension.csv'

delimited by ',' row delimited by '\x0A' quotes on escapes off

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 15Customer

Important Query Plan Nodes

Root Leaf

Scrolling Cursor Store Join (12 types)

Filter Subquery

Group By (3 types) Union All

Order By Parallel Combiner

Store Parallel Order By

Semi-Join Filter filler

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 16Customer

Query HTML Plan

• Look for the areas where most of the execution time of spent.

- Optimization

- Vertical (index-based condition execution)

- Projection

- Horizontal (tuple-based) query operators

- Result transmission

- Waiting for requests

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 17Customer

Query Html Plan

Optimization Time:

Two phases of optimization:

1. Before vertical condition execution

2. Between the end of condition execution and the start of the first tuple fetch

Query timing diagram will show how long each of these phases last

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 18Customer

Query HTML Plan

Optimization Time:

Phase 2 of optimization includes:

- Join order selection

- Join and grouping algorithm selection

- Generation of the executable query tree

Phase 2 can in some cases be significant.

- Set Join_Simplification_Threshold = 9

- This can reduce join order selection costs, but it introduces a small risk that the optimizer may fail to

find the optimal join order.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 19Customer

Query HTML Plan

Vertical Condition Execution Time:

Time is shown by:

- The blue bar in the query timing diagram

- Query plan line labeled: “Elapsed Condition Time”

Includes all conditions labeled:

“Condition <N> (Invariant)”

Condition execution time is dependent upon:

- The conditions

- The indexes available, and the index chosen for each condition

- The order the conditions are executed

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 20Customer

Query HTML Plan

Vertical Condition Execution Time:

The optimizer generally chooses the optimal order to evaluate conditions, and the optimal indexes to use.

For conditions where no metadata is available the optimizer’s choice may be suboptimal.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 21Customer

Query HTML Plan

• To influence the optimizer’s choices for the execution of vertical conditions:

- Add indexes as suggested by the Index Advisor

- Supply a user-supplied condition selectivity

- Set the Index_Preference option

- Supply a condition hint which allows the user to specify for one condition:

- Selectivity

- Index preference

- Usefulness

- Execution phase

Example 1:

( (T.VCHAR1 LIKE ‘%summ%’, 10.50)

AND (T.VCHAR2 LIKE ‘%wint%’, ‘S:2.25’)

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 22Customer

Query HTML Plan

To override the optimizer’s choices for the execution of vertical conditions (cont’d):

Example 2:

SELECT CUSTOMER.*

FROM CUSTOMER, LOCATION

WHERE C_LOC_CODE = L_LOC_CODE

AND L_ZIPCODE = @hv1

AND (C_LAST_NAME LIKE @hv2, ‘E:D, U:1.0’)

A template query where the author knows an HPDJ algorithm will be used. The condition hint tells the optimizer to delay the execution of the LIKE condition until after the pushed join predicate is evaluated.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 23Customer

Query HTML Plan

Projection Time:

Visible via either:

- IQ performance monitor

- Operating system I/O statistics utilities

Not directly visible in query plans.

Pre-fetching generally means this issue is relevant only if the I/O bandwidth is saturated

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 24Customer

Query HTML Plan

Horizontal Query Operator Time:

The largest cost in most queries.

Visible in query timing diagrams as the yellow (first fetch) or green (subsequent fetches) bars.

Cost of each operator is affected by:

Number of tuples flowing through the operator

Number and width of tuples which must be copied, if any

Number of expressions or aggregates which must be evaluated for this operator

Whether the selected algorithm was able to get sufficient resources at run-time to operate efficiently

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 25Customer

Important Nodes

• Some Nodes contain a wealth of information

• Detail in other nodes is not as important

• Start with these Nodes when analyzing queries:

Root

Leaf

Join

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 26Customer

Root Node

The Root node is at the top of the

(inverted) query tree

Provides general information and

timings for the entire query

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 27Customer

Root Node

Root Node Details

• Result Rows (actual and estimated)

• User Name (login)

• IQ Temp Space Used (total)

• Active users on server

• Number of CPUs

• Cache sizes

• Threads used for all invariant predicates

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 28Customer

Query Timings in the Root node

Overall query timing is listed in the Root node• Query_Plan_After_Run and Query_Timing options must be ‘ON’ to report timings

• Total time to resolve all predicates is listed as ‘Elapsed Condition time’

Time of Cursor

Creation 2013-02-11 11:18:04.863474

End of DFE

Generation 2013-02-11 11:18:04.865834

End of DFO

Generation 2013-02-11 11:18:04.865937

End of DFO

Transforms 2013-02-11 11:18:04.866264

Time of First

Prepare 2013-02-11 11:18:04.871551

Time of First

Prepare Return2013-02-11 11:18:04.874527

Time of First

Fetch Call 2013-02-11 11:18:04.875013

Time of First

Fetch Return 2013-02-11 11:18:05.396055

Time of Second

Fetch Call 2013-02-11 11:18:05.396055

Time of Last

Fetch Return 2013-02-11 11:18:05.396056

Time of Complete

Call 2013-02-11 11:18:05.397327

Time of Complete

Return 2013-02-11 11:18:05.397328

Elapsed

Condition time0:00:00.000000

Working time 0:00:00.532583

Elapsed time 0:00:00.533854

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 29Customer

Scrolling Cursor Store node

Scrolling Cursor Store node usually found just below the Root Node• Affected by the database option Force_No_Scroll_Cursors (default ‘ON’)

The store buffers all rows from a query result set• Allows scrolling through results (forward and backwards)

– i.e. ‘Scrollable Cursor’

Consumes Temp Cache and Temp Store (for large results)• Typically this option is set OFF for better query performance

• Some query client and ETL tools may require this option be set ‘ON’

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 30Customer

Temp memory usage with Scrolling Cursors

Force_No_Scroll_Cursors = ‘OFF’ Force_No_Scroll_Cursors = ‘ON’

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 31Customer

Leaf node

All table indexes and columns used in the query are identified and accessed in the Leaf node

Types of Leaf nodes:

• (Regular) Leaf

• Aggregation Leaf

• Grouping Leaf

• Distincting Leaf

• Ordered Leaf

• Proxy Leaf (Proxy Table)

• SA Leaf (table in the SA database)

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 32Customer

Leaf node – a wealth of information

• All column metadata needed by the optimizer is identified in the Leaf node

- Row Counts

- Total rows in the table

- Estimated count after executing all conditions

- Generated count after executing all conditions

- Conditions (predicates) in the table showing:

- Selectivity

- Usefulness

- Index used to resolve the predicate

- Index Advisor Messages

- Predicate execution phase

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 33Customer

Leaf node details

Rows Counts

Total rows in the table

Generated rows after conditons (if any)

Conditions (Predicates)

Selectivity

Usefulness

Elapsed time (to execute)

Rows remaining (post execution)

Index used

Threads used

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 34Customer

Selectivity, Index Selection and Usefulness

For each predicate the optimizer determines

• Selectivity

- Portion (percentage) of the table rows that satisfy that predicate

- HG, LF and FP indexes provide this metadata

• Index used to resolve the predicate

- When multiple indexes exist, the best index for the predicate

- Cost and resources required influence index selection

• Usefulness Score

- The order the predicates have been executed

- Execution phase, cost and resources required influence Usefulness scoring

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 35Customer

Estimated Selectivity

Percent of table

Predicate Type (Estimated) Selectivity

Equality (=) 20% 0.2000000

Open Range (>) 40% 0.4000000

Between 40% 0.4000000

Like (%) 20% 0.2000000

Inter-column equality (t.a = t.b) 30% 0.3000000

Inter-column comparison (t.a < t.b) 50% 0.5000000

If no metadata exists the optimizer estimates selectivity based on the predicate operator

Selectivity with trailing zeroes may indicate an optimizer estimate

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 36Customer

Index-based selectivity

When a ‘useable’ index exists it can provide accurate selectivity

• Exact selectivity is shown as a precise number without trailing zeroes such as: 0.34982376

Note: Some functions can negate the ability to use an index for selectivity

Example: SUBSTRING( t.a, 5, 5 ) = ‘homes’

Optimizer might use default electivity20% for an equality search ( .20000000)

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 37Customer

Usefulness

Usefulness determines predicate execution order

Usefulness value ranges: 0.0 to 10.0 (10.0 = most useful)

• Predicate with highest score is executed first

• Remaining predicates executed in descending order of usefulness score

Factors determining Usefulness scoring

• Selectivity – how many rows in the table will be eliminated by this predicate

• Execution Phase

• Type of predicate (operator)

• Index available (including optimized FP index)

• Speed of predicate execution

• Resources required to execute the predicate

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 38Customer

Local predicate Execution Phase

Three categories of local predicates

Invariant

Delayed

Bound

- Category is based upon when the data required for predicate execution is available

- Predicate category usually dictates the phase (i.e. timing) when it is executed in the query

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 39Customer

Invariant predicates

• Invariant: expression does not change during query execution

- Most local predicates are invariant predicates

Example: WHERE cust_id = 12345

- Invariant predicates are usually executed in an early phase to ensure accurate row count

information is available to subsequent phases of query optimization

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 40Customer

Delayed predicates

• These are predicates that need to be executed once but require information that will become

available after another part of a query is executed:

Example:

WHERE cust_id in

(Select cust_id from customer where state = ‘VA’)

- The Where cust_id in predicate is ‘delayed’ since the subquery must be executed first

- Delayed predicates are typically found with uncorrelated subqueries (above) and some

“Push-Down” joins

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 41Customer

Bound predicates

• Bound predicates

- Need to be executed repeatedly and require information that will only be available after some other

portion of the query has been executed

• Bound predicates are commonly used in

- Correlated subqueries

- Nested-Loop Push-Down joins

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 42Customer

Predicate execution phases

• Local predicates are typically executed in this order -

1. Resolve all invariant predicates

2. Execute Delayed predicates

3. Execute Bound predicates

• Usefulness scores are related to a predicate execution phase

Note: Execution phase can be user influenced by using Optimizer Hints written into query code

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 43Customer

Query operators and indexes

• Predicates typically use these indexes with these operators

- Equality and Inequality( = , != ), IN and NOT IN lists

- Ranges ( <, >, <=, >=, Between) including NOT Between

- Like ( % ) or Not Like ( % ), Contains

LF and HG

DATE, HNG,

LF and HG

FP, WD,TEXT

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 44Customer

HG and LF indexes in predicates

• The HG and LF indexes provide exact metadata to the optimizer:

- The number of distinct values in the column and

- The number of rows found for each value

• These indexes also provide fast run-time access to the rows

- Directly influences predicate execution cost and execution phase

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 45Customer

FP indexes in predicates

• Optimized FP(Nbit) indexes (tokens) also provide metadata:

- Number of distinct values for a column

- Number of rows for each value

• Tokens dramatically reduce storage, disk I/O and memory

- Used with sorts to dramatically reduce memory

• FP index searches can use multiple threads for predicate execution

- Optimizer may prefer using FP index over other indexes since it can exploit many threads (and be

executed faster)

- IQ server work load at query time can influence this decision

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 46Customer

Column detail in the Leaf node

The Leaf Node lists all the columns accessed for the query

• Data type and indexes – no DATE index on this column

• With no DATE index the optimizer chose a multi-threaded FP search

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 47Customer

Index Advice in leaf nodes

Query tree highlights Leaf nodes in red indicating Index Advice is available in the node

Database option Index_Advisor must be set ON

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 48Customer

Other Indexes in predicates

• Except for special cases the other IQ indexes are used to locate values and do not provide

metadata

- Only the LF, HG and FP indexes provide metadata

• Some of the special cases are:

- CONTAINS predicate with a WD or TEXT index

- DATEPART function predicate on a column with a DATE index

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 49Customer

Other types of Leaf nodes

• Under the right circumstances optimizer uses an ‘Index Based’ algorithm to perform work within a node

- Examples where these algorithms are used:

Aggregation LeafSELECT COUNT(*) FROM ...

SELECT SUM(PAID_AMT) FROM …

Grouping Leaf

SELECT MEMBER_GENDER, COUNT(*)

FROM …

GROUP BY MEMBER_GENDER

Distincting LeafSELECT DISTINCT (shipdate)FROM …

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 50Customer

Leaf node(summary)

• Leaf nodes provide a wealth of information about the data and indexes used in a query

• List of all table predicates including:

- Selectivity

- Usefulness

- Execution phase

- Index used to resolve predicate

• All columns referenced, including:

- Available indexes

- Data type

- Distinct count

• Index advice

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 51Customer

Join nodes

• Join nodes merge a set of rows from one table with rows from another table passing all

combinations which satisfy the join condition(s)

• For joins the optimizer must determine two things:

- Join order (if more than 2 tables)

- Join algorithm (12 types)

• For these decisions the optimizer uses a cost-based process

- Relies heavily on the metadata for the join columns

- Metadata comes from indexes in Leaf nodes

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 52Customer

Metadata for join performance

Tips for providing the right metadata

• Individual columns which are primary keys always declare as a PRIMARY KEY

(or create a UNIQUE HG index)

- Candidate key columns create a UNIQUE HG index if used in joins

• All other join columns create an HG (or LF) index

• Multi-column primary keys where tables will be joined using all the keys create a multi-

column PRIMARY KEY (or a multi-column UNIQUE HG index)

- If all keys are NOT used in a Join then a Primary Key (or multi-column Unique index) is not

necessary

- For example, the Primary Key for a Fact table would almost never be necessary (except to maintain

entity integrity)

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 53Customer

Join algorithms

• IQ has 12 different table join algorithms

• Valid algorithms for a given join depend upon:

- Type of join condition

- Data types on the join columns (matched or mismatched)

- Outer join (if so which kind)

- Indexes on the join columns

- Join constraint (Many to Many, One to Many)

• Valid algorithms are compared against each other based on their estimated costs and

available server resources

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 54Customer

Join Algorithms

• Knowledge of the join algorithms, their strengths and weaknesses, may allow you to identify

ways you could change the query or the schema to improve query performance

• Each join algorithms has different advantages:

- Some are better for joining large tables to small tables

- Some are better for joining two equally large tables

- Some require large amounts of Temp Cache be available

- Some require large amounts of Main Cache be available

- Some are good only for very selective joins

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 55Customer

Types Of Join Algorithms

Sort-merge Join Partitioned Hash Join

Nested-Loop Join Partitioned Hash-Push Down Join

Nested-Loop Push-Down Join Partitioned Sort-Merge Join

Hash Join Partitioned Sort-Merge Push-Down

Hash Push-Down Join Sort-Merge Push-Down Join

Asymmetric Sort-Merge Join Asymmetric Sort-Merge Push-Down Join

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 56Customer

Inaccurate Leaf Node Estimates

Semi-join predicate effects poorly estimated

. . #01: Leaf

Generated Result Rows: 58590

Estimated Result Rows: 30

Pre-PushDown Est. Result Rows: 60175

Table Name: lineitem

Table Row Count: 60175

Condition 1 (Pushed): (lineitem.l_partkey IN BV(1))

Condition 1 Selectivity: 0.00050000

Condition 1 Usefulness: 5.99950000

Condition 1 Execution Method: Vertical Parallel Hash in FP

Condition 1 Index: FP lineitem.DBA.ASIQ_IDX_T205_C2_FP

Condition 1 Actual Distincts: 1949

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 57Customer

Inaccurate Leaf Node Estimates

Semi-join predicate (cont’)

. #03: Join (Hash PushDown)

Left Child Node: #01

Right Child Node: #02

Generated Result Rows: 58590

Estimated Result Rows: 31

Valid Join Algorithms: NLJ, NLPDJ, SMJ, SMPDJ, HJ, HPDJ

Left Input Table 1: lineitem

Estimated Left Inputs: 30

Estimated Left Pre-Pushdown Inputs: 60175

Right Input Table 1: part

Estimated Right Inputs: 1

Join Result Constraint: Many to 1

Condition 1: (lineitem.l_partkey = part.p_partkey)

Condition 2 (Pushed): (lineitem.l_partkey IN BV(1))

Optimizer est. max hash rows for these keys: 27115

Hash actual number of keys: 1949

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 58Customer

Inaccurate Group By Estimates

GROUP BY result estimates depend on both:

- Incoming row estimates.

- Estimates of the number of distinct values for each of the grouping expressions after leaf filtering.

- Hint 1: Look at each grouping column for a missing index:

Column 1 Base Distincts (Est.): 412

- Hint 2: Look for missing multi-column primary key definitions

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 59Customer

Inaccurate Join Estimates

• Join result estimates depend on:

- Incoming row estimates.

- Estimates of the number of distinct values for each of the simple equality join expressions.

The primary and foreign key information

Hint 1: Look at each join column for a missing index:

Column 1 Base Distincts (Est.): 412

Hint 2: Look for missing multi-column primary key definitions,

Join Result Constraint: Many to Many

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 60Customer

CheckList

• Missing Indexes

• Missing IQ Unique

• Mismatched Datatypes

• Sub-Optimal Datatype Choice

• Review Join Conditions

• Check for Over-Aggressive Push-Downs

• Check for Cache Thrashing

• Excessive Data Skew

• Missing Referential Constraint

• Thread Starvation

• Excessive Expression Evaluations

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 61Customer

Missing Indexes

• Check for: Base Distincts (Est.)

- Only concentrate on this line if the optimizer needed the distinct count

- Requires LF, HG or have flat N-bit

• Wide-range predicates on numbers requires HNG

• Infra-table column comparisons require CMP index

• Selectivity ‘magic’ numbers

- Check for selectivity e.g 0.2000000

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 62Customer

Missing IQ Unique

• Look for ‘Projection Bytes’ in Leaf node for low cardinality columns

• Each release of IQ adds more algorithms that can be pushed into the enumeration table

• Can make large performance difference for large tables

• Cannot be added by Alter Table

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 63Customer

Mismatched Datatypes

• IQ assumes that in a data warehouse time was taken to design the schema before

implementing it, including using consistent data types for all join key columns

• Some more efficient join algorithms cannot be used with mismatched datatypes

(signed/unsigned, char/varchar length etc)

• May cause less efficient algorithms to be used

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 64Customer

Suboptimal Datatype Choice

• Common cause of sub-optimal performance is not using optimal data types

• All tables should have keys, and keys should be an integral data type, and preferably

unsigned

• Don’t use CHAR or VARCHAR columns as keys for joins, ORDER BYs, or GROUP BYs if at all

possible

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 65Customer

Review Join Conditions

• Look for: Join (Nested Loop)

• Optionally followed by a Filter node

• May be a missing join condition

• May be a join condition that precludes using an efficient join algorithm

• Disjunctions (<condition> OR <condition>)

• Range predicates (<, >, Between, Like)

• Complex expressions (T1.x + T2.y = 5)

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 66Customer

Over-Aggressive Push-Downs

• Duplicate predicate selectivity

• NLPD: Especially sensitive to too many pushes

• Meaning of semi-join in plan: actually IN cost estimates

SemiJoin V: 2 D: 77446.5 T: 0 A: ORs S: 0 O: 0

V – Actual # distinct values for the IN

D – Est. # rows per distinct value

T – Actual time taken

A – Algorithm chosen: bitmap OR or column scan

S – Est. cost using column scan

O – Est. cost using bitmap disjunction

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 67Customer

Cache Thrashing

• Hash thrashing

- Happens when significant under-estimate

- Causes excessive reads and writes from the temp cache

- Solution: fix estimates or use temp table or avoid hash

• Correlated subquery thrashing

- Causes excessive reads and writes from the main cache

- Solution: rewrite the query as a join instead of a subquery

• Confirm using IQ buffer cache monitor report

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 68Customer

Excessive Data Skew

• IQ assumes modest data skew

• Sometimes data is heavily skewed

- Mostly affects push-down joins

- No simple way to work around

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 69Customer

Missing Referential Integrity Constraint

• multicolumn primary keys must be defined

• In joins, where it affects estimates, look for:

• Join Result Constraint: Many to Many

• In a GROUP BY it affects performance directly by forcing the engine to sort or hash on extra

columns.

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 70Customer

Thread Starvation

• Configured by startup Switch:

- Total threads in server -iqmt

- Set Max_IQ_Threads_Per_Connection

- Can generally be set safely up to 100

• Examine using IQ cache monitor with -debug

• Look for ‘Serial’ in Query_Plan_After_Run on UNION, and non-push-down join nodes

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 71Customer

Excessive Expression Evaluations

• Look at production lists:

Production 1: (T.B * T.C)

Production 2: (T.A * (T.B * T.C))

• Seeing a CAST in a production may be a symptom of mismatched data types

• Finding nodes with very high row counts that are producing complex expressions may imply a less than optimal schema design

- Consider creating a derived column to hold a pre-computed expression result for each commonly used expression

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 72Customer

Common Links For The Information

• Product documentation

http://help.sap.com/iq

• SAP IQ Performance and Tuning

• Support Portal

https://support.sap.com/home.html

• How to get best results from an SAP search

https://service.sap.com/sap/support/notes/2081285

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 73Customer

Common Links For the Information

• SAP IQ communties

http://scn.sap.com/community/iq

• SAP IQ Wiki

http://wiki.scn.sap.com/wiki/display/SYBIQ/IQ

• IQ 16 Best Practices

http://scn.sap.com/docs/DOC-39896

• IQ 16 Hardware Sizing Guide

http://scn.sap.com/docs/DOC-41455

© 2015 SAP SE or an SAP affiliate company. All rights reserved. 74Customer

Common Links For the Information

• Social Media Product Support Channels

https://twitter.com/SAPSupporthelp

https://www.facebook.com/SapProductSupport

© 2015 SAP SE or an SAP affiliate company. All rights reserved.

Thank you