Saroj Bagai/SAP Global Product Support Oct 8, 2015 Customer
Transcript of Saroj Bagai/SAP Global Product Support Oct 8, 2015 Customer
Customer Coffee Corner for SAP IQ – Troubleshooting
SAP IQ Query Performance
Saroj Bagai/SAP Global Product Support
Oct 8, 2015 Customer
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 2Customer
Agenda
Troubleshooting SAP IQ Query Performance Presentation
Open Discussion about Query Performance Issues
Common links for information
Product components
BC-SYB-IQ (IQ)
BW-SYS-DB-IQ (IQNLS)
XX-SER-LIKEY-SYB ( License Key)
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 3Customer
Background Information
• Is it generated query?
• Any Schema Changes?
• Any Changes on the OS/HW?
• Are you using any front tool?
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 4Customer
Isolating Performance Issue
• Is performance problem due to OS/HW
- Monitor OS, HW and storage for any bottlenecks or Issues
- Check for high CPU use, high CPU system time, low CPU user time, high wait time.
- Check for IO average service times that are above 10ms.
• Performance problems within SAP Sybase IQ
- Look for missing indexes by enabling index advisor
- Enable query html plan if issue is only with isolated query.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 5Customer
OS Diagnostics Tools
OS Utility Description
Unix Top, TopsProvides runtime look of processor activity in real time. Top:
Available on Solaris, Linux and HP-UX. Topas: AIX
ps Reports process status.
vmstatProvides information about system processes memory, paging,
block Q, traps and CPU activity
iostat -t –d –x 5 20 Provides disk subsystem information.
sar Writes selected OS activity results to standard output.
Windows Task Manager,
Resource Manager
Provides detailed information about windows performance, and
running applications, CPU usage, and other system services.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 6Customer
Common Performance Problems
• Disk Swapping and Paging
• Index and Row fragmentation
• Catalog file growth
• Thrashing and Query Execution
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 7Customer
Diagnostics To Collect
• Monitor debug output
(DBA)> create table iqmondummy (c1 int)
(DBA)> iq utilities main into iqmondummy start monitor '-debug -interval 20'
(DBA)> iq utilities private into iqmondummy start monitor '-debug -interval 20'
(DBA)> iq utilities main into iqmondummy stop monitor
(DBA)> iq utilities private into iqmondummy stop monitor
(DBA)> drop table iqmondummy
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 8Customer
Diagnostics To Collect
• Query html plan
set temporary option query_plan = 'ON';
set temporary option query_detail = 'ON';
set temporary option query_plan_as_html ='ON' ;
set temporary option query_plan_after_run='on';
set temporary option query_timing='on';
set temporary option DML_OPTIONS10='on';
set temporary option FORCE_NO_SCROLL_CURSORS ='on' ;
set temporary option index_advisor='on' ;
Exec query
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 9Customer
Common Causes For Poor Query Performance
• Inaccurate estimates:
- Row Count
- Distinct Value Counts
• Schema Design:
- Normalized: specific better performance is seen with selective normalization e.g like
conditions
- Denormalized: selective denormalization is good for situations with skewed correlations
between joined tables.
• Query expressions less than ideal
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 10Customer
Query HTML Plans
• Query html plans components:
- IQ Version: SAP Sybase IQ version used to generate query plan
- DFO Tree: Tree representation of the query execution plan
- Query Timing Chart: Each operation has row providing details about the time being spent on that
operator during various query execution phases.
- Thread diagram: Provides information about thread usage for executing query
- CPU utilization diagram: Provides information about the CPU usage throughout the lifecycle of
query. Red represents System CPU usage and Green represents User CPU usage.
- Query Text: The Query SQL text.
- Query Plan Detailed Nodes: A HTML table for every query node including individual details for each
query plan node.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 11Customer
Query Timing Chart
Depicts time expended at each node
Wall clock
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 12Customer
Thread Utilization Diagram
Displays threads used over the life of the query
Thread usage corresponds to timing Chart
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 13Customer
CPU Usage Diagram
• CPU utilization diagram: Provides information about the CPU usage throughout the lifecycle
of query. Red represents System CPU usage and Green represents User CPU usage.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 14Customer
Query SQL TEXT
• Query Text: The Query SQL text.
Load into table
– create table fact (name varchar(255), minute timestamp, metric float) ;
– create table timedimension (minute timestamp) ;
– LOAD TABLE fact ("name" ',', "minute" ',', "metric") FROM '/work/data/fact.csv'
DELIMITED BY ',' ROW DELIMITED BY '\x0a' QUOTES ON ESCAPES OFF ;
– commit ;
"DBA"."timedimension"( "minute" ) using file '/work/data/timedimension.csv'
delimited by ',' row delimited by '\x0A' quotes on escapes off
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 15Customer
Important Query Plan Nodes
Root Leaf
Scrolling Cursor Store Join (12 types)
Filter Subquery
Group By (3 types) Union All
Order By Parallel Combiner
Store Parallel Order By
Semi-Join Filter filler
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 16Customer
Query HTML Plan
• Look for the areas where most of the execution time of spent.
- Optimization
- Vertical (index-based condition execution)
- Projection
- Horizontal (tuple-based) query operators
- Result transmission
- Waiting for requests
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 17Customer
Query Html Plan
Optimization Time:
Two phases of optimization:
1. Before vertical condition execution
2. Between the end of condition execution and the start of the first tuple fetch
Query timing diagram will show how long each of these phases last
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 18Customer
Query HTML Plan
Optimization Time:
Phase 2 of optimization includes:
- Join order selection
- Join and grouping algorithm selection
- Generation of the executable query tree
Phase 2 can in some cases be significant.
- Set Join_Simplification_Threshold = 9
- This can reduce join order selection costs, but it introduces a small risk that the optimizer may fail to
find the optimal join order.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 19Customer
Query HTML Plan
Vertical Condition Execution Time:
Time is shown by:
- The blue bar in the query timing diagram
- Query plan line labeled: “Elapsed Condition Time”
Includes all conditions labeled:
“Condition <N> (Invariant)”
Condition execution time is dependent upon:
- The conditions
- The indexes available, and the index chosen for each condition
- The order the conditions are executed
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 20Customer
Query HTML Plan
Vertical Condition Execution Time:
The optimizer generally chooses the optimal order to evaluate conditions, and the optimal indexes to use.
For conditions where no metadata is available the optimizer’s choice may be suboptimal.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 21Customer
Query HTML Plan
• To influence the optimizer’s choices for the execution of vertical conditions:
- Add indexes as suggested by the Index Advisor
- Supply a user-supplied condition selectivity
- Set the Index_Preference option
- Supply a condition hint which allows the user to specify for one condition:
- Selectivity
- Index preference
- Usefulness
- Execution phase
Example 1:
( (T.VCHAR1 LIKE ‘%summ%’, 10.50)
AND (T.VCHAR2 LIKE ‘%wint%’, ‘S:2.25’)
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 22Customer
Query HTML Plan
To override the optimizer’s choices for the execution of vertical conditions (cont’d):
Example 2:
SELECT CUSTOMER.*
FROM CUSTOMER, LOCATION
WHERE C_LOC_CODE = L_LOC_CODE
AND L_ZIPCODE = @hv1
AND (C_LAST_NAME LIKE @hv2, ‘E:D, U:1.0’)
A template query where the author knows an HPDJ algorithm will be used. The condition hint tells the optimizer to delay the execution of the LIKE condition until after the pushed join predicate is evaluated.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 23Customer
Query HTML Plan
Projection Time:
Visible via either:
- IQ performance monitor
- Operating system I/O statistics utilities
Not directly visible in query plans.
Pre-fetching generally means this issue is relevant only if the I/O bandwidth is saturated
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 24Customer
Query HTML Plan
Horizontal Query Operator Time:
The largest cost in most queries.
Visible in query timing diagrams as the yellow (first fetch) or green (subsequent fetches) bars.
Cost of each operator is affected by:
Number of tuples flowing through the operator
Number and width of tuples which must be copied, if any
Number of expressions or aggregates which must be evaluated for this operator
Whether the selected algorithm was able to get sufficient resources at run-time to operate efficiently
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 25Customer
Important Nodes
• Some Nodes contain a wealth of information
• Detail in other nodes is not as important
• Start with these Nodes when analyzing queries:
Root
Leaf
Join
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 26Customer
Root Node
The Root node is at the top of the
(inverted) query tree
Provides general information and
timings for the entire query
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 27Customer
Root Node
Root Node Details
• Result Rows (actual and estimated)
• User Name (login)
• IQ Temp Space Used (total)
• Active users on server
• Number of CPUs
• Cache sizes
• Threads used for all invariant predicates
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 28Customer
Query Timings in the Root node
Overall query timing is listed in the Root node• Query_Plan_After_Run and Query_Timing options must be ‘ON’ to report timings
• Total time to resolve all predicates is listed as ‘Elapsed Condition time’
Time of Cursor
Creation 2013-02-11 11:18:04.863474
End of DFE
Generation 2013-02-11 11:18:04.865834
End of DFO
Generation 2013-02-11 11:18:04.865937
End of DFO
Transforms 2013-02-11 11:18:04.866264
Time of First
Prepare 2013-02-11 11:18:04.871551
Time of First
Prepare Return2013-02-11 11:18:04.874527
Time of First
Fetch Call 2013-02-11 11:18:04.875013
Time of First
Fetch Return 2013-02-11 11:18:05.396055
Time of Second
Fetch Call 2013-02-11 11:18:05.396055
Time of Last
Fetch Return 2013-02-11 11:18:05.396056
Time of Complete
Call 2013-02-11 11:18:05.397327
Time of Complete
Return 2013-02-11 11:18:05.397328
Elapsed
Condition time0:00:00.000000
Working time 0:00:00.532583
Elapsed time 0:00:00.533854
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 29Customer
Scrolling Cursor Store node
Scrolling Cursor Store node usually found just below the Root Node• Affected by the database option Force_No_Scroll_Cursors (default ‘ON’)
The store buffers all rows from a query result set• Allows scrolling through results (forward and backwards)
– i.e. ‘Scrollable Cursor’
Consumes Temp Cache and Temp Store (for large results)• Typically this option is set OFF for better query performance
• Some query client and ETL tools may require this option be set ‘ON’
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 30Customer
Temp memory usage with Scrolling Cursors
Force_No_Scroll_Cursors = ‘OFF’ Force_No_Scroll_Cursors = ‘ON’
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 31Customer
Leaf node
All table indexes and columns used in the query are identified and accessed in the Leaf node
Types of Leaf nodes:
• (Regular) Leaf
• Aggregation Leaf
• Grouping Leaf
• Distincting Leaf
• Ordered Leaf
• Proxy Leaf (Proxy Table)
• SA Leaf (table in the SA database)
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 32Customer
Leaf node – a wealth of information
• All column metadata needed by the optimizer is identified in the Leaf node
- Row Counts
- Total rows in the table
- Estimated count after executing all conditions
- Generated count after executing all conditions
- Conditions (predicates) in the table showing:
- Selectivity
- Usefulness
- Index used to resolve the predicate
- Index Advisor Messages
- Predicate execution phase
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 33Customer
Leaf node details
Rows Counts
Total rows in the table
Generated rows after conditons (if any)
Conditions (Predicates)
Selectivity
Usefulness
Elapsed time (to execute)
Rows remaining (post execution)
Index used
Threads used
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 34Customer
Selectivity, Index Selection and Usefulness
For each predicate the optimizer determines
• Selectivity
- Portion (percentage) of the table rows that satisfy that predicate
- HG, LF and FP indexes provide this metadata
• Index used to resolve the predicate
- When multiple indexes exist, the best index for the predicate
- Cost and resources required influence index selection
• Usefulness Score
- The order the predicates have been executed
- Execution phase, cost and resources required influence Usefulness scoring
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 35Customer
Estimated Selectivity
Percent of table
Predicate Type (Estimated) Selectivity
Equality (=) 20% 0.2000000
Open Range (>) 40% 0.4000000
Between 40% 0.4000000
Like (%) 20% 0.2000000
Inter-column equality (t.a = t.b) 30% 0.3000000
Inter-column comparison (t.a < t.b) 50% 0.5000000
If no metadata exists the optimizer estimates selectivity based on the predicate operator
Selectivity with trailing zeroes may indicate an optimizer estimate
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 36Customer
Index-based selectivity
When a ‘useable’ index exists it can provide accurate selectivity
• Exact selectivity is shown as a precise number without trailing zeroes such as: 0.34982376
Note: Some functions can negate the ability to use an index for selectivity
Example: SUBSTRING( t.a, 5, 5 ) = ‘homes’
Optimizer might use default electivity20% for an equality search ( .20000000)
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 37Customer
Usefulness
Usefulness determines predicate execution order
Usefulness value ranges: 0.0 to 10.0 (10.0 = most useful)
• Predicate with highest score is executed first
• Remaining predicates executed in descending order of usefulness score
Factors determining Usefulness scoring
• Selectivity – how many rows in the table will be eliminated by this predicate
• Execution Phase
• Type of predicate (operator)
• Index available (including optimized FP index)
• Speed of predicate execution
• Resources required to execute the predicate
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 38Customer
Local predicate Execution Phase
Three categories of local predicates
Invariant
Delayed
Bound
- Category is based upon when the data required for predicate execution is available
- Predicate category usually dictates the phase (i.e. timing) when it is executed in the query
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 39Customer
Invariant predicates
• Invariant: expression does not change during query execution
- Most local predicates are invariant predicates
Example: WHERE cust_id = 12345
- Invariant predicates are usually executed in an early phase to ensure accurate row count
information is available to subsequent phases of query optimization
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 40Customer
Delayed predicates
• These are predicates that need to be executed once but require information that will become
available after another part of a query is executed:
Example:
WHERE cust_id in
(Select cust_id from customer where state = ‘VA’)
- The Where cust_id in predicate is ‘delayed’ since the subquery must be executed first
- Delayed predicates are typically found with uncorrelated subqueries (above) and some
“Push-Down” joins
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 41Customer
Bound predicates
• Bound predicates
- Need to be executed repeatedly and require information that will only be available after some other
portion of the query has been executed
• Bound predicates are commonly used in
- Correlated subqueries
- Nested-Loop Push-Down joins
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 42Customer
Predicate execution phases
• Local predicates are typically executed in this order -
1. Resolve all invariant predicates
2. Execute Delayed predicates
3. Execute Bound predicates
• Usefulness scores are related to a predicate execution phase
Note: Execution phase can be user influenced by using Optimizer Hints written into query code
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 43Customer
Query operators and indexes
• Predicates typically use these indexes with these operators
- Equality and Inequality( = , != ), IN and NOT IN lists
- Ranges ( <, >, <=, >=, Between) including NOT Between
- Like ( % ) or Not Like ( % ), Contains
LF and HG
DATE, HNG,
LF and HG
FP, WD,TEXT
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 44Customer
HG and LF indexes in predicates
• The HG and LF indexes provide exact metadata to the optimizer:
- The number of distinct values in the column and
- The number of rows found for each value
• These indexes also provide fast run-time access to the rows
- Directly influences predicate execution cost and execution phase
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 45Customer
FP indexes in predicates
• Optimized FP(Nbit) indexes (tokens) also provide metadata:
- Number of distinct values for a column
- Number of rows for each value
• Tokens dramatically reduce storage, disk I/O and memory
- Used with sorts to dramatically reduce memory
• FP index searches can use multiple threads for predicate execution
- Optimizer may prefer using FP index over other indexes since it can exploit many threads (and be
executed faster)
- IQ server work load at query time can influence this decision
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 46Customer
Column detail in the Leaf node
The Leaf Node lists all the columns accessed for the query
• Data type and indexes – no DATE index on this column
• With no DATE index the optimizer chose a multi-threaded FP search
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 47Customer
Index Advice in leaf nodes
Query tree highlights Leaf nodes in red indicating Index Advice is available in the node
Database option Index_Advisor must be set ON
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 48Customer
Other Indexes in predicates
• Except for special cases the other IQ indexes are used to locate values and do not provide
metadata
- Only the LF, HG and FP indexes provide metadata
• Some of the special cases are:
- CONTAINS predicate with a WD or TEXT index
- DATEPART function predicate on a column with a DATE index
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 49Customer
Other types of Leaf nodes
• Under the right circumstances optimizer uses an ‘Index Based’ algorithm to perform work within a node
- Examples where these algorithms are used:
Aggregation LeafSELECT COUNT(*) FROM ...
SELECT SUM(PAID_AMT) FROM …
Grouping Leaf
SELECT MEMBER_GENDER, COUNT(*)
FROM …
GROUP BY MEMBER_GENDER
Distincting LeafSELECT DISTINCT (shipdate)FROM …
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 50Customer
Leaf node(summary)
• Leaf nodes provide a wealth of information about the data and indexes used in a query
• List of all table predicates including:
- Selectivity
- Usefulness
- Execution phase
- Index used to resolve predicate
• All columns referenced, including:
- Available indexes
- Data type
- Distinct count
• Index advice
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 51Customer
Join nodes
• Join nodes merge a set of rows from one table with rows from another table passing all
combinations which satisfy the join condition(s)
• For joins the optimizer must determine two things:
- Join order (if more than 2 tables)
- Join algorithm (12 types)
• For these decisions the optimizer uses a cost-based process
- Relies heavily on the metadata for the join columns
- Metadata comes from indexes in Leaf nodes
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 52Customer
Metadata for join performance
Tips for providing the right metadata
• Individual columns which are primary keys always declare as a PRIMARY KEY
(or create a UNIQUE HG index)
- Candidate key columns create a UNIQUE HG index if used in joins
• All other join columns create an HG (or LF) index
• Multi-column primary keys where tables will be joined using all the keys create a multi-
column PRIMARY KEY (or a multi-column UNIQUE HG index)
- If all keys are NOT used in a Join then a Primary Key (or multi-column Unique index) is not
necessary
- For example, the Primary Key for a Fact table would almost never be necessary (except to maintain
entity integrity)
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 53Customer
Join algorithms
• IQ has 12 different table join algorithms
• Valid algorithms for a given join depend upon:
- Type of join condition
- Data types on the join columns (matched or mismatched)
- Outer join (if so which kind)
- Indexes on the join columns
- Join constraint (Many to Many, One to Many)
• Valid algorithms are compared against each other based on their estimated costs and
available server resources
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 54Customer
Join Algorithms
• Knowledge of the join algorithms, their strengths and weaknesses, may allow you to identify
ways you could change the query or the schema to improve query performance
• Each join algorithms has different advantages:
- Some are better for joining large tables to small tables
- Some are better for joining two equally large tables
- Some require large amounts of Temp Cache be available
- Some require large amounts of Main Cache be available
- Some are good only for very selective joins
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 55Customer
Types Of Join Algorithms
Sort-merge Join Partitioned Hash Join
Nested-Loop Join Partitioned Hash-Push Down Join
Nested-Loop Push-Down Join Partitioned Sort-Merge Join
Hash Join Partitioned Sort-Merge Push-Down
Hash Push-Down Join Sort-Merge Push-Down Join
Asymmetric Sort-Merge Join Asymmetric Sort-Merge Push-Down Join
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 56Customer
Inaccurate Leaf Node Estimates
Semi-join predicate effects poorly estimated
. . #01: Leaf
Generated Result Rows: 58590
Estimated Result Rows: 30
Pre-PushDown Est. Result Rows: 60175
Table Name: lineitem
Table Row Count: 60175
Condition 1 (Pushed): (lineitem.l_partkey IN BV(1))
Condition 1 Selectivity: 0.00050000
Condition 1 Usefulness: 5.99950000
Condition 1 Execution Method: Vertical Parallel Hash in FP
Condition 1 Index: FP lineitem.DBA.ASIQ_IDX_T205_C2_FP
Condition 1 Actual Distincts: 1949
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 57Customer
Inaccurate Leaf Node Estimates
Semi-join predicate (cont’)
. #03: Join (Hash PushDown)
Left Child Node: #01
Right Child Node: #02
Generated Result Rows: 58590
Estimated Result Rows: 31
Valid Join Algorithms: NLJ, NLPDJ, SMJ, SMPDJ, HJ, HPDJ
Left Input Table 1: lineitem
Estimated Left Inputs: 30
Estimated Left Pre-Pushdown Inputs: 60175
Right Input Table 1: part
Estimated Right Inputs: 1
Join Result Constraint: Many to 1
Condition 1: (lineitem.l_partkey = part.p_partkey)
Condition 2 (Pushed): (lineitem.l_partkey IN BV(1))
Optimizer est. max hash rows for these keys: 27115
Hash actual number of keys: 1949
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 58Customer
Inaccurate Group By Estimates
GROUP BY result estimates depend on both:
- Incoming row estimates.
- Estimates of the number of distinct values for each of the grouping expressions after leaf filtering.
- Hint 1: Look at each grouping column for a missing index:
Column 1 Base Distincts (Est.): 412
- Hint 2: Look for missing multi-column primary key definitions
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 59Customer
Inaccurate Join Estimates
• Join result estimates depend on:
- Incoming row estimates.
- Estimates of the number of distinct values for each of the simple equality join expressions.
The primary and foreign key information
Hint 1: Look at each join column for a missing index:
Column 1 Base Distincts (Est.): 412
Hint 2: Look for missing multi-column primary key definitions,
Join Result Constraint: Many to Many
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 60Customer
CheckList
• Missing Indexes
• Missing IQ Unique
• Mismatched Datatypes
• Sub-Optimal Datatype Choice
• Review Join Conditions
• Check for Over-Aggressive Push-Downs
• Check for Cache Thrashing
• Excessive Data Skew
• Missing Referential Constraint
• Thread Starvation
• Excessive Expression Evaluations
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 61Customer
Missing Indexes
• Check for: Base Distincts (Est.)
- Only concentrate on this line if the optimizer needed the distinct count
- Requires LF, HG or have flat N-bit
• Wide-range predicates on numbers requires HNG
• Infra-table column comparisons require CMP index
• Selectivity ‘magic’ numbers
- Check for selectivity e.g 0.2000000
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 62Customer
Missing IQ Unique
• Look for ‘Projection Bytes’ in Leaf node for low cardinality columns
• Each release of IQ adds more algorithms that can be pushed into the enumeration table
• Can make large performance difference for large tables
• Cannot be added by Alter Table
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 63Customer
Mismatched Datatypes
• IQ assumes that in a data warehouse time was taken to design the schema before
implementing it, including using consistent data types for all join key columns
• Some more efficient join algorithms cannot be used with mismatched datatypes
(signed/unsigned, char/varchar length etc)
• May cause less efficient algorithms to be used
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 64Customer
Suboptimal Datatype Choice
• Common cause of sub-optimal performance is not using optimal data types
• All tables should have keys, and keys should be an integral data type, and preferably
unsigned
• Don’t use CHAR or VARCHAR columns as keys for joins, ORDER BYs, or GROUP BYs if at all
possible
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 65Customer
Review Join Conditions
• Look for: Join (Nested Loop)
• Optionally followed by a Filter node
• May be a missing join condition
• May be a join condition that precludes using an efficient join algorithm
• Disjunctions (<condition> OR <condition>)
• Range predicates (<, >, Between, Like)
• Complex expressions (T1.x + T2.y = 5)
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 66Customer
Over-Aggressive Push-Downs
• Duplicate predicate selectivity
• NLPD: Especially sensitive to too many pushes
• Meaning of semi-join in plan: actually IN cost estimates
SemiJoin V: 2 D: 77446.5 T: 0 A: ORs S: 0 O: 0
V – Actual # distinct values for the IN
D – Est. # rows per distinct value
T – Actual time taken
A – Algorithm chosen: bitmap OR or column scan
S – Est. cost using column scan
O – Est. cost using bitmap disjunction
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 67Customer
Cache Thrashing
• Hash thrashing
- Happens when significant under-estimate
- Causes excessive reads and writes from the temp cache
- Solution: fix estimates or use temp table or avoid hash
• Correlated subquery thrashing
- Causes excessive reads and writes from the main cache
- Solution: rewrite the query as a join instead of a subquery
• Confirm using IQ buffer cache monitor report
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 68Customer
Excessive Data Skew
• IQ assumes modest data skew
• Sometimes data is heavily skewed
- Mostly affects push-down joins
- No simple way to work around
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 69Customer
Missing Referential Integrity Constraint
• multicolumn primary keys must be defined
• In joins, where it affects estimates, look for:
• Join Result Constraint: Many to Many
• In a GROUP BY it affects performance directly by forcing the engine to sort or hash on extra
columns.
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 70Customer
Thread Starvation
• Configured by startup Switch:
- Total threads in server -iqmt
- Set Max_IQ_Threads_Per_Connection
- Can generally be set safely up to 100
• Examine using IQ cache monitor with -debug
• Look for ‘Serial’ in Query_Plan_After_Run on UNION, and non-push-down join nodes
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 71Customer
Excessive Expression Evaluations
• Look at production lists:
Production 1: (T.B * T.C)
Production 2: (T.A * (T.B * T.C))
• Seeing a CAST in a production may be a symptom of mismatched data types
• Finding nodes with very high row counts that are producing complex expressions may imply a less than optimal schema design
- Consider creating a derived column to hold a pre-computed expression result for each commonly used expression
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 72Customer
Common Links For The Information
• Product documentation
http://help.sap.com/iq
• SAP IQ Performance and Tuning
• Support Portal
https://support.sap.com/home.html
• How to get best results from an SAP search
https://service.sap.com/sap/support/notes/2081285
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 73Customer
Common Links For the Information
• SAP IQ communties
http://scn.sap.com/community/iq
• SAP IQ Wiki
http://wiki.scn.sap.com/wiki/display/SYBIQ/IQ
• IQ 16 Best Practices
http://scn.sap.com/docs/DOC-39896
• IQ 16 Hardware Sizing Guide
http://scn.sap.com/docs/DOC-41455
© 2015 SAP SE or an SAP affiliate company. All rights reserved. 74Customer
Common Links For the Information
• Social Media Product Support Channels
https://twitter.com/SAPSupporthelp
https://www.facebook.com/SapProductSupport