Reducing Order Enforcement Cost in Complex Query Plans
Ravindra Guravannavar and S. Sudarshan(To appear in ICDE 2007)
2
Background Sort-based query processing algorithms
Sort-merge Join (also Union/Intersection) Sort-based grouping and duplicate elimination
Explicit “order by” Notion of “Interesting Sort Orders” (System-R)
Find and remember the best plan for each sort order that may be useful
Optimization goal in Volcano : (expr, sort-order)
3
The Problem Interesting orders can be too many!
Factorial in number of attributes involved Plan cost can vary substantially with the
choice of interesting order Clustering and covering indices Other operators in the input sub-expressions Possibility of partial sorting
G Group By {a2,a4,a5,…}
R S
R.a1=S.a1 and R.a2=S.a2 … R.an=S.an
4
Motivation Joins in data integration and decision support
involve large number of attributes Increasing use of covering indices
Several alternative sort orders Partial sorting
Query patterns Attributes common to multiple operators
Known techniques Work only for unary operators like group-by
5
Outline of the Talk Partial sorting
Changes to external sort Optimizer changes to handle partial sort orders
Interesting orders for a join tree : A special case Problem is NP-Hard A 2-approximation for the special case
The general problem Notion of favorable orders Plan generation using favorable orders Post-optimization phase
Experimental results
6
Exploiting Partial Sort Orders
Sort on (a1, a2) given (a1) Standard external-sort
Cost is independent of input sort order Replacement-selection
Produces single run but incurs I/O Both methods break the pipeline – first o/p tuple
after reading all i/p
R S
R.a1=S.a1 and R.a2=S.a2
C. Index on (R.a1)
(a1) (a1,a2) () (a1,a2)
7
A Minor Change to External Sorting
Multiple “partial sort segments” Hold only one segment at any given
time When a new segment starts
Sort the current segment and output
No run generation I/O if each segment fits in memory
Early output (good for Top-K) Reduced comparisons
O(n log n/k) Vs. O(n log n), k = # segments
a1 a2
1 2
1 1
1 5
1 3
2 4
2 1
2 6
6 3
… …
8
Optimizer Changes to Handle Partial Sort Orders Cost Model for Partial Sort:
Let the input order be o1
Required (output) order be o2
Let os=Longest common prefix between o1 and o2
Let or=o2 – os (i.e, os + or = o2) A(o) = Attribute set of order o Є : Empty (no) sort order
coe(e, o1,o2) = D(e, A(os)) X coe(e’, Є, or), where e’=p(e) and p equates A(os) to a constant.
9
Optimizer Changes to Handle Partial Sort Orders Cost Model for Partial Sort:
coe(e, o1,o2) = D(e, A(os)) X coe(e’, Є, or), where e’=p(e) and p equates A(os) to a constant.
o1=(a,b)
o2=(a,c)
os=(a), or=(c), e’=(a=k)(e)
e
10
Flexible Order Requirements Most operators have interest in any order on the
attributes involved Merge-Join, Merge-Union, Group By, Duplicate Elimination Binary operators demand the same order from inputs
G {a1, a2}
{a1,a2,a3,a4}
{a4,a7}{a3,a5,a6}
11
Finding Optimal is NP-Hard A special case:
All relations/intermediate results of the same size
All attribute cardinalities same
We try to maximize the length of common prefixes
Maximize LCP(pi, pj)
Reduction from graph layout problem SUM-CUT Optimal algorithm for paths and 2-approximation for binary trees
12
A 2-Approximation Algorithm Optimal algorithm for paths
s2s1 sns3 Sn-1
OPT(i,j) = max {OPT(i,k) + OPT(k+1,j) + c(i,j)}, i ≤ k < j
2-Approximation for binary trees
- OPT ≤ OPT-EVEN + OPT-ODD- Take the one with higher benefit
Even levelsOdd levels
13
General Case Logical plan space for inputs not expanded
(i.e, Join order not fixed)
Varying sizes of relations and intermediate results
All orders on base relations do not have the same cost (due to clustering and covering indices)
14
Overview of the Approach Identify a small set of favorable orders
Orders that are relatively inexpensive Should not require expanding the input plan space
Plan generation (Phase-1) Deduce the interesting orders from the favorable
orders Try each of the interesting order, retain the best
Plan refinement (Phase-2) Use the 2-approximation algorithm and refine the
sort orders further
15
Favorable Orders Benefit of an order:
benefit(o, e) = cbp(e, Є) + coe (e, Є, o) – cpb(e,o)Positive benefit The order can be obtained at cost
less than the full sort of unordered result (e.g., the
clustering order)
Favorable orders:ford(e)={ o : benefit(o,e) > 0 } Can be a huge set E.g., Every order having the clustering order as its
prefix is a favorable order.
16
Minimal Favorable Orders A favorable order o that satisfies:
1. o’ ≤ o s.t. cbp(e, o’) + coe(e, o’, o) = cbp(e,o)
2. o” s.t. o ≤ o” and cbp(e, o”) = cbp(e,o)E.g., Relation R with clustering index on (a1,a2)
(a1,a2) is a minimal favorable order
(a1 ), (a1,a2,a3) are not
ford-min(e): Set of all minimal favorable orders for expression e
For base relations size of ford-min limited to the number of covering indices
E
E
17
Computing Favorable Orders: Issues Defined in terms of cost of best plan
Need them before optimizing input sub-expressions Even ford-min can get prohibitively large for
join, group-by expressions
R S
J1
J2
ford-min contains every permutation of
the join attributes
18
Heuristics for Computing ford-min
e=R : {o: o is clustering or covering index order}
e=p(e1) : {o: o ford-min(e1)}
e=L(e1) :{o: o’ ford-min(e1) and o=o’ ^ L} a,b(e1), ford-min(e1)={(a,c,b)} ford-min(e)={(a)}
e=e1 e2 : Let T=ford-min(e1) U ford-min(e2) T U {o: o’ T and o=((o’ ^ S) permute(S – A(o’ ^ S)))
UU
U
19
Heuristics for Computing ford-minS={a,b,c,d}
ford-min={(a,b,e),(b)} ford-min={(a)}
T = {(a,b,e), (b), (a)}
Input F.Order (o) o ^ {a,b,c,d} Extended Order
(a,b,e) (a,b) (a,b,c,d)
(b) (b) (b,a,c,d)
(a) (a)
20
Plan Generation (Phase-1) Form the set I of interesting orders to try
Collect input favorable orders and rqd. o/p order Take LCP with the set of join attributes Extend the orders (arbitrarily) to include remaining
attributes For each order o in I, generate optimization
sub-goals for input sub-expressions
21
Plan Refinement (Phase-2)
Identify the suffix that can be freely reordered Use the 2-approximation algorithm to reorder
the suffix
R2(a)
(a,b,c,h)
(a,d,h)
R4(a)
R3(a)
R1(a)
(a,e,h){a,d,h} {a,e,h}
{a,e,h}
(a,h,e)
(a,h,b,c)
(a,h,d)
22
Experiments1. Benefits of exploiting partial sort orders
2. Evaluate the plans produced by our optimizer extensions
Systems Compared
PostgreSQL 8.1.3, SQLServer 2005,
DB2 8.2, PYRO
Test Machine Intel P4 (HT) PC, 512 MB
Dataset TPC-H 1GB and synthetic
Queries Synthetic and from a real application
23
Experiment 1SELECT suppkey, partkey FROM lineitem
ORDER BY suppkey, partkey;
(suppkey) (suppkey, partkey)
24
Experiment 2
R(c1,c2,c3), 10 M records, (c1)(c1,c2), card(c1)=10,000
25
Experiment 3
26
Experiment 4
SELECT ps_suppkey, ps_partkey, ps_availqty,
sum(l_quantity) AS total_required
FROM partsupp, lineitem
WHERE ps_suppkey=l_suppkey AND ps_partkey=l_partkey
AND l_linestatus='O'
GROUP BY ps_partkey, ps_suppkey, ps_availqty,
HAVING sum(l_quantity) > ps_availqty
ORDER BY ps_partkey;
Parts running out of stock:
27
Experiment 4 - Plans
Merge-Join Plan on SYS1 and SYS2 Plan Generated by PYRO-O
28
Experiment 4 & 5 - Timings
29
Experiments with Variants of PYRO
PYRO : Baseline PYROPYRO-O-: No partial sortPYRO-P : Postgres HeuristicPYRO-O : Our ApproachPYRO-E : Exhaustive
30
Optimization Overheads
31
Questions?
Top Related