8/8/2019 Database design and Implementation 09.queryopt
1/17
Database Systems Implementation, Bongki Moon 1
Relational Query Optimization
Ramakrishnan&Gehrke: Chap. 15.1-15.4
Selinger et al.: ACM SIGMOD79
Database Systems Implementation, Bongki Moon 2
Overview
Queries (SQL) are declarative; No informationgiven as to how to process queries.
Query Optimizer needs to find efficient plans toprocess:processing order and access methods.
Framework: Transforms an SQL query into a relational algebra.
Enumerates equivalent evaluation plans. (E.g.) early selection, early projection, etc.
Estimates the costs and select the best one.
8/8/2019 Database design and Implementation 09.queryopt
2/17
Database Systems Implementation, Bongki Moon 3
Main Issues
Two main issues from the framework: For a given query, what plans are considered?
We do not want to consider every single possibility.
Needs heuristics to limit the search space, and algorithms tofind the best plan.
How is the cost of a plan estimated? Use statistics about relations and cost formulas.
We will discuss
The System R Query Optimizer [Selinger, SIGMOD79]
Database Systems Implementation, Bongki Moon 4
Schema for Motivating Examples
Reserves:
Each tuple is 40 bytes long, 100 tuples per page.
TR=100,000 tuples, BR=1000 pages.
Sailors:
Each tuple is 50 bytes long, 80 tuples per page.
TS=40,000 tuples, BS=500 pages.
Sailors (sid: integer, sname: string, rating: integer, age: real)Reserves (sid: integer, bid: integer, day: dates, rname: string)
8/8/2019 Database design and Implementation 09.queryopt
3/17
Database Systems Implementation, Bongki Moon 5
Plans for a Sample Query
SQL Query:SELECT S.sname
FROM Reserves R, Sailors S
WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5
Algebraic plans
S.sname(R.bid=100(S.rating>5(R S))) : Plan 1
S.sname((R.bid=100R) (S.rating>5S)) : Plans 2 & 3
S.sname(S.rating>5((R.bid=100R) S)) : Plans 4 & 5
Database Systems Implementation, Bongki Moon 6
Algebraic Equivalences
Allow us to choose different join orders and to`push selections and projections ahead of joins.
Selections: (Cascade)( ) ( )( ) c cn c cnR R1 1 ... . . .
( )( ) ( )( ) c c c cR R1 2 2 1 (Commutative)
Projections: ( ) ( )( )( ) a a anR R1 1 . . . (Cascade)
Joins: >< >< >< (Commutative)
R (S T) (T R) S Show that: >< >< >< >
8/8/2019 Database design and Implementation 09.queryopt
4/17
Database Systems Implementation, Bongki Moon 7
Algebraic Equivalences (2)
Selection & Projection:
If the condition C involves only attributes A1,A2,,An,
Selection & Cartesian-product:
if all the attributes of C are in R,
if all the attributes of C1 are in R and those of C2 are in S,
))(())((,...,2,1,...,2,1
RRAnAACCAnAA
SRSRCC
)()(
)()()(2121
SRSRCCCC
Database Systems Implementation, Bongki Moon 8
Algebraic Equivalences (3)
Selection & Join:
if all the attributes of C are in R,
if all the attributes of C are in
Projection & Join: if A includes join attributes,
SRc
SRc
)()(
)()()( SRSRCCC
)()()(21
SRSRAAA
SR
8/8/2019 Database design and Implementation 09.queryopt
5/17
Database Systems Implementation, Bongki Moon 9
Example: Plan 1 (Nave)
Cost: BS + BS*BR =500+500*1000 (Page I/Os)
By no means the worst plan!
SELECT S.snameFROM Reserves R, Sailors SWHERE R.sid=S.sid AND
R.bid=100 AND S.rating>5
Reserves Sailors
sid=sid
bid=100 rating > 5
sname
(Simple Nested Loops)
(On-the-fly)
(On-the-fly)Plan 1:
Database Systems Implementation, Bongki Moon 10
Example: Plan 2 (Push Selection)
T1 = bid=100R; T2 = rating>5S;
To read R and write T1, BR+fRBR.
To read S and write T2, BS+fSBS.
If fRBR=10 and fSBS=250, the cost to join
T1 and T2 is fRBR+fRBR*fSBS.
Total cost is (1000+10) + (500+250) +(10+10*250) = 4270 (page I/Os).
By pushing projections, we can makeT1and T2 smaller: T1(sid), T2(sid, sname).
Reserves Sailors
sid=sid
bid=100
sname(On-the-fly)
rating > 5
(Simple NestedLoops)
)(21
TTsidsidsname =
8/8/2019 Database design and Implementation 09.queryopt
6/17
Database Systems Implementation, Bongki Moon 11
Example: Plan 3
(Push Selection)
Use Sort-MergeJoin instead of Simple Nested-Loop Join.
With 5 buffers, cost of plan: To read R and write T1, BR+fRBR = 1000 + 10.
To read S and write T2, BS+fSBS = 500 + 250.
To sort T1 by 2-pass merge, 2*10 * 2 (read and write).
To sort T2 by 4-pass merge, 4*250 * 2 (read and write). To read T1 and T2, 10 + 250.
Total: 4060 page I/Os.
Reserves Sailors
sid=sid
bid=100
sname(On-the-fly)
rating > 5(Scan;write totemp T1)
(Scan;write totemp T2)
(Sort-Merge Join)
Database Systems Implementation, Bongki Moon 12
Example: Plan 4 (With Indexes)
Clustered hash index on bid of Reserves.
Clustered hash index on sid of Sailors.
Use Indexed Nested-LoopJoin with Pipelining.
1000 tuples (10 pages) are selected from R via index.
For each tuple from R, access S via hash index on sid; atmost one tuple is fetched from S (sid is key of S).
About 1.2 pages are accessed from I(S) (overflow).
Cost: 1.2 (I(R): Rs index page)+10 (pages of
R)+(1.2+1)*1000 (pages of I(S) and S) = 2211 (pages). If you push rating>5 down to Sailors, you cannot use
the index for S.Reserves
Sailors
sid=sid
bid=100
sname
(On-the-fly)rating > 5
(Index NL)
(On-the-fly)
8/8/2019 Database design and Implementation 09.queryopt
7/17
Database Systems Implementation, Bongki Moon 13
Example: Plan 5 (With Indexes)
Clustered hash index on bid of Reserves.
Clustered hash index on sid of Sailors.
Use Indexed Nested-LoopJoin T1 = bid=100 R; T1 has 1000 tuples (10 pages)
but the number of unique sid values may bemuch smaller.
Sort T1 by sid values; then the number ofpages fetched from S is the number of uniquesid values in T1.
Cost: (Cost to compute and sort T1) + 10
(pages of T1) + (1.2+1)*(unique sid valuesin T1).
Reserves
Sailors
sid=sid
bid=100
sname (On-the-fly)
rating > 5
(Index NL)
(On-the-fly)
(Write to T1;Sort by sid)
Database Systems Implementation, Bongki Moon 14
Result Size Estimation
For each plan considered,
Must estimate the size of result from each operator in theplan tree. Use information about the input relations.
Usually assume uniform distribution of values, independence ofpredicates.
The intermediate result from an operator will become
input to the next operator in the plan tree.
8/8/2019 Database design and Implementation 09.queryopt
8/17
Database Systems Implementation, Bongki Moon 15
Result Size Estimate: Cartesian Product
Size of R S, BRS = BRTS+TRBS . The length of an R S tuple is B/pR + B/pS.
The number of R S tuples per page is
pRS = B / (B/pR + B/pS) = (1/pR + 1/pS)-1.
Therefore,
BRS = TRS / pRS = TR TS (1/pR + 1/pS) = BRTS+TRBS.
Database Systems Implementation, Bongki Moon 16
Result Size Estimate: Join
Assumptions IR and IS are the domain sizes of join attributes of R and S.
Assume domain(Rj) domain(Sj) or vice versa.
Size of R S, BRS = (BRTS+TRBS)/max(IR,IS). The number of R S tuples per page is pRS = (1/pR + 1/pS)
-1.
The total number of tuples in R S is TRS = TRTS /max(IR,IS).
For each join attribute value x common in R and S, TR/IR tuples in R andTS/IS tuples in S are expected to have x as their join attribute value.
Since there are min(IR,IS) common values like x,
TRS = (TR/IR) (TS/IS) IR = TRTS /max(IR,IS).
Therefore, BRS = TRS / pRS = TRTS (1/pR + 1/pS)/max(IR,IS)
= (BRTS+TRBS)/max(IR,IS).
8/8/2019 Database design and Implementation 09.queryopt
9/17
Database Systems Implementation, Bongki Moon 17
Query Optimization: Overview
Assumptions
Users need not know physical structures and access paths.
Users do not specify join orders.
What Optimizer does:
Look up catalog for statistics (e.g., data type & length ofcolumn).
Perform access path selection for single-relation queryblock.
Determine join order and join methods. Determine evaluation order among query blocks.
Change the structure of parse trees.
Database Systems Implementation, Bongki Moon 18
System R Optimizer
Cost estimation: Approximate art at best. Statistics, maintained in system catalogs, used to estimate cost of
operations and result sizes.
Considers combination of CPU and I/O costs.
Plan Space: Too large, must be pruned. Only the space of left-deep plans is considered.
Left-deep plans allow output of each operator to bepipelined into the nextoperator without storing it in a temporary relation.
Intermediate results are physically stored only if a sort is required for thenext join step.
Cartesian products avoided.
Impact: Most widely used currently; works well for < 10 joins.
8/8/2019 Database design and Implementation 09.queryopt
10/17
Database Systems Implementation, Bongki Moon 19
Statistics and Catalogs
Need information about the relations and indexes involved.Catalogs typically contain at least: TR=# tuples and BR=# pages for each relation R.
KI=# distinct key values and BI=# pages for each index I.
dI=index depth, lI/hi=low/high key values for each tree index I.
Catalogs updated periodically. Updating whenever data changes is too expensive; lots of
approximation anyway, so slight inconsistency ok.
Contemporary DBMSs useHistograms to store more detailed statistics. Frequency distribution of attribute values in a relation
Equi-depth, equi-width, serial, biased, end-biased histograms, etc.
Database Systems Implementation, Bongki Moon 20
Single-Relation Plans: Selectivity
SARG-able (Search Argument) predicates: Can determine qualification of individual tuples. Relation.column comparison-op constant (e.g., salary > 50K)
Selectivity factor (SF) associated with each sargable predicate reflects itsimpact on reducing result size. col=value
SF = 1/KI, given index I on col, SF = 1/10 o/w.
col1=col2 SF = 1/max(KI1,KI2), given index I1 and I2, SF = 1/10 o/w.
col>value SF = (hR-value) / (hR- lR) if arithmetic type, SF = 1/3 o/w.
val1
8/8/2019 Database design and Implementation 09.queryopt
11/17
Database Systems Implementation, Bongki Moon 21
Selectivity for Conjunctive Predicates
Assume that sargable predicates are independent.
For a query with a CNF of N sargable predicates,
SF1^2^^N= SF1 * SF2 * * SFN Result cardinality = TR * (SF1 * SF2 * * SFN)
(E.g.) If SF(age=30)=10% and SF(dept=CS)=5%, thenSF(age=30 and dept=CS) = 10% * 5% = .5%.
Database Systems Implementation, Bongki Moon 22
Single-Relation Plans: Cost Estimates
I/O cost for selection operations with index Primary key equality-match selection (unique)
dI+1 (for a B+ tree) or 1.2+1 (for hash index).
With clustered index matching one or more (BI+ BR) * selectivity-factors
With non-clustered index matching one or more
(BI+ TR) * selectivity-factors
System R considers both I/O and CPU costs Cost formula=I/O (index + data) + w*(per-tuple CPU).
8/8/2019 Database design and Implementation 09.queryopt
12/17
Database Systems Implementation, Bongki Moon 23
Example
Result size: (1/KI) * TR = (1/10) * 40000 = 4000 tuples Plan 1: If we use an index on rating:
Clustered index: (1/ KI) * (BI + BR) = (1/10) * (50+500) = 55 pagesare retrieved (assuming SF = 1/10).
Unclustered index: (1/ KI) * (BI + TR) = (1/10) * (50+40000) = 4005pages are retrieved.
Plan 2: If we use an index on sid: Would have to retrieve all tuples/pages. With a clustered index,
the cost is BI + BR = 50+500, with unclustered index, BI + TR =50+40000.
Looks useless, but the resulting tuples may be in an interestingorder, which may be useful for the later stages of processing.
Plan 3: Do a file scan: Retrieve all file pages (500).
SELECT S.sidFROM Sailors SWHERE S.rating=8
Database Systems Implementation, Bongki Moon 24
Multiple-Relation Plans
Fundamental decision in System R: only left-deep join treesare considered. As the number of joins increases, the number of alternative plans
grows rapidly; we need to restrict the search space.
Left-deep trees allow us to generatefully pipelined plans.
Intermediate results not written to temporary files.
Not all left-deep trees are fully pipelined (e.g., SM join).
BA
C
D
BA
C
D
C DBA
8/8/2019 Database design and Implementation 09.queryopt
13/17
Database Systems Implementation, Bongki Moon 25
Enumeration of Left-Deep Plans
Left-deep plans differ only in the order of relations, the access method foreach relation, and the join method for each join. (That is, N!join orders forN relations.)
Once the first k relations are joined, the method to join the result to the(k+1)th relation is independent of the order of joining the first k relations.
Dynamic Programming with N passes. Pass 1: Find best 1-relation plan for each relation.
Pass 2: Find best way to join result of each 1-relation plan (as outer) toanother relation. (All 2-relation plans.)
Pass N: Find best way to join result of a (N-1)-relation plan (as outer) to theNth relation. (All N-relation plans.)
For each subset of relations, retain only: Cheapest plan overall, plus
Cheapest plan for each interesting orderof the tuples.
Database Systems Implementation, Bongki Moon 26
Enumeration of Plans (Contd.)
Avoid Cartesian products if possible. For example, if there is no join predicate between R1 and R3, the
following orders are not considered: R1 - R3 - R2, R3 - R1 - R2.
ORDER BY, GROUP BY, aggregates etc. handled as a finalstep, using either an interestingly ordered plan or anadditional sort operator.
Interesting orders if sort-merge join is considered or
if a query contains ORDER-BY or GROUP-BY
In spite of pruning plan space, this approach is stillexponential in the # of relations.
8/8/2019 Database design and Implementation 09.queryopt
14/17
Database Systems Implementation, Bongki Moon 27
Example
Pass1:
Sailors: B+ tree matches rating>5, and is probablycheapest. However, if this selection is expected toretrieve a lot of tuples, and index is unclustered, filescan may be cheaper. Still, B+ tree plan kept (because tuples are in rating
order).
Reserves: B+ tree matches bid=100; cheapest.
Sailors:B+ tree on ratingHash on sid
Reserves:B+ tree on bid
Pass 2:
We consider each plan retained from Pass 1 as the outer, and consider
how to join it with the (only) other relation. e.g., Reserves as outer: Hash index can be used to get Sailors tuples
that satisfy sid = outer tuples sid value.
Reserves Sailors
sid=sid
bid=100 rating > 5
sname
Database Systems Implementation, Bongki Moon 28
Example Database and Query
8/8/2019 Database design and Implementation 09.queryopt
15/17
Database Systems Implementation, Bongki Moon 29
Pass 1: Best Single-Relation Plan
Database Systems Implementation, Bongki Moon 30
Pass 2: Best 2-Relation Plan (NL Join)
8/8/2019 Database design and Implementation 09.queryopt
16/17
Database Systems Implementation, Bongki Moon 31
Pass 2: Best 2-Relation Plan (SM Join)
Database Systems Implementation, Bongki Moon 32
Pass 3: Best 3-Relation Join
8/8/2019 Database design and Implementation 09.queryopt
17/17
Database Systems Implementation, Bongki Moon 33
Query Opt Issues Not Covered
Optimization of nested query blocks Nested blocks may or may be correlated.
Flattening a correlated nested query, etc.
Optimization of GROUP-BY, aggregation Early, late, stagewise groupings, etc.
Multiple Query Optimization
Histograms
Optimality of histograms, Multidimensional histograms, etc. And much more
Top Related