Database design and Implementation 09.queryopt

8/8/2019 Database design and Implementation 09.queryopt

1/17

Database Systems Implementation, Bongki Moon 1

Relational Query Optimization

Ramakrishnan&Gehrke: Chap. 15.1-15.4

Selinger et al.: ACM SIGMOD79


Overview

Queries (SQL) are declarative; No informationgiven as to how to process queries.

Query Optimizer needs to find efficient plans toprocess:processing order and access methods.

Framework: Transforms an SQL query into a relational algebra.

Enumerates equivalent evaluation plans. (E.g.) early selection, early projection, etc.

Estimates the costs and select the best one.


2/17


Main Issues

Two main issues from the framework: For a given query, what plans are considered?

We do not want to consider every single possibility.

Needs heuristics to limit the search space, and algorithms tofind the best plan.

How is the cost of a plan estimated? Use statistics about relations and cost formulas.

We will discuss

The System R Query Optimizer [Selinger, SIGMOD79]


Schema for Motivating Examples

Reserves:

Each tuple is 40 bytes long, 100 tuples per page.

TR=100,000 tuples, BR=1000 pages.

Sailors:

Each tuple is 50 bytes long, 80 tuples per page.

TS=40,000 tuples, BS=500 pages.

Sailors (sid: integer, sname: string, rating: integer, age: real)Reserves (sid: integer, bid: integer, day: dates, rname: string)


3/17


Plans for a Sample Query

SQL Query:SELECT S.sname

FROM Reserves R, Sailors S

WHERE R.sid=S.sid AND R.bid=100 AND S.rating>5

Algebraic plans

S.sname(R.bid=100(S.rating>5(R S))) : Plan 1

S.sname((R.bid=100R) (S.rating>5S)) : Plans 2 & 3

S.sname(S.rating>5((R.bid=100R) S)) : Plans 4 & 5


Algebraic Equivalences

Allow us to choose different join orders and to`push selections and projections ahead of joins.

Selections: (Cascade)( ) ( )( ) c cn c cnR R1 1 ... . . .

( )( ) ( )( ) c c c cR R1 2 2 1 (Commutative)

Projections: ( ) ( )( )( ) a a anR R1 1 . . . (Cascade)

Joins: >< >< >< (Commutative)

R (S T) (T R) S Show that: >< >< >< >


4/17


Algebraic Equivalences (2)

Selection & Projection:

If the condition C involves only attributes A1,A2,,An,

Selection & Cartesian-product:

if all the attributes of C are in R,

if all the attributes of C1 are in R and those of C2 are in S,

))(())((,...,2,1,...,2,1

RRAnAACCAnAA

SRSRCC

)()(

)()()(2121

SRSRCCCC


Algebraic Equivalences (3)

Selection & Join:

if all the attributes of C are in R,

if all the attributes of C are in

Projection & Join: if A includes join attributes,

SRc

SRc

)()(

)()()( SRSRCCC

)()()(21

SRSRAAA

SR


5/17


Example: Plan 1 (Nave)

Cost: BS + BS*BR =500+500*1000 (Page I/Os)

By no means the worst plan!

SELECT S.snameFROM Reserves R, Sailors SWHERE R.sid=S.sid AND

R.bid=100 AND S.rating>5

Reserves Sailors

sid=sid

bid=100 rating > 5

sname

(Simple Nested Loops)

(On-the-fly)

(On-the-fly)Plan 1:


Example: Plan 2 (Push Selection)

T1 = bid=100R; T2 = rating>5S;

To read R and write T1, BR+fRBR.

To read S and write T2, BS+fSBS.

If fRBR=10 and fSBS=250, the cost to join

T1 and T2 is fRBR+fRBR*fSBS.

Total cost is (1000+10) + (500+250) +(10+10*250) = 4270 (page I/Os).

By pushing projections, we can makeT1and T2 smaller: T1(sid), T2(sid, sname).

Reserves Sailors

sid=sid

bid=100

sname(On-the-fly)

rating > 5

(Simple NestedLoops)

)(21

TTsidsidsname =


6/17


Example: Plan 3

(Push Selection)

Use Sort-MergeJoin instead of Simple Nested-Loop Join.

With 5 buffers, cost of plan: To read R and write T1, BR+fRBR = 1000 + 10.

To read S and write T2, BS+fSBS = 500 + 250.

To sort T1 by 2-pass merge, 2*10 * 2 (read and write).

To sort T2 by 4-pass merge, 4*250 * 2 (read and write). To read T1 and T2, 10 + 250.

Total: 4060 page I/Os.

Reserves Sailors

sid=sid

bid=100

sname(On-the-fly)

rating > 5(Scan;write totemp T1)

(Scan;write totemp T2)

(Sort-Merge Join)


Example: Plan 4 (With Indexes)

Clustered hash index on bid of Reserves.

Clustered hash index on sid of Sailors.

Use Indexed Nested-LoopJoin with Pipelining.

1000 tuples (10 pages) are selected from R via index.

For each tuple from R, access S via hash index on sid; atmost one tuple is fetched from S (sid is key of S).

About 1.2 pages are accessed from I(S) (overflow).

Cost: 1.2 (I(R): Rs index page)+10 (pages of

R)+(1.2+1)*1000 (pages of I(S) and S) = 2211 (pages). If you push rating>5 down to Sailors, you cannot use

the index for S.Reserves

Sailors

sid=sid

bid=100

sname

(On-the-fly)rating > 5

(Index NL)

(On-the-fly)


7/17


Example: Plan 5 (With Indexes)

Clustered hash index on bid of Reserves.

Clustered hash index on sid of Sailors.

Use Indexed Nested-LoopJoin T1 = bid=100 R; T1 has 1000 tuples (10 pages)

but the number of unique sid values may bemuch smaller.

Sort T1 by sid values; then the number ofpages fetched from S is the number of uniquesid values in T1.

Cost: (Cost to compute and sort T1) + 10

(pages of T1) + (1.2+1)*(unique sid valuesin T1).

Reserves

Sailors

sid=sid

bid=100

sname (On-the-fly)

rating > 5

(Index NL)

(On-the-fly)

(Write to T1;Sort by sid)


Result Size Estimation

For each plan considered,

Must estimate the size of result from each operator in theplan tree. Use information about the input relations.

Usually assume uniform distribution of values, independence ofpredicates.

The intermediate result from an operator will become

input to the next operator in the plan tree.


8/17


Result Size Estimate: Cartesian Product

Size of R S, BRS = BRTS+TRBS . The length of an R S tuple is B/pR + B/pS.

The number of R S tuples per page is

pRS = B / (B/pR + B/pS) = (1/pR + 1/pS)-1.

Therefore,

BRS = TRS / pRS = TR TS (1/pR + 1/pS) = BRTS+TRBS.


Result Size Estimate: Join

Assumptions IR and IS are the domain sizes of join attributes of R and S.

Assume domain(Rj) domain(Sj) or vice versa.

Size of R S, BRS = (BRTS+TRBS)/max(IR,IS). The number of R S tuples per page is pRS = (1/pR + 1/pS)

-1.

The total number of tuples in R S is TRS = TRTS /max(IR,IS).

For each join attribute value x common in R and S, TR/IR tuples in R andTS/IS tuples in S are expected to have x as their join attribute value.

Since there are min(IR,IS) common values like x,

TRS = (TR/IR) (TS/IS) IR = TRTS /max(IR,IS).

Therefore, BRS = TRS / pRS = TRTS (1/pR + 1/pS)/max(IR,IS)

= (BRTS+TRBS)/max(IR,IS).


9/17


Query Optimization: Overview

Assumptions

Users need not know physical structures and access paths.

Users do not specify join orders.

What Optimizer does:

Look up catalog for statistics (e.g., data type & length ofcolumn).

Perform access path selection for single-relation queryblock.

Determine join order and join methods. Determine evaluation order among query blocks.

Change the structure of parse trees.


System R Optimizer

Cost estimation: Approximate art at best. Statistics, maintained in system catalogs, used to estimate cost of

operations and result sizes.

Considers combination of CPU and I/O costs.

Plan Space: Too large, must be pruned. Only the space of left-deep plans is considered.

Left-deep plans allow output of each operator to bepipelined into the nextoperator without storing it in a temporary relation.

Intermediate results are physically stored only if a sort is required for thenext join step.

Cartesian products avoided.

Impact: Most widely used currently; works well for < 10 joins.


10/17


Statistics and Catalogs

Need information about the relations and indexes involved.Catalogs typically contain at least: TR=# tuples and BR=# pages for each relation R.

KI=# distinct key values and BI=# pages for each index I.

dI=index depth, lI/hi=low/high key values for each tree index I.

Catalogs updated periodically. Updating whenever data changes is too expensive; lots of

approximation anyway, so slight inconsistency ok.

Contemporary DBMSs useHistograms to store more detailed statistics. Frequency distribution of attribute values in a relation

Equi-depth, equi-width, serial, biased, end-biased histograms, etc.


Single-Relation Plans: Selectivity

SARG-able (Search Argument) predicates: Can determine qualification of individual tuples. Relation.column comparison-op constant (e.g., salary > 50K)

Selectivity factor (SF) associated with each sargable predicate reflects itsimpact on reducing result size. col=value

SF = 1/KI, given index I on col, SF = 1/10 o/w.

col1=col2 SF = 1/max(KI1,KI2), given index I1 and I2, SF = 1/10 o/w.

col>value SF = (hR-value) / (hR- lR) if arithmetic type, SF = 1/3 o/w.

val1


11/17


Selectivity for Conjunctive Predicates

Assume that sargable predicates are independent.

For a query with a CNF of N sargable predicates,

SF1^2^^N= SF1 * SF2 * * SFN Result cardinality = TR * (SF1 * SF2 * * SFN)

(E.g.) If SF(age=30)=10% and SF(dept=CS)=5%, thenSF(age=30 and dept=CS) = 10% * 5% = .5%.


Single-Relation Plans: Cost Estimates

I/O cost for selection operations with index Primary key equality-match selection (unique)

dI+1 (for a B+ tree) or 1.2+1 (for hash index).

With clustered index matching one or more (BI+ BR) * selectivity-factors

With non-clustered index matching one or more

(BI+ TR) * selectivity-factors

System R considers both I/O and CPU costs Cost formula=I/O (index + data) + w*(per-tuple CPU).


12/17


Example

Result size: (1/KI) * TR = (1/10) * 40000 = 4000 tuples Plan 1: If we use an index on rating:

Clustered index: (1/ KI) * (BI + BR) = (1/10) * (50+500) = 55 pagesare retrieved (assuming SF = 1/10).

Unclustered index: (1/ KI) * (BI + TR) = (1/10) * (50+40000) = 4005pages are retrieved.

Plan 2: If we use an index on sid: Would have to retrieve all tuples/pages. With a clustered index,

the cost is BI + BR = 50+500, with unclustered index, BI + TR =50+40000.

Looks useless, but the resulting tuples may be in an interestingorder, which may be useful for the later stages of processing.

Plan 3: Do a file scan: Retrieve all file pages (500).

SELECT S.sidFROM Sailors SWHERE S.rating=8


Multiple-Relation Plans

Fundamental decision in System R: only left-deep join treesare considered. As the number of joins increases, the number of alternative plans

grows rapidly; we need to restrict the search space.

Left-deep trees allow us to generatefully pipelined plans.

Intermediate results not written to temporary files.

Not all left-deep trees are fully pipelined (e.g., SM join).

BA

C

D

BA

C

D

C DBA


13/17


Enumeration of Left-Deep Plans

Left-deep plans differ only in the order of relations, the access method foreach relation, and the join method for each join. (That is, N!join orders forN relations.)

Once the first k relations are joined, the method to join the result to the(k+1)th relation is independent of the order of joining the first k relations.

Dynamic Programming with N passes. Pass 1: Find best 1-relation plan for each relation.

Pass 2: Find best way to join result of each 1-relation plan (as outer) toanother relation. (All 2-relation plans.)

Pass N: Find best way to join result of a (N-1)-relation plan (as outer) to theNth relation. (All N-relation plans.)

For each subset of relations, retain only: Cheapest plan overall, plus

Cheapest plan for each interesting orderof the tuples.


Enumeration of Plans (Contd.)

Avoid Cartesian products if possible. For example, if there is no join predicate between R1 and R3, the

following orders are not considered: R1 - R3 - R2, R3 - R1 - R2.

ORDER BY, GROUP BY, aggregates etc. handled as a finalstep, using either an interestingly ordered plan or anadditional sort operator.

Interesting orders if sort-merge join is considered or

if a query contains ORDER-BY or GROUP-BY

In spite of pruning plan space, this approach is stillexponential in the # of relations.


14/17


Example

Pass1:

Sailors: B+ tree matches rating>5, and is probablycheapest. However, if this selection is expected toretrieve a lot of tuples, and index is unclustered, filescan may be cheaper. Still, B+ tree plan kept (because tuples are in rating

order).

Reserves: B+ tree matches bid=100; cheapest.

Sailors:B+ tree on ratingHash on sid

Reserves:B+ tree on bid

Pass 2:

We consider each plan retained from Pass 1 as the outer, and consider

how to join it with the (only) other relation. e.g., Reserves as outer: Hash index can be used to get Sailors tuples

that satisfy sid = outer tuples sid value.

Reserves Sailors

sid=sid

bid=100 rating > 5

sname


Example Database and Query


15/17


Pass 1: Best Single-Relation Plan


Pass 2: Best 2-Relation Plan (NL Join)


16/17


Pass 2: Best 2-Relation Plan (SM Join)


Pass 3: Best 3-Relation Join


17/17


Query Opt Issues Not Covered

Optimization of nested query blocks Nested blocks may or may be correlated.

Flattening a correlated nested query, etc.

Optimization of GROUP-BY, aggregation Early, late, stagewise groupings, etc.

Multiple Query Optimization

Histograms

Optimality of histograms, Multidimensional histograms, etc. And much more

Database design and Implementation 09.queryopt

Documents

Transcript of Database design and Implementation 09.queryopt