Query Processing and Optimization. Query Processing Efficient Query Processing crucial for good or...

Click here to load reader

  • date post

    22-Dec-2015
  • Category

    Documents

  • view

    217
  • download

    1

Embed Size (px)

Transcript of Query Processing and Optimization. Query Processing Efficient Query Processing crucial for good or...

  • Slide 1
  • Query Processing and Optimization
  • Slide 2
  • Query Processing Efficient Query Processing crucial for good or even effective operations of a database Query Processing depends on a variety of factors, not everything under the control of the DBMS Insufficient or incorrect information can result in vastly ineffective plans Query Cataloging
  • Slide 3
  • Steps in Query Execution SQL Query Scanning, Parsing, Validating Intermediate query representation (logical query plan tree)
  • Slide 4
  • Steps in Query Execution Intermediate Query Representation Query Optimizer Query Execution Plan (physical query plan tree)
  • Slide 5
  • Steps in Query Execution Query Execution Plan Query Code Generator Code to execute query
  • Slide 6
  • Steps in Query Execution Code to Execute Query Run-time Database Processor Query Results
  • Slide 7
  • Query Processing Intermediate Form Usually a relational algebra form of the SQL query Uses heuristics and cost-based measures for optimization Physical Query Plan In a language that is interpreted and executed on the machine or compiled into machine code.
  • Slide 8
  • Physical-Query-Plan Operators Basic set of operators that define the language of physical query execution plans Comprises the set of relational operators in addition to some more required operators
  • Slide 9
  • Physical-Query-Plan Operators Table scan operator: Scan and return an entire relation R Scan and return only those tuples of R that satisfy a given predicate Table scan: reading all blocks of a sorted data file in sequence Index scan: Use an index to read al blocks of a data file in sequence
  • Slide 10
  • Sorting while Scanning Sort-scan operator sorts a relation R while scanning it into memory If sorting is done on an indexed key attribute, no need to do anything; scanning will read R in sorted order If R is small enough to fit in main memory, perform sorting after scanning Else, use external sort-merge techniques to implement sort-scan
  • Slide 11
  • Iterators Iterators are physical-query-plan operators that comprise of three stages: Open() where an iteratable object (such as a relation is opened) GetNext() returns the next element of the object Close() closes control on the object.
  • Slide 12
  • Iterators Example of a Table scan iterator: Open() { b first block of R; t first tuple of b; } GetNext() { if (t is beyond the last tuple in b) { increment b; if (b is beyond last block) RETURN NoMoreData; else t first block of b; }
  • Slide 13
  • Iterators Example of a Table scan iterator (contd): oldt t; increment t; RETURN oldt; } Close() { }
  • Slide 14
  • Iterators Computing Bag union R+S using iterators over iterators: Open() { R.Open(); CurRel R; } GetNext() { If (CurRel = R) { t R.GetNext(); if (t != NoMoreData) return t;
  • Slide 15
  • Iterators Computing Bag union R+S using iterators over iterators: ELSE { /* R is exhausted */ S.Open(); CurRel S; } } ELSE { /* if (CurRel = R) */ RETURN S.GetNext; } /* If S is exhausted, it returns NoMoreData, which is what should be returned by this GetNext as well */ Close() { R.Close(); S.Close() }
  • Slide 16
  • Database Access Algorithms Algorithms for database access can be broadly divided in the following categories: Sorting-based methods Hash-based methods Index-based methods
  • Slide 17
  • Physical-Query-Plan Operator Types Tuple-at-a-time unary operators: Can read only one block at a time and required to work with only one tuple Full-relation unary operators: Requires knowledge of all or most of the relation. Read into memory for small relations in one-pass algorithms Full-relation binary operators: Same as above, but on two relations.
  • Slide 18
  • Tuple-at-a-time Unary Operations Examples: (R), (R) etc A strategy for a one-pass algorithm: R Input Buffer Unary Operator Output Buffer
  • Slide 19
  • Relation-at-a-time Unary Operations Example: UNIQUE, GROUP BY, etc A strategy for a one-pass algorithm R Input Buffer Unary Operator Output Buffer Data-structure Holding history
  • Slide 20
  • Relation-at-a-time Binary Operators One-pass strategies for binary relation-at-a-time operators vary between different operators. Almost all of them require at least one of the relation to be completely stored in memory.
  • Slide 21
  • Strategies Set Union R U S Assuming R is the bigger relation: Read S into memory completely and make it accessible through an in-memory index structure Output all tuples of S while reading For each tuple of R, search if it already exists in S, and output if not.
  • Slide 22
  • Strategies Set Intersection R S Assuming R is the bigger relation: Read S into memory completely and make it accessible through an in-memory index structure For each tuple of R, search if it already exists in S, and output if true.
  • Slide 23
  • Strategies Set Difference R - S Assuming R is the bigger relation: Read S into memory completely and make it accessible through an in-memory index structure For each tuple of R, search if it already exists in S. If tuple exists in S, then ignore; else output the tuple
  • Slide 24
  • Strategies Set Difference S - R Assuming R is the bigger relation: Read S into memory completely and make it accessible through an in-memory index structure For each tuple of R, search if it already exists in S, and delete it from S if it exists Output all remaining tuples of S.
  • Slide 25
  • Strategies Cross Product R x S Assuming R is the bigger relation: Read S into memory completely and store it in a buffer. No special data structure required. For each tuple of R, combine it with each tuple of S and output the result.
  • Slide 26
  • Strategies Natural Join R * S. Assume R(X,Y) and S(Y,Z) Assuming R is the bigger relation: Read S into memory completely and store it in an balanced tree index structure or a hash table. For each tuple of R, search S to see if a matching tuple exists. Output if matching tuple found.
  • Slide 27
  • One-pass algorithms One-pass algorithms are applicable only when one of the relation fits completely into memory In addition, there is enough memory to store at least one block of the other relation Hence, if M memory buffers are available, then one of the relations should have a maximum size of M-1.
  • Slide 28
  • One-pass Algorithms One pass algorithms rely on correctly estimating relation sizes and allocating memory buffers If too many buffers are allocated, there is a possibility of thrashing If too few buffers are allocated, then one-pass algorithms may not run
  • Slide 29
  • Summary Stages in Query Processing Logical Query Plan and Physical Query Plan Intermediate Query Language Physical Query Plan language constructs One-pass algorithms for unary and binary operators.
  • Slide 30
  • Query Processing and Optimization (contd.)
  • Slide 31
  • Query Processing Efficient Query Processing crucial for good or even effective operations of a database Query Processing depends on a variety of factors, not everything under the control of the DBMS Insufficient or incorrect information can result in vastly ineffective plans Query Cataloging
  • Slide 32
  • Steps in Query Execution SQL Query Scanning, Parsing, Validating Intermediate query representation (logical query plan tree)
  • Slide 33
  • Steps in Query Execution Intermediate Query Representation Query Optimizer Query Execution Plan (physical query plan tree)
  • Slide 34
  • Steps in Query Execution Query Execution Plan Query Code Generator Code to execute query
  • Slide 35
  • Steps in Query Execution Code to Execute Query Run-time Database Processor Query Results
  • Slide 36
  • Query Processing Intermediate Form Usually a relational algebra form of the SQL query Uses heuristics and cost-based measures for optimization Physical Query Plan In a language that is interpreted and executed on the machine or compiled into machine code.
  • Slide 37
  • Physical-Query-Plan Operators Basic set of operators that define the language of physical query execution plans Comprises the set of relational operators in addition to some more required operators Example operators: Table-scan, Index- scan, Sort-scan, Iterator, etc
  • Slide 38
  • One-pass Algorithms One-pass algorithms are applicable only when one of the relation fits completely into memory In addition, there is enough memory to store at least one block of the other relation Hence, if M memory buffers are available, then one of the relations should have a maximum size of M-1.
  • Slide 39
  • One-pass Algorithms One pass algorithms rely on correctly estimating relation sizes and allocating memory buffers If too many buffers are allocated, there is a possibility of thrashing If too few buffers are allocated, then one-pass algorithms may not run
  • Slide 40
  • Multi-pass Algorithms Used when entire relations cannot be read into memory Requires alternate computation and retrieval of intermediate results Many multi-pass algorithms are generalizations of their corresponding two pass algorithms
  • Slide 41
  • Basic Idea: Two-pass Algorithms Based on Sorting Suppose relation R is too big to fit in memory which can accommodate only M blocks of data. The sorting-based 2-pass algorithms have the following basic structure:
  • Slide 42
  • Basic Idea: Two-pass Algorithms Based on Sorting 1.Read M blocks of records into memory and sort them 2.Write them back to disk 3.Continue steps 1 and 2 until R is exhausted 4.Use a variety of query-merge techniques to extract relevant results from all the sorted M-blocks on disk.
  • Slide 43
  • Duplicate Elimination Using Sorting 1.Let relation R, in which duplicates have to be eliminated, be too big to fit in memory 2.Read M
  • Parse Trees Parse tree for: DeptName Salary>300000 (Manager Manager.Dno = Dept.DNO Dept) DeptName Salary>300000 Manager.DNO = Dept.DNO Manager Dept
  • Slide 69
  • Checks on Parse Trees Syntactic Checks: Is the syntax of every operator correct? Entity checks: Does every relation name refer to a valid relation? View Expansion: If a relation name refers to a view, replace the relation node with the parse tree of the view Attribute checks: Does every attribute name refer to valid attributes? Type checks: Does each attribute participating in an expression have the proper type?
  • Slide 70
  • Rewriting Parse Trees Queries are optimized by rewriting parse trees Rewriting parse trees is guided by a set of rewrite rules Parse tree should be expanded to its maximum extent before rewriting (Ex: views should be replaced by relevant parse trees) Some rewrite rules are situation specific: they work if certain conditions hold on the data set.
  • Slide 71
  • Pushing Selects Since a select statement reduces the size of a relation, they can be pushed as far down a parse tree as possible: DeptName Salary>300000 Manager.DNO = Dept.DNO Manager Dept The parse tree shown here can be rewritten as
  • Slide 72
  • Pushing Selects DeptName Salary>300000 Manager.DNO = Dept.DNO Manager Dept Instead of pairing all managers with their respective departments, choose only those who have Salary > 300000.
  • Slide 73
  • Pushing Selects Conjunctive Selects can be split and pushed to form cascading Selects that progressively reduce relation size: C AND D (R) C ( D (R))
  • Slide 74
  • Pushing Selects When a query contains a view, Selects may have to be first moved up before they are moved down: Consider relations Movie (title, year, director, language) StarsIn (title, year, StarName, language) and the view: CREATE VIEW BengaliMovies AS SELECT * FROM Movie WHERE language=Bengali;
  • Slide 75
  • Pushing Selects Consider the query: which star worked under which director in Bengali movies? SELECT starname, director FROM BengaliMovies NATURAL JOIN StarsIn; starname, director language=Bengali Movie StarsIn Parse Tree for BengaliMovies
  • Slide 76
  • Pushing Selects starname, director language=Bengali Movie StarsIn All tuples of StarsIn are selected, even if they are joined with tuples having language = Bengali starname, director language=Bengali Movie StarsIn language=Bengali Rewritten parse tree
  • Slide 77
  • Pushing Selects If select over a join involves attributes of only one of the relations, move select below the join. Consider the following query over the Movies database comprising of Movie, StarsIn and BengaliMovies relations. Which stars acted under the direction of Satyajit Ray in Bengali Movies. In SQL: SELECT starname FROM BengaliMovies NATURAL JOIN StarsIn WHERE director = Satyajit Ray
  • Slide 78
  • Pushing Selects Corresponding Parse Tree Optimized Tree starname, director language=Bengali Movie StarsIn director=Satyajit Ray Expansion of view starname, director language=Bengali Movie StarsIn director=Satyajit Ray
  • Slide 79
  • Inserting Projects Extra projects can be added near the leaves of the parse tree to reduce the size of tuples going up the tree: DeptName Salary>300000 Manager.DNO = Dept.DNO Manager Dept DeptName Salary>300000 Manager.DNO = Dept.DNO Manager Dept DNO,Salary DNO,DeptName
  • Slide 80
  • Cost-Based Optimization Factors affecting query cost: Access cost to secondary storage Storage cost of intermediate files Computation cost Memory usage cost Communication cost (between the DBMS server and its client)
  • Slide 81
  • Catalogs Catalogs in a Database store information for cost estimation Catalogs are meta-data that could be either: Table specific Field specific Index specific Database wide
  • Slide 82
  • Catalog Examples B(R) Number of blocks taken by relation R T(R) Number of tuples in relation R V(R,a) Number of distinct values relation R has for value a. V(R, [a 1,a 2,a n ]) is the number of distinct values relation R has for the combined set of attributes a 1,a 2,a n.
  • Slide 83
  • Cost Estimation Examples Estimating the cost of selection. Consider a select of the form: S = A=c (R), where c is a constant and A is an attribute of R. T(S) = estimate of the number of tuples in S = T(R) / V(R,A) Gives a good estimate if all values of A have uniform probabilities of occurrence (in the selection query).
  • Slide 84
  • Cost Estimation Examples Estimating the cost of selection. Consider an inequality condition in select: S = A c (R), where c is a constant and A is an attribute of R. T(S) = estimate of the number of tuples in S = T(R) (V(R,A)-1) / V(R,A) Gives a good estimate if all values of A have uniform probabilities of occurrence (in the selection query).
  • Slide 85
  • Cost Estimation Examples Consider a composite condition in select: S = C OR D (R), where C and D are conditions on attributes of R. Let T(R) = n. Let p be the set of conditions satisfying C, and q be the set of conditions satisfying D. The probability that a given tuple will match C OR D is given by: (1 - (1 p/n)(1 q/n). Hence T(S) estimate is given by: n (1 (1-p/n)(1-q/n)).
  • Slide 86
  • Estimating size of natural joins Consider a natural join R(X,Y) * S(Y,Z). For simplicity, assume Y is a single attribute, while X and Z could be sets. Case 1: If V(R,Y) = V(S,Y) the opposite argument holds and V(R,Y) appears in the denominator.
  • Slide 87
  • Estimating size of natural joins In a general sense, the maximum of V(R,Y) and V(S,Y) appear in the denominator. Estimate = T(R)T(S) / max(V(R,Y), V(S,Y)) When Y is a composite parameter, the max of each corresponding attribute in Y are compared and multiplied in the denominator.
  • Slide 88
  • Summary Index Based Algorithms Query Optimization by rewriting parse tree Pushing selects Cascading selects Pulling selects from views Extra projects Cost estimation of query components based on catalog information