Download - 1 Introduction to Query Processing and Query Optimization Techniques.

1 Introduction to Query Processing and Query Optimization Techniques

2 Introduction to Query Processing Query optimization: The process of choosing a suitable execution strategy for processing a query. Amongst all equivalent evaluation plans choose the one with lowest cost. Cost is estimated using statistical information from the database catalog e.g. number of tuples in each relation, size of tuples Two internal representations of a query: Query Tree Query Graph

3 Basic Steps in Query Processing

4 Select balance From account where balance < 2500 balance 2500 ( balance (account)) is equivalent to balance ( balance 2500 (account)) Annotated relational algebra expression specifying detailed evaluation strategy is called an evaluation-plan. E.g., can use an index on balance to find accounts with balance 2500, or can perform complete relation scan and discard accounts with balance 2500

5 Measures of Query Cost Cost is generally measured as total elapsed time for answering query. Many factors contribute to time cost disk accesses, CPU, or even network communication Typically disk access is the predominant cost, and is also relatively easy to estimate. For simplicity we just use number of block transfers from disk as the cost measure Costs depends on the size of the buffer in main memory Having more memory reduces need for disk access

6 Translating SQL Queries into Relational Algebra Query Block: The basic unit that can be translated into the algebraic operators and optimized. A query block contains a single SELECT-FROM- WHERE expression, as well as GROUP BY and HAVING clause if these are part of the block. Nested queries within a query are identified as separate query blocks Aggregate operators in SQL must be included in the extended algebra.

7 Translating SQL Queries into Relational Algebra SELECT LNAME, FNAME FROM EMPLOYEE WHERE SALARY > (SELECT MAX (SALARY) FROMEMPLOYEE WHERE DNO = 5); SELECTMAX (SALARY) FROMEMPLOYEE WHERE DNO = 5 SELECT LNAME, FNAME FROM EMPLOYEE WHERE SALARY > C LNAME, FNAME ( SALARY>C (EMPLOYEE)) MAX SALARY ( DNO=5 (EMPLOYEE))

8 Algorithms for External Sorting Sorting is needed in Order by, join, union, intersection, distinct, For relations that fit in memory, techniques like quicksort can be used. External sorting: Refers to sorting algorithms that are suitable for large files of records stored on disk that do not fit entirely in main memory, such as most database files. For relations that dont fit in memory, external sort-merge is a good choice

9 External Sort-Merge External sort-merge algorithm has two steps: Partial sort step, called runs. Merge step, merges the sorted runs. Sort-Merge strategy: Starts by sorting small subfiles (runs) of the main file and then merges the sorted runs, creating larger sorted subfiles that are merged in turn. Sorting phase: Sorts n B pages at a time n B = # of main memory pages buffer creates n R = b/n B initial sorted runs on disk b = # of file blocks (pages) to be sorted Sorting Cost = read b blocks + write b blocks = 2 b

10 External Sort-Merge Example: n B = 5 blocks and file size b = 1024 blocks, then n R = (b/n B ) = 1024/5 = 205 initial sorted runs each of size 5 bocks (except the last run which will have 4 blocks) n B = 2, b = 7, n R = b/n B = 4 run

11 External Sort-Merge Sort Phase: creates n R sorted runs. i = 0 Repeat Read next n B blocks into RAM Sort the in-memory blocks Write sorted data to run file R i i = i + 1 Until the end of the relation n R = i

12 External Sort-Merge Merging phase: The sorted runs are merged during one or more passes. The degree of merging (d M ) is the number of runs that can be merged in each pass. d M = Min (n B -1, n R ) n P = (log dM (n R )) n P : number of passes. In each pass, One buffer block is needed to hold one block from each of the runs being merged, and One block is needed for containing one block of the merged result.

13 External Sort-Merge We assume (for now) that n R n B. Merge the runs (n R -way merge). Use n R blocks of memory to buffer input runs, and 1 block to buffer output. Merge n R Runs Step Read 1 st block of each n R runs R i into its buffer page Repeat Select 1 st record (sort order) among n R buffer pages Write the record to the output buffer. If the output buffer is full write it to disk Delete the record from its input buffer page If the buffer page becomes empty then read the next block (if any) of the run into the buffer Until all input buffer pages are empty

14 External Sort-Merge Merge n B - 1 Runs Step If n R n B, several merge passes are required. merge a group of contiguous n B - 1 runs using one buffer for output A pass reduces the number of runs by a factor of n B - 1, and creates runs longer by the same factor. E.g. If n B = 11, and there are 90 runs, one pass reduces the number of runs to 9, each 10 times the size of the initial runs Repeated passes are performed till all runs have been merged into one

15 External Sort-Merge Degree of merging (d M ) # of runs that can be merged together in each pass = min (n B - 1, n R ) Number of passes n P = (log dM (n R )) In our example d M = 4 (four-way merging) min (n B -1, n R ) = min(5-1, 205) = 4 Number of passes n P = (log dM (n R )) = (log 4 (205)) = 4 First pass: 205 initial sorted runs would be merged into 52 sorted runs Second pass: 52 sorted runs would be merged into 13 Third pass: 13 sorted runs would be merged into 4 Fourth pass: 4 sorted runs would be merged into 1

16 External Sort-Merge Blocking factor bfr = 1 record, n B = 3, b = 12, n R = 4, d M = min(3-1, 4) = 2

17 External Sort-Merge

18 External Sort-Merge External Sort-Merge: Cost Analysis Disk accesses for initial run creation (sort phase) as well as in each merge pass is 2b reads every block once and writes it out once Initial # of runs is n R = b/n B and # of runs decreases by a factor of n B - 1 in each merge pass, then the total # of merge passes is n p = log d M (n R ) In general, the cost performance of Merge-Sort is Cost = sort cost + merge cost Cost = 2b + 2b * n p Cost = 2b + 2b * log dM n R = 2b ( log d M (n R ) + 1)

19 External Sort-Merge

20 Catalog Information File r:# of records in the file R:record size b:# of blocks in the file bfr:blocking factor Index x:# of levels of a multilevel index b I1 :# of first-level index blocks

21 Catalog Information Attribute d:# of distinct values of an attribute sl (selectivity): the ratio of the # of records satisfying the condition to the total # of records in the file. s (selection cardinality) = sl * r average # of records that will satisfy an equality condition on the attribute For a key attribute: d = r,sl = 1/r,s = 1 For a nonkey attribute: assuming that d distinct values are uniformly distributed among the records the estimated sl = 1/d,s = r/d

22 File Scans Types of scans File scan search algorithms that locate and retrieve records that fulfill a selection condition. Index scan search algorithms that use an index selection condition must be on search-key of index. Cost estimate C = # of disk blocks scanned

23 Algorithms for SELECT Operations Implementing the SELECT Operation Examples: (OP1): SSN='123456789' (EMP) (OP2): DNUMBER>5 (DEPT) (OP3): DNO=5 (EMP) (OP4): DNO=5 AND SALARY>30000 AND SEX=F (EMP) (OP5): ESSN=123456789 AND PNO=10 (WORKS_ON)

24 Algorithms for Selection Operation Search Methods for Simple Selection: S1 (linear search) Retrieve every record in the file, and test whether its attribute values satisfy the selection condition. If selection is on a nonkey attribute, C = b If selection is equality on a key attribute, if record found, average cost C = b /2, else C = b

25 Algorithms for Selection Operation S2 (binary search) Applicable if selection is an equality comparison on the attribute on which file is ordered. Assume that the blocks of a relation are stored contiguously If selection is on a nonkey attribute: C = log 2 b: cost of locating the 1 st tuple + s/bfr - 1: # of blocks containing records that satisfy selection condition If selection is equality on a key attribute: C = log 2 b,since s = 1, in this case

26 Selections Using Indices S3 (primary or hash index on a key, equality) Retrieve a single record that satisfies the corresponding equality condition If the selection condition involves an equality comparison on a key attribute with a primary index (or a hash key), use the primary index (or the hash key) to retrieve the record. Primary index: retrieve 1 more block than the # of index levels, C = x + 1: Hash index: C = 1:for static or linear hashing C = 2:for extendable hashing

27 Selections Using Indices S4 (primary index on a key, range selection) S4 Using a primary index to retrieve multiple records: If the comparison condition is >, ,

51 Using Heuristics in Query Optimization Heuristic Optimization of Query Trees: The same query could correspond to many different relational algebra expressions and hence many different query trees. The task of heuristic optimization of query trees is to find a final query tree that is efficient to execute. Example: SELECT LNAME FROM EMPLOYEE, WORKS_ON, PROJECT WHERE PNAME = AQUARIUS AND PNMUBER=PNO AND ESSN=SSN AND BDATE > 1957-12-31;

52 Using Heuristics in Query Optimization

55 Transformation Rules for RA Operations 1.Cascade of : A conjunctive selection condition can be broken up into a cascade (sequence) of individual operations: c1 AND c2 AND... AND cn (R) c1 ( c2 (...( cn (R))...)) 2.Commutativity of : The operation is commutative: c1 ( c2 (R)) c2 ( c1 (R)) 3.Cascade of : In a cascade (sequence) of operations, all but the last one can be ignored: List1 ( List2 (...( Listn (R))...)) = List1 (R) 4.Commuting with : If the selection condition c involves only the attributes A1,..., An in the projection list, the two operations can be commuted: A1, A2,..., An ( c (R)) = c ( A1, A2,..., An (R))

56 Transformation Rules for RA Operations 5.Commutativity of (or ): The operation is commutative as is the operation: R c S = S c R; R S = S R 6.Commuting with (or ): If all the attributes in the selection condition c involve only the attributes of one of the relations being joined say, R the two operations can be commuted as follows: c (R S) ( c (R)) S Alternatively, if the selection condition c can be written as (c1 and c2), where condition c1 involves only the attributes of R and condition c2 involves only the attributes of S, the operations commute as follows: c (R S) ( c1 (R)) ( c2 (S))

57 Transformation Rules for RA Operations 7.Commuting with (or ): Suppose that the projection list is L = {A1,..., An, B1,..., Bm}, where A1,..., An are attributes of R and B1,..., Bm are attributes of S. If the join condition c involves only attributes in L, the two operations can be commuted as follows: L (R c S) ( A1,..., An (R)) c ( B1,..., Bm (S)) If the join condition c contains additional attributes not in L, these must be added to the projection list, and a final operation is needed.

58 Transformation Rules for RA Operations 8.Commutativity of set operations: The set operations and are commutative but is not. 9.Associativity of , , , and : These four operations are individually associative; that is, if stands for any one of these 4 operations (throughout the expression), we have (R S) T R (S T) 10.Commuting with set operations: The operation commutes with , , and . If stands for any one of these 3 operations, we have c (R S) ( c (R)) ( c (S))

59 Transformation Rules for RA Operations 11.The operation commutes with . L (R S) ( L (R)) ( L (S)) 12.Converting a ( sequence into : If the condition c of a that follows a corresponds to a join condition, convert the ( sequence into a as ( c (R S)) = (R c S)

60 Heuristic Algebraic Optimization Algorithm Using rule 1, break up any select operations with conjunctive conditions into a cascade of select operations. c1 AND c2 AND AND cn (R) c1 ( c2 ( ( cn (R)) )) Using rules 2, 4, 6, and 10 concerning the commutativity of select with other operations, move each select operation as far down the query tree as is permitted by the attributes involved in the select condition. Using rule 9 concerning associativity of binary operations, rearrange the leaf nodes of the tree: Position the leaf node relation with the most restrictive operations so they are executed first, Make sure that the ordering of leaf nodes does not cause CARTESIAN PRODUCT operations

61 Heuristic Algebraic Optimization Algorithm 4. Using Rule 12, combine a cartesian product operation with a subsequent select operation in the tree into a join operation. Combine a with a subsequent in the tree into a 5.Using rules 3, 4, 7, and 11 concerning the cascading of project and the commuting of project with other operations, break down and move lists of projection attributes down the tree as far as possible by creating new project operations as needed. 6.Identify subtrees that represent groups of operations that can be executed by a single algorithm.

62 Summary of Heuristics for Algebraic Optimization The main heuristic is to apply first the operations that reduce the size of intermediate results. Perform select operations as early as possible to reduce the number of tuples and perform project operations as early as possible to reduce the number of attributes. (This is done by moving select and project operations as far down the tree as possible.) The select and join operations that are most restrictive should be executed before other similar operations. (This is done by reordering the leaf nodes of the tree among themselves and adjusting the rest of the tree appropriately.)

63 Query Execution Plans An execution plan for a relational algebra query consists of a combination of the relational algebra query tree and information about the access methods to be used for each relation as well as the methods to be used in computing the relational operators stored in the tree. Materialized evaluation: the result of an operation is stored as a temporary relation. Pipelined evaluation: as the result of an operator is produced, it is forwarded to the next operator in sequence.

64 Using Selectivity and Cost Estimates in Query Optimization Cost-based query optimization: Estimate and compare the costs of executing a query using different execution strategies and choose the strategy with the lowest cost estimate. (Compare to heuristic query optimization) Issues Cost function Number of execution strategies to be considered

65 Using Selectivity and Cost Estimates in Query Optimization Cost Components for Query Execution 1.Access cost to secondary storage 2.Storage cost 3.Computation cost 4.Memory usage cost 5.Communication cost Note: Different database systems may focus on different cost components.

66 Using Selectivity and Cost Estimates in Query Optimization Catalog Information Used in Cost Functions Information about the size of a file number of records (tuples) (r), record size (R), number of blocks (b) blocking factor (bfr) Information about indexes and indexing attributes of a file Number of levels (x) of each multilevel index Number of first-level index blocks (bI1) Number of distinct values (d) of an attribute Selectivity (sl) of an attribute Selection cardinality (s) of an attribute. (s = sl * r)

67 Using Selectivity and Cost Estimates in Query Optimization Examples of Cost Functions for SELECT S1. Linear search (brute force) approach C S1a = b; For an equality condition on a key, C S1a = (b/2) if the record is found; otherwise C S1a = b. S2. Binary search: C S2 = log 2 b + (s/bfr) 1 For an equality condition on a unique (key) attribute, C S2 =log 2 b S3. Using a primary index (S3a) or hash key (S3b) to retrieve a single record C S3a = x + 1; C S3b = 1 for static or linear hashing; C S3b = 1 for extendible hashing;

68 Using Selectivity and Cost Estimates in Query Optimization Examples of Cost Functions for SELECT (contd.) S4. Using an ordering index to retrieve multiple records: For the comparison condition on a key field with an ordering index, C S4 = x + (b/2) S5. Using a clustering index to retrieve multiple records: C S5 = x + (s/bfr) S6. Using a secondary (B+-tree) index: For an equality comparison, C S6a = x + s; For an comparison condition such as >, =, or

70 10. Semantic Query Optimization Semantic Query Optimization: Uses constraints specified on the database schema in order to modify one query into another query that is more efficient to execute. Consider the following SQL query, SELECTE.LNAME, M.LNAME FROMEMPLOYEE E M WHEREE.SUPERSSN=M.SSN AND E.SALARY>M.SALARY Explanation: Suppose that we had a constraint on the database schema that stated that no employee can earn more than his or her direct supervisor. If the semantic query optimizer checks for the existence of this constraint, it need not execute the query at all because it knows that the result of the query will be empty. Techniques known as theorem proving can be used for this purpose.

71 Example (1) An un-optimized relational algebra expression: Name ( GPA 3.5 and Title = 'Ada Programming Language and Students.SSN = Enrollment.SSN and Enrollment.Course_no = Courses.Course_no (Students Enrollment Courses))

72 Example (2) Initial query tree: Name GPA 3.5 and Title = 'Ada Programming Language and Students.SSN = Enrollment.SSN and Enrollment.Course_no = Courses.Course_no Courses Enrollment Students

73 Example (3) Perform selections as early as possible. Name Title = 'Ada Programming Language Courses Enrollment Students Enrollment.Course_no = Courses.Course_no Students.SSN = Enrollment.SSN GPA 3.5

74 Example (4) Replace Cartesian products by joins. Name Title = 'Ada Programming Language Courses Enrollment Students GPA 3.5

75 Example (5) Perform more restrictive joins first. Name Title = 'Ada Programming Language Courses EnrollmentStudents GPA 3.5

76 Example (6) Project out useless attributes early. Name Title = 'Ada Programming Language Courses Enrollment Students GPA 3.5 SSN, Name SSN, Course_no Course_no SSN

77 Example (7) The optimized algebra expression is: Name (( SSN, Name ( GPA 3.5 (Students))) (( SSN, Course_no (Enrollments)) ( Course_no ( Title = 'Ada Programming Language (Courses))))) Projections and selections on the same relation are usually performed using the same scan of the relation.