# 1 Algorithms for Query Processing and Optimization.

date post

21-Dec-2015Category

## Documents

view

232download

2

Embed Size (px)

### Transcript of 1 Algorithms for Query Processing and Optimization.

- Slide 1
- 1 Algorithms for Query Processing and Optimization
- Slide 2
- 2 0. Basic Steps in Query Processing 1.Parsing and translation: Translate the query into its internal form. This is then translated into relational algebra. Parser checks syntax, verifies relations 2.Query optimization: The process of choosing a suitable execution strategy for processing a query. 3.Evaluation: The query evaluation engine takes a query execution plan, executes that plan, and returns the answers to the query.
- Slide 3
- 3 Basic Steps in Query Processing
- Slide 4
- 4
- Slide 5
- 5 Two internal representations of a query: Query Tree Query Graph Select balance From account where balance < 2500 balance 2500 ( balance (account)) is equivalent to balance ( balance 2500 (account))
- Slide 6
- 6 Basic Steps in Query Processing Annotated relational algebra expression specifying detailed evaluation strategy is called an evaluation-plan. E.g., can use an index on balance to find accounts with balance 2500, or can perform complete relation scan and discard accounts with balance 2500 Query Optimization: Amongst all equivalent evaluation plans choose the one with lowest cost. Cost is estimated using statistical information from the database catalog e.g. number of tuples in each relation, size of tuples
- Slide 7
- 7 Measures of Query Cost Cost is generally measured as total elapsed time for answering query. Many factors contribute to time cost disk accesses, CPU, or even network communication Typically disk access is the predominant cost, and is also relatively easy to estimate. For simplicity we just use number of block transfers from disk as the cost measure Costs depends on the size of the buffer in main memory Having more memory reduces need for disk access
- Slide 8
- 8 1. Translating SQL Queries into Relational Algebra (1) Query Block: The basic unit that can be translated into the algebraic operators and optimized. A query block contains a single SELECT-FROM- WHERE expression, as well as GROUP BY and HAVING clause if these are part of the block. Nested queries within a query are identified as separate query blocks Aggregate operators in SQL must be included in the extended algebra.
- Slide 9
- 9 Translating SQL Queries into Relational Algebra (2) SELECT LNAME, FNAME FROM EMPLOYEE WHERE SALARY > (SELECT MAX (SALARY) FROMEMPLOYEE WHERE DNO = 5); SELECTMAX (SALARY) FROMEMPLOYEE WHERE DNO = 5 SELECT LNAME, FNAME FROM EMPLOYEE WHERE SALARY > C LNAME, FNAME ( SALARY>C (EMPLOYEE)) MAX SALARY ( DNO=5 (EMPLOYEE))
- Slide 10
- 10 2. Algorithms for External Sorting (1) Sorting is needed in Order by, join, union, intersection, distinct, For relations that fit in memory, techniques like quicksort can be used. External sorting: Refers to sorting algorithms that are suitable for large files of records stored on disk that do not fit entirely in main memory, such as most database files. For relations that dont fit in memory, external sort-merge is a good choice
- Slide 11
- 11 External Sort-Merge External sort-merge algorithm has two steps: Partial sort step, called runs. Merge step, merges the sorted runs. Sort-Merge strategy: Starts by sorting small subfiles (runs) of the main file and then merges the sorted runs, creating larger sorted subfiles that are merged in turn. Sorting phase: Sorts n B pages at a time n B = # of main memory pages buffer creates n R = b/n B initial sorted runs on disk b = # of file blocks (pages) to be sorted Sorting Cost = read b blocks + write b blocks = 2 b
- Slide 12
- 12 External Sort-Merge Example: n B = 5 blocks and file size b = 1024 blocks, then n R = (b/n B ) = 1024/5 = 205 initial sorted runs each of size 5 bocks (except the last run which will have 4 blocks) n B = 2, b = 7, n R = b/n B = 4 run
- Slide 13
- 13 External Sort-Merge Sort Phase: creates n R sorted runs. i = 0 Repeat Read next n B blocks into RAM Sort the in-memory blocks Write sorted data to run file R i i = i + 1 Until the end of the relation n R = i
- Slide 14
- 14 External Sort-Merge Merging phase: The sorted runs are merged during one or more passes. The degree of merging (d M ) is the number of runs that can be merged in each pass. In each pass, One buffer block is needed to hold one block from each of the runs being merged, and One block is needed for containing one block of the merged result.
- Slide 15
- 15 External Sort-Merge We assume (for now) that n R n B. Merge the runs (n R -way merge). Use n R blocks of memory to buffer input runs, and 1 block to buffer output. Merge n R Runs Step Read 1 st block of each n R runs R i into its buffer page Repeat Select 1 st record (sort order) among n R buffer pages Write the record to the output buffer. If the output buffer is full write it to disk Delete the record from its input buffer page If the buffer page becomes empty then read the next block (if any) of the run into the buffer Until all input buffer pages are empty
- Slide 16
- 16 External Sort-Merge Merge n B - 1 Runs Step If n R n B, several merge passes are required. merge a group of contiguous n B - 1 runs using one buffer for output A pass reduces the number of runs by a factor of n B - 1, and creates runs longer by the same factor. E.g. If n B = 11, and there are 90 runs, one pass reduces the number of runs to 9, each 10 times the size of the initial runs Repeated passes are performed till all runs have been merged into one
- Slide 17
- 17 External Sort-Merge Degree of merging (d M ) # of runs that can be merged together in each pass = min (n B - 1, n R ) Number of passes n P = (log dM (n R )) In our example d M = 4 (four-way merging) min (n B -1, n R ) = min(5-1, 205) = 4 Number of passes n P = (log dM (n R )) = (log 4 (205)) = 4 First pass: 205 initial sorted runs would be merged into 52 sorted runs Second pass: 52 sorted runs would be merged into 13 Third pass: 13 sorted runs would be merged into 4 Fourth pass: 4 sorted runs would be merged into 1
- Slide 18
- 18 External Sort-Merge Blocking factor bfr = 1 record, n B = 3, b = 12, n R = 4, d M = min(3-1, 4) = 2
- Slide 19
- 19 External Sort-Merge
- Slide 20
- 20 External Sort-Merge External Sort-Merge: Cost Analysis Disk accesses for initial run creation (sort phase) as well as in each merge pass is 2b reads every block once and writes it out once Initial # of runs is n R = b/n B and # of runs decreases by a factor of n B - 1 in each merge pass, then the total # of merge passes is n p = log d M (n R ) In general, the cost performance of Merge-Sort is Cost = sort cost + merge cost Cost = 2b + 2b * n p Cost = 2b + 2b * log dM n R = 2b ( log d M (n R ) + 1)
- Slide 21
- 21 External Sort-Merge
- Slide 22
- 22 Catalog Information File r:# of records in the file R:record size b:# of blocks in the file bfr:blocking factor Index x:# of levels of a multilevel index b I1 :# of first-level index blocks
- Slide 23
- 23 Catalog Information Attribute d:# of distinct values of an attribute sl (selectivity): the ratio of the # of records satisfying the condition to the total # of records in the file. s (selection cardinality) = sl * r average # of records that will satisfy an equality condition on the attribute For a key attribute: d = r,sl = 1/r,s = 1 For a nonkey attribute: assuming that d distinct values are uniformly distributed among the records the estimated sl = 1/d,s = r/d
- Slide 24
- 24 File Scans Types of scans File scan search algorithms that locate and retrieve records that fulfill a selection condition. Index scan search algorithms that use an index selection condition must be on search-key of index. Cost estimate C = # of disk blocks scanned
- Slide 25
- 25 3. Algorithms for SELECT Operations Implementing the SELECT Operation Examples: (OP1): SSN='123456789' (EMP) (OP2): DNUMBER>5 (DEPT) (OP3): DNO=5 (EMP) (OP4): DNO=5 AND SALARY>30000 AND SEX=F (EMP) (OP5): ESSN=123456789 AND PNO=10 (WORKS_ON)
- Slide 26
- 26 Algorithms for Selection Operation Search Methods for Simple Selection: S1 (linear search) Retrieve every record in the file, and test whether its attribute values satisfy the selection condition. If selection is on a nonkey attribute, C = b If selection is equality on a key attribute, if record found, average cost C = b /2, else C = b
- Slide 27
- 27 Algorithms for Selection Operation S2 (binary search) Applicable if selection is an equality comparison on the attribute on which file is ordered. Assume that the blocks of a relation are stored contiguously If selection is on a nonkey attribute: C = log 2 b: cost of locating the 1 st tuple + s/bfr - 1: # of blocks containing records that satisfy selection condition If selection is equality on a key attribute: C = log 2 b,since s = 1, in this case
- Slide 28
- 28 Selections Using Indices S3 (primary or hash index on a key, equality) Retrieve a single record that satisfies the corresponding equality condition If the selection condition involves an equality comparison on a key attribute with a primary index (or a hash key), use the primary index (or the hash key) to retrieve the record. Primary index: retrieve 1 more block than the # of index levels, C = x + 1: Hash index: C = 1:for static or linear hashing C = 2:for extendable hashing
- Slide 29
- 29 Selections Using Indices S4 (primary index on a key, range selection) S4 Using a primary index to retrieve multiple records: If the comparison condition is >, , 1957-12-31;">
- 88 Using Heuristics in Query Optimization (6) Heuristic Optimization of Query Trees: The same query could correspond to many different relational algebra expressions and hence many different query trees. The task of heuristic optimization of query trees is to find a final query tree that is efficient to execute. Example: Q: SELECT LNAME FROM EMPLOYEE, WORKS_ON, PROJECT WHERE PNAME = AQUARIUS AND PNMUBER=PNO AND ESSN=SSN AND BDATE > 1957-12-31;
- Slide 89
- 89 Using Heuristics in Query Optimization (7)
- Slide 90
- 90 Using Heuristics in Query Optimization (8)
- Slide 91
- 91 Transformation Rules for RA Operations 1.Cascade of : A conjunctive selection condition can be broken up into a cascade (sequence) of individual operations: c1 AND c2 AND... AND cn (R) c1 ( c2 (...( cn (R))...)) 2.Commutativity of : The operation is commutative: c1 ( c2 (R)) c2 ( c1 (R)) 3.Cascade of : In a cascade (sequence) of operations, all but the last one can be ignored: List1 ( List2 (...( Listn (R))...)) = List1 (R) 4.Commuting with : If the selection condition c involves only the attributes A1,..., An in the projection list, the two operations can be commuted: A1, A2,..., An ( c (R)) = c ( A1, A2,..., An (R))
- Slide 92
- 92 Transformation Rules for RA Operations 5.Commutativity of (or ): The operation is commutative as is the operation: R c S = S c R; R S = S R 6.Commuting with (or ): If all the attributes in the selection condition c involve only the attributes of one of the relations being joined say, R the two operations can be commuted as follows: c (R S) ( c (R)) S Alternatively, if the selection condition c can be written as (c1 and c2), where condition c1 involves only the attributes of R and condition c2 involves only the attributes of S, the operations commute as follows: c (R S) ( c1 (R)) ( c2 (S))
- Slide 93
- 93 Transformation Rules for RA Operations 7.Commuting with (or ): Suppose that the projection list is L = {A1,..., An, B1,..., Bm}, where A1,..., An are attributes of R and B1,..., Bm are attributes of S. If the join condition c involves only attributes in L, the two operations can be commuted as follows: L (R c S) ( A1,..., An (R)) c ( B1,..., Bm (S)) If the join condition c contains additional attributes not in L, these must be added to the projection list, and a final operation is needed.
- Slide 94
- 94 Transformation Rules for RA Operations 8.Commutativity of set operations: The set operations and are commutative but is not. 9.Associativity of , , , and : These four operations are individually associative; that is, if stands for any one of these 4 operations (throughout the expression), we have (R S) T R (S T) 10.Commuting with set operations: The operation commutes with , , and . If stands for any one of these 3 operations, we have c (R S) ( c (R)) ( c (S))
- Slide 95
- 95 Transformation Rules for RA Operations 11.The operation commutes with . L (R S) ( L (R)) ( L (S)) 12.Converting a ( sequence into : If the condition c of a that follows a corresponds to a join condition, convert the ( sequence into a as ( c (R S)) = (R c S) 13.Other transformations (See page 564)
- Slide 96
- 96 Heuristic Algebraic Optimization Algorithm Using rule 1, break up any select operations with conjunctive conditions into a cascade of select operations. c1 AND c2 AND AND cn (R) c1 ( c2 ( ( cn (R)) )) Using rules 2, 4, 6, and 10 concerning the commutativity of select with other operations, move each select operation as far down the query tree as is permitted by the attributes involved in the select condition. Using rule 9 concerning associativity of binary operations, rearrange the leaf nodes of the tree: Position the leaf node relation with the most restrictive operations so they are executed first, Make sure that the ordering of leaf nodes does not cause CARTESIAN PRODUCT operations
- Slide 97
- 97 Heuristic Algebraic Optimization Algorithm 4. Using Rule 12, combine a cartesian product operation with a subsequent select operation in the tree into a join operation. Combine a with a subsequent in the tree into a 5.Using rules 3, 4, 7, and 11 concerning the cascading of project and the commuting of project with other operations, break down and move lists of projection attributes down the tree as far as possible by creating new project operations as needed. 6.Identify subtrees that represent groups of operations that can be executed by a single algorithm.
- Slide 98
- 98 Summary of Heuristics for Algebraic Optimization The main heuristic is to apply first the operations that reduce the size of intermediate results. Perform select operations as early as possible to reduce the number of tuples and perform project operations as early as possible to reduce the number of attributes. (This is done by moving select and project operations as far down the tree as possible.) The select and join operations that are most restrictive should be executed before other similar operations. (This is done by reordering the leaf nodes of the tree among themselves and adjusting the rest of the tree appropriately.)
- Slide 99
- 99 Query Execution Plans An execution plan for a relational algebra query consists of a combination of the relational algebra query tree and information about the access methods to be used for each relation as well as the methods to be used in computing the relational operators stored in the tree. Materialized evaluation: the result of an operation is stored as a temporary relation. Pipelined evaluation: as the result of an operator is produced, it is forwarded to the next operator in sequence.
- Slide 100
- 100 8. Using Selectivity and Cost Estimates in Query Optimization (1) Cost-based query optimization: Estimate and compare the costs of executing a query using different execution strategies and choose the strategy with the lowest cost estimate. (Compare to heuristic query optimization) Issues Cost function Number of execution strategies to be considered
- Slide 101
- 101 Using Selectivity and Cost Estimates in Query Optimization (2) Cost Components for Query Execution 1.Access cost to secondary storage 2.Storage cost 3.Computation cost 4.Memory usage cost 5.Communication cost Note: Different database systems may focus on different cost components.
- Slide 102
- 102 Using Selectivity and Cost Estimates in Query Optimization (3) Catalog Information Used in Cost Functions Information about the size of a file number of records (tuples) (r), record size (R), number of blocks (b) blocking factor (bfr) Information about indexes and indexing attributes of a file Number of levels (x) of each multilevel index Number of first-level index blocks (bI1) Number of distinct values (d) of an attribute Selectivity (sl) of an attribute Selection cardinality (s) of an attribute. (s = sl * r)
- Slide 103
- 103 Using Selectivity and Cost Estimates in Query Optimization (4) Examples of Cost Functions for SELECT S1. Linear search (brute force) approach C S1a = b; For an equality condition on a key, C S1a = (b/2) if the record is found; otherwise C S1a = b. S2. Binary search: C S2 = log 2 b + (s/bfr) 1 For an equality condition on a unique (key) attribute, C S2 =log 2 b S3. Using a primary index (S3a) or hash key (S3b) to retrieve a single record C S3a = x + 1; C S3b = 1 for static or linear hashing; C S3b = 1 for extendible hashing;
- Slide 104
- 104 Using Selectivity and Cost Estimates in Query Optimization (5) Examples of Cost Functions for SELECT (contd.) S4. Using an ordering index to retrieve multiple records: For the comparison condition on a key field with an ordering index, C S4 = x + (b/2) S5. Using a clustering index to retrieve multiple records: C S5 = x + (s/bfr) S6. Using a secondary (B+-tree) index: For an equality comparison, C S6a = x + s; For an comparison condition such as >, =, or