Query Optimization

overview

Application Programmer (e.g., business analyst,Data architect)Sophisticated Application Programmer (e.g., SAP admin)DBA, TunerHardware [Processor(s), Disk(s), Memory]Operating SystemConcurrency ControlRecoveryStorage SubsystemIndexesQuery ProcessorApplication

Overview of query processingParserQuery OptimizerStatisticsCost ModelQEPParsed QueryDatabaseHigh Level QueryQuery Result QueryEvaluatorPlan GeneratorPlan cost EstimatorCatalog Manager

Statistics and CatalogsNeed information about the relations and indexes involved. Catalogs typically contain at least:# tuples (NTuples) and # pages (NPages) for each relation.# distinct key values (NKeys) and NPages for each index.Index height, low/high key values (Low/High) for each tree index.Catalogs updated periodically.Updating whenever data changes is too expensive; lots of approximation anyway, so slight inconsistency ok.More detailed information (e.g., histograms of the values in some field) are sometimes stored.

An example of catalog tableSchema of a catalog table is also stored in the catalog table

Overview of Query OptimizationOutput of query optimization is QEP (Query evaluation plan)QEPTree of Relational Algbra oprations, with access methods for each table and choice of algorithms for each operation.Each operator typically implemented using a `pull interface: when an operator is `pulled for the next output tuples, it `pulls on its inputs and computes them.Two main issues in query optimization:For a given query, what plans are considered?Algorithm to search plan space for cheapest (estimated) plan. How is the cost of a plan estimated?Ideally: Want to find best plan. Practically: Avoid worst plans!We will study the System R approach.

Example of QEPThe query can be expressed in relational algebra as follows:sname(bid=100^rating>5(Reservessid=sidSailors)SELECT S.snameFROM Reserves R, Sailors SWHERE R.sid = S.sidAND R.bid = 100 AND S.rating > 5

Pipelined EvaluationPipelined vs MaterializedWhen a query is composed of several operators, the result of one operator is sometimes pipelined to another operator without creating a temporary relation to hold the intermediate result.Intermediate results are produced, consumed, and discarded one page at a timeOtherwise, the tuples are materializedPipeline is preferred, since read and write overhead is omitted

Left-deep planLinear tree: at least one child is a base tableLeft-deep tree: the right child of each join node is a base tableBusy treeLeft-deep plan can be fully-pipelined : all joins are evaluated using pipelining (inner table always be materialized because entire inner table needs to be examined)Some join, e.g. hash join, sort-merge join can not be fully pipelined

Busy treeLeft-deep treeBusy treeLinear tree

Highlights of System R OptimizerImpact:Most widely used currently; works well for < 10 joins.Cost estimation: Approximate art at best.Statistics, maintained in system catalogs, used to estimate cost of operations and result sizes.Considers combination of CPU and I/O costs.Plan Space: Too large, must be pruned.Only the space of left-deep plans is considered.Left-deep plans allow output of each operator to be pipelined into the next operator without storing it in a temporary relation. Cartesian products avoided.

Decomposing a query into blocksA query block (or simply block) is a SQL query with no nesting and exactly one SELECT clause and one FROM clause and at most one WHERE clause, GROUP BY clause, and HAVING clauseSELECT S.sid, MIN (R.day)FROM Sailors S, Reserves R, Boats BWHERE S.sid = R.sid AND R.bid = B.bid AND B.color = `red' ANDS.rating = ( SELECT MAX (S2.rating)FROM Sailors S2 )GROUP BY S.sidHAVING COUNT (*) > 1SELECT MAX (S2.rating)FROM Sailors S2SELECT S.sid, MIN (R.day)FROM Sailors S, Reserves R, Boats BWHERE S.sid = R.sid AND R.bid = B.bid AND B.color = `red' ANDS.rating = Reference to nested blockGROUP BY S.sidHAVING COUNT (*) > 1Nested blockOuter block

Query Blocks: Units of OptimizationAn SQL query is parsed into a collection of query blocks, and these are optimized one block at a time.Nested blocks are usually treated as calls to a subroutine, made once per outer tuple. (This is an oversimplification, but serves for now.)For each block, the plans considered are:All available access methods, for each reln in FROM clause.All left-deep join trees (i.e., all ways to join the relations one at-a-time, with the inner reln in the FROM clause, considering all reln permutations and join methods.)

Estimating the cost of plansThe cost of a plan consistThe cost of performing the operation of a node in QEPThe size of the result of each node in QEP

Estimating the result sizesSize =the product of the cardinalities of the relations in the FROM clause* reduction factors of where clauseReduction factor: the ratio of the (expected) result size to the input size considering only the selection represented by the termSELECT attribute listFROM relation listWHERE term1 ^ term2 ^ : : : ^ termn

Simplified estimateColumn=value1/NKeys(I)column1 = column2:1/MAX (NKeys(I1);NKeys(I2))column > value: (High(I) value)/(High(I) Low(I))column IN (list of values):the reduction factor for column = value multiplied by the number of items in the listThe assumption of above estimation: uniform data distribution

More accurate estimateUniform vs nonuiform distributionsConsider the relation with 45 tuples and the seletion age>13, using uniform assumption, 45*(1/15)=3; the real value is 9