SQL Pivot and Prune Queries - Keeping an Eye on Performance
-
Upload
brendan-furey -
Category
Documents
-
view
120 -
download
7
description
Transcript of SQL Pivot and Prune Queries - Keeping an Eye on Performance
SQL PIVOT AND PRUNE QUERIES - KEEPING AN EYE ON PERFORMANCE
Author: Brendan Furey
Creation Date: 2 May 2011
Version: 1.3
Last Updated: 25 September 2012
document.doc . Page 1 of 53
Table of Contents
Introduction 4Hardware/Software Summary 4
Overview 5Problem Description 5Test Data Sets 6Pivot and Prune Query Strategies 6
Pivot Strategies 6Prune Strategies 7Strategy Modifiers 7Test Combinations 7
Testing Strategy 8Output Files 8
The Queries 9JNSQ: Join, Subquery 9
Query Text 9Query Diagram 10Execution Plan Example (W64-D16) 10Results 11
WJSQ: With, Subquery 12Query Text 12Query Diagram 13Execution Plan Example (W128-D64) 13Results 14
WJAN: With, Analytic 15Query Text 15Query Diagram 17Execution Plan Example (W128-D64) 17Results 18
WJKP: With, Keep 19Query Text 19Query Diagram 20Execution Plan Example (W128-D64) 20Results 21
PVAN: Pivot, Analytic 22Query Text 22Query Diagram 22Execution Plan Example (W128-D64) 22Results 23
PVANIV: Pivot, Analytic, View 24Query Text 24Query Diagram 24Execution Plan Example (W128-D64) 25Results 25
PVKP: Pivot, Keep 26Query Text 26Query Diagram 26Execution Plan Example (W128-D64) 27Results 27
PVKPIV: Pivot, Keep, View 28Query Text 28Query Diagram 28Execution Plan Example (W128-D64) 29Results 29
FNSC: Database Function 30document.doc Page 2 of 53
Query Text - Main 30Query Text - Function 30Query/Function Diagram 31Execution Plan Example - Main 31Execution Plan Example – Function 31Results 32
SSKP: Select Scalar Subquery, Keep33Query Text 33Query Diagram 34Execution Plan Example (W1-D1024) 34Results 34
SSSM: Select Scalar Subquery, Max 34Query Text 34Query Diagram 35Execution Plan Example (W1-D1024) 35Results 35
SSOB: Select Scalar Subquery, Order By 35Query Text 35Query Diagram 36Results 36
Analysis of Results 37Slice Analysis 37
Narrow Slice 37Wide Slice 38Shallow Slice 40Deep Slice 41
Summary Analysis 43Tables and Graphs of Relative Performance 43Which Query is Best? 44
Cost Based Optimizer 44CBO Cardinalities 44Inline View Modifier 45Subquery Factoring Modifier 45Performance of JNSQ 45Performance of Database Function and Scalar Subquery Strategies 46General Conclusions 46
Query Testing Program 47Call Structure Diagram 47Data Flow Diagram 48Table Structures 48
Generic Tables 48Specific Tables for our HR Queries 49
Program Logic 49Example Output 50
References 52
Change RecordDate Author Version Change Reference
02-May-2011 BPF 1.0 Initial03-May-2011 BPF 1.1 Typos and references20-Jun-2011 BPF 1.2 More typos and reference to new article25-Sep-2011 BPF 1.3 References now hyperlinks, and minor tidying
document.doc Page 3 of 53
IntroductionIt is a very common requirement in SQL to join two record sets where there is a one to many relationship between the two sets, but where the cardinality of the result set is the same as that of the set on the 'one' side. The obvious case is for standard grouping and aggregation querying, such as simply counting the number of records in the 'many' set for each record in the 'one' set. There are also some slightly less obvious cases where there may be various SQL techniques available, with varying performance and complexity characteristics. This article looks at two such cases: the first where one wishes to join multiple subtypes of a given entity – this is generally referred to as ‘pivoting’ from rows to columns; and the second where one wishes to join just one record from the 'many' set, but does not have a pure join condition to identify the record and so must use an ordering condition instead – I will call this ‘pruning’.
This work attempts to find the best SQL techniques for such queries in Oracle 11g, mainly in terms of performance. It does this by running a variety of queries within the context of an outbound interface against a deliberately simple data model across a two-dimensional range of data sizes. A simple generic PL/SQL package has been written to perform the testing efficiently, and it uses a previously described object type for timing (a more recent version is described here: Code Timing and Object Orientation and Zombies). Visio diagrams are provided for query structures, based on a similar approach previously described (A Structured Approach to SQL Query Design), and Microsoft Excel graphs are used to display comparative performances. The results reveal some interesting features of the behaviour of the Cost Based Optimiser in Oracle 11g.
The output file is provided here.
Hardware/Software SummaryComponent DescriptionDatabase Oracle Database 11g Express Edition Release 11.2.0.2.0 - BetaDiagrammer Microsoft Visio 2003 (11.3216.5606)Grapher Microsoft Excel 2007Operating System Microsoft Windows 7 Home Premium (32 bit)Computer Samsung X120, 3GB memory, Intel U4100 @ 1.3GHz x 2 (XE only uses 1 though)
document.doc . Page 4 of 53
Overview
Problem DescriptionThe SQL problem is to obtain a single output record for each master record, to include a single ‘best’ value of a field in the detail record for each of several subtypes.
We consider a simple example scenario where the master table is employees in Oracle’s demo HR schema, and the detail table is a new table, phone_numbers, a simplified version of hz_contact_points in Oracle Applications. The phone_numbers table has records of four subtypes: HOME, WORK, MOBILE, FAX, and a valid from date, where the most recent number is preferred for a given type, with the maximum number used as a tie-breaker. We’ll assume that an employee should always be returned even if he has no phone numbers.
In the real world, this kind of query might be used in an outbound interface that supplies the latest contact details on a company’s customers to a third-party supplier for marketing or other purposes. In that kind of case, there would typically be a small number of detail records across a relative large table. This would be the wide-shallow scenario discussed below. Another kind of real world example might be
document.doc Page 5 of 53
a CRM (Customer Relations Management) system in which one wants a report on the most recent customer interactions, and this could fall into the wide/narrow-deep scenarios.
Test Data SetsWe will test the performance of the queries across a range of sizes of the master table, denoted by a width dimension, and the detail table, denoted by a depth dimension. The maximum sizes are constrained by the execution time and storage capacity on a small laptop running Oracle 11g XE, so we will use a wide data set and a deep data set for better coverage. The data sets are shown in the table below, where the cells contain the number of detail records against each dimension pair, and the second row (# W =>) has the numbers of master records.
Starting from the base sizes (W1, D1) of 4 details (one of each subtype) for each of 107 master records (being the number of employees in the HR demo schema), the dimensions double successively. Test data are generated by copying the 107 seed employees with a sequence number added to the names, and generating random 9-digit phone numbers with the valid from date a random day within a given period. We used a single year for the wide data set, which would have a lot of duplicates, and a 50 year period for the deep data set, with fewer duplicates (the difference didn’t affect performance significantly). In order to test outer-joinery, one employee has all phone numbers deleted and another has all FAX numbers deleted (that’s why the cell values are slightly smaller than the products of the dimension values). Rows D1-D64 form the wide data set, while columns W1-W8 form the deep data set.
# D / W W1 W2 W4 W8 W16 W32 W64 W128# W => 107 214 428 856 1,712 3,424 6,848 13,696
D1 4 423 851 1,707 3,419 6,843 13,691 27,387 54,779D2 8 846 1,702 3,414 6,838 13,686 27,382 54,774 109,558
D4 16 1,692 3,404 6,828 13,676 27,372 54,764 109,548 219,116D8 32 3,384 6,808 13,656 27,352 54,744 109,528 219,096 438,232
D16 64 6,768 13,616 27,312 54,704 109,488 219,056 438,192 876,464D32 128 13,536 27,232 54,624 109,408 218,976 438,112 876,384 1,752,928
D64 256 27,072 54,464 109,248 218,816 437,952 876,224 1,752,768 3,505,856D128 512 54,144 108,928 218,496 437,632 875,904 1,752,448 3,505,536 7,011,712
D256 1,024 108,288 217,856 436,992 875,264 1,751,808 3,504,896 7,011,072 14,023,424D512 2,048 216,576 435,712 873,984 1,750,528 3,503,616 7,009,792 14,022,144 28,046,848
D1024 4,096 433,152 871,424 1,747,968 3,501,056 7,007,232 14,019,584 28,044,288 56,093,696
Pivot and Prune Query Strategies
Pivot Strategies
Oracle Database 11g introduced an explicit SQL syntax for pivoting rows to columns, using the new keyword PIVOT. Prior to 11g one could simulate this by the use of grouping and CASE expressions, but there seems little value in considering this approach now. There are other approaches however that are sufficiently different for it to be interesting to evaluate them.
Pivot Strategy Notes
Pivot Oracle 11g explicit PIVOT syntax.
Join
Repeat the tables and joins for each subtype. This is quite a commonly used strategy but might be expected to be inefficient due to additional table reads. Efficiency can be improved by combining with the Oracle 10g feature of subquery factoring, using the WITH keyword.
Select Scalar Subquery
Obtain each subtype column using a separate scalar subquery on the detail table in the Select clause, correlated to the master record. The detail table then does not appear in the main query. This might also be expected to be inefficient since multiple queries are effectively repeated for every row; however, there may be scenarios where this effect is outweighed by benefits such as improved index usage.
Database function
This is similar to the select scalar subquery strategy in that the detail table is removed from the main query, but here is moved into a separate function call. This might be
document.doc Page 6 of 53
expected to have similar performance characteristics, but with some added overhead from the PL/SQL context switch; it also has a theoretical disadvantage in that read-consistency is not guaranteed. This strategy is quite widely used
Prune Strategies
Oracle Database version 9.0.1 introduced an explicit SQL syntax for pruning rows via aggregate functions, using the keyword KEEP, with FIRST or LAST. Prior to this, pruning could be achieved by combining various non-purpose-built techniques, and it seems that these alternatives continue to be widely used today.
Prune Strategy Notes
Keep
The maximum function can be applied to an expression on the detail record with the KEEP clause specifying that the maximum is over only those records ranking first (or last) in a given ordering. This is Oracle’s explicit pruning syntax and so might be expected to perform well. From the SQL manual (REF-1): When you need a value from the first or last row of a sorted group, but the needed value is not the sort key, the FIRST and LAST functions eliminate the need for self-joins or views and enable better performance.
Analytic sThe detail table can be placed within an inline view where the analytic function Row_Number can be selected to rank the records; a condition is then placed outside the view that the rank equals 1.
Select Scalar Subquery
The same approach as mentioned above for pivoting can also be used for pruning, where the pruning takes place within its own subquery in the Select clause; there are various possible subquery implementations
Join Scalar Subquery
Select an expression defining the preferred record in a subquery correlated to the master record, and join the main record. Aside from the obvious performance disadvantage of additional reads of the detail table, applying this strategy directly prevents outer joining; however, when used in conjunction with subquery factoring, outer joining to the factor is possible
Order By
The detail records are selected within an inline view in the preferred order, then a condition is applied outside the view that ROWNUM = 1. This has to be used within a database function: we tried to use it within a select scalar subquery but the SQL is invalid.
MaxWithin a select scalar subquery a maximum is taken of an expression that prepends the ordering fields to the desired select field, using appropriate type casting (e.g. taking the Julian format for a date)
Strategy Modifiers
There are some general SQL techniques that can be used in conjunction with the above pivot and prune strategies in some cases, and that may affect performance.
Strategy Modifier Notes
Subquery Factoring
This feature was introduced in Oracle 10g, and, using the WITH keyword, allows a subquery to be defined once and subsequently referenced several times by its alias. Where appropriate Oracle will run the subquery in advance and store the results in a temporary table to improve performance. Unlike with subqueries in the where clause of a query, outer joins can be used.
Inline View
An inline view is a subquery in brackets in the position of a table in the FROM clause of a query. It can affect performance by influencing the execution plan, as we’ll see.
Test Combinations
The table below lists the queries tested with the strategies employed. Note that not all combinations of strategies make sense, but we have tried to cover all that do.
document.doc Page 7 of 53
Code Pivot Prune Modifier NotesMain query set tested on both wide and deep data sets, except JNSQ wide onlyJNSQ Join Join Scalar SubqueryWJSQ Join Join Scalar Subquery Subquery FactoringWJAN Join Analytic s Subquery FactoringWJKP Join Keep Subquery FactoringPVAN Pivot Analytic s
PVANIV Pivot Analytic s Inline ViewPruning is done within the inline view
PVKP Pivot Keep
PVKPIV Pivot Keep Inline ViewPruning is done within the inline view
FNSC Database Function Order By Inline ViewAdditional queries tested on W1-D1024 only - we tried to match the performance characteristics of FNSC on the narrow-deep data set without using a database function
SSSM Select Scalar Subquery Max
SSKP Select Scalar Subquery Keep
SSOB Select Scalar Subquery Order By Inline View
Same pruning strategy as FNSC but within query. The query is invalid owing to the need to correlate through two levels
Testing StrategyIn order to provide a realistic scenario, the queries are executed within the context of a simple outbound interface that writes each record to a file as a comma-separated string. A small PL/SQL package has been written to automate the testing. The program loops over width and depth dimensions, and for each data set point makes a call to a separate package to set up the test data and have the CBO statistics gathered; it then loops over a set of queries defined in the same separate package as strings that are executed by the main package. Detailed timings are made using the timing objects mentioned earlier and written to a generic log table. The total CPU and elapsed times for the query and data set point are written to a special run statistics table, from which they can be output in a form easily loaded into Microsoft Excel. The times can be viewed as a function of the 2-dimensional width-depth domain, and displayed as a 3-dimensional surface graph for each query.
The execution plan is obtained in each case, using an Oracle API, and is written to the generic log. The query string includes a random number that guarantees a hard-parse and thus recalculation of the execution plans at each data set point. Further details of the packages and supporting tables are given in a later section.
The testing strategy here seemed to me to provide much greater insight than the more conventional approach of testing on a single large data set, and was later used again in Forming Range-Based Break Groups with Advanced SQL, and other articles.
Output FilesThe complete output files are provided in SQL Pivot and Prune Queries – Output.
LST File NotesTest_Phone_8-7.LST Wide data set – first 9 queriesTest_Phone_4-11.LST Deep data set - first 9 queries, except JNSQTest_Phone_1-11.LST W1-D1024, SSKP, SSSM, FNSC – testing select scalar subqueries
document.doc Page 8 of 53
Test_Phone_1-11_3.LST W1-D1024, PVKP, PVKPG, PVKPIV – CBO grouping problem
document.doc Page 9 of 53
The QueriesFor each query subsection, the following are provided (for the main queries, the last three were added later and have a bit less):
The query text in static form for readability; the program in fact uses dynamic SQL and consequently the random number at the end of the CSV string ensures that the query is hard-parsed for each data set point
A query diagram. The diagrams use extended versions of notation first described in A Structured Approach to SQL Query Design
The last Execution Plan output on the wide data set (usually)
A results section, split into wide and deep data sets, with tables of CPU times and 3-d graphs generated from Microsoft Excel. Analysis is reserved for a later section
JNSQ: Join, Subquery
Query TextSELECT /* JNSQ*/'"' || emp.first_name || ' ' || emp.last_name || '","' || pho_h.phone_number || '","' || pho_w.phone_number || '","' || pho_m.phone_number || '","' || pho_f.phone_number || '8098"' FROM employees emp JOIN phone_numbers pho_h ON pho_h.employee_id = emp.employee_id AND pho_h.phone_type = 'HOME' AND TO_CHAR (pho_h.valid_from, 'J') || pho_h.phone_number = (
SELECT MAX (TO_CHAR (sbq_h.valid_from, 'J') || sbq_h.phone_number) FROM phone_numbers sbq_h WHERE sbq_h.employee_id = pho_h.employee_id AND sbq_h.phone_type = pho_h.phone_type)
JOIN phone_numbers pho_w ON pho_w.employee_id = emp.employee_id AND pho_w.phone_type = 'WORK' AND TO_CHAR (pho_w.valid_from, 'J') || pho_w.phone_number = (
SELECT MAX (TO_CHAR (sbq_w.valid_from, 'J') || sbq_w.phone_number) FROM phone_numbers sbq_w WHERE sbq_w.employee_id = pho_w.employee_id AND sbq_w.phone_type = pho_w.phone_type)
JOIN phone_numbers pho_m ON pho_m.employee_id = emp.employee_id AND pho_m.phone_type = 'MOBILE' AND TO_CHAR (pho_m.valid_from, 'J') || pho_m.phone_number = (
SELECT MAX (TO_CHAR (sbq_m.valid_from, 'J') || sbq_m.phone_number) FROM phone_numbers sbq_m WHERE sbq_m.employee_id = pho_m.employee_id AND sbq_m.phone_type = pho_m.phone_type)
JOIN phone_numbers pho_f ON pho_f.employee_id = emp.employee_id AND pho_f.phone_type = 'FAX' AND TO_CHAR (pho_f.valid_from, 'J') || pho_f.phone_number = (
SELECT MAX (TO_CHAR (sbq_f.valid_from, 'J') || sbq_f.phone_number) FROM phone_numbers sbq_f WHERE sbq_f.employee_id = pho_f.em ployee_id AND sbq_f.phone_type = pho_f.phone_type)
ORDER BY 1
document.doc Page 10 of 53
Query Diagram
Execution Plan Example (W64-D16)
In the example below, the top level cardinality estimate of 1 is clearly poor, since the number of rows returned is the number of employees less 2 due to the inner joins where phone numbers were deleted, giving 6,846. In addition, it is highly unlikely that not driving from the master table could be optimal.
--------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |--------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 1777 (100)| |
document.doc Page 11 of 53
| 1 | SORT ORDER BY | | 1 | 220 | 1777 (2)| 00:00:22 || 2 | NESTED LOOPS | | 1 | 220 | 1776 (1)| 00:00:22 || 3 | NESTED LOOPS | | 1 | 201 | 1710 (1)| 00:00:21 || 4 | NESTED LOOPS | | 1 | 182 | 1644 (2)| 00:00:20 || 5 | NESTED LOOPS | | 1 | 155 | 1642 (2)| 00:00:20 || 6 | NESTED LOOPS | | 1 | 128 | 1577 (2)| 00:00:19 || 7 | NESTED LOOPS | | 3 | 327 | 1379 (2)| 00:00:17 || 8 | NESTED LOOPS | | 1 | 82 | 1314 (2)| 00:00:16 ||* 9 | HASH JOIN | | 1 | 55 | 1313 (2)| 00:00:16 || 10 | VIEW | VW_SQ_1 | 4840 | 99K| 658 (2)| 00:00:08 || 11 | HASH GROUP BY | | 4840 | 122K| 658 (2)| 00:00:08 ||* 12 | TABLE ACCESS FULL | PHONE_NUMBERS | 109K| 2774K| 653 (1)| 00:00:08 ||* 13 | TABLE ACCESS FULL | PHONE_NUMBERS | 109K| 3628K| 654 (1)| 00:00:08 || 14 | TABLE ACCESS BY INDEX ROWID| EMPLOYEES | 1 | 27 | 1 (0)| 00:00:01 ||* 15 | INDEX UNIQUE SCAN | EMP_EMP_ID_PK | 1 | | 0 (0)| ||* 16 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 432 | 65 (0)| 00:00:01 ||* 17 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 2 (0)| 00:00:01 ||* 18 | VIEW PUSHED PREDICATE | VW_SQ_4 | 1 | 19 | 66 (0)| 00:00:01 || 19 | SORT GROUP BY | | 1 | 26 | 66 (0)| 00:00:01 ||* 20 | TABLE ACCESS BY INDEX ROWID| PHONE_NUMBERS | 16 | 416 | 66 (0)| 00:00:01 ||* 21 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 3 (0)| 00:00:01 ||* 22 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 432 | 65 (0)| 00:00:01 ||* 23 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 2 (0)| 00:00:01 ||* 24 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 432 | 2 (0)| 00:00:01 ||* 25 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 2 (0)| 00:00:01 ||* 26 | VIEW PUSHED PREDICATE | VW_SQ_3 | 1 | 19 | 66 (0)| 00:00:01 || 27 | SORT GROUP BY | | 1 | 26 | 66 (0)| 00:00:01 ||* 28 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 416 | 66 (0)| 00:00:01 ||* 29 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 3 (0)| 00:00:01 ||* 30 | VIEW PUSHED PREDICATE | VW_SQ_2 | 1 | 19 | 66 (0)| 00:00:01 || 31 | SORT GROUP BY | | 1 | 26 | 66 (0)| 00:00:01 ||* 32 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 16 | 416 | 66 (0)| 00:00:01 ||* 33 | INDEX RANGE SCAN | PHONE_NUMBERS_N2 | 64 | | 3 (0)| 00:00:01 |--------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
9 - access("VW_COL_1"=TO_CHAR(INTERNAL_FUNCTION("VALID_FROM"),'j')||"PHONE_NUMBER" AND "ITEM_1"="PHO_F"."EMPLOYEE_ID" AND "ITEM_2"="PHO_F"."PHONE_TYPE") 12 - filter("SBQ_F"."PHONE_TYPE"='FAX') 13 - filter("PHO_F"."PHONE_TYPE"='FAX') 15 - access("PHO_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 16 - filter("PHO_H"."PHONE_TYPE"='HOME') 17 - access("PHO_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 18 - filter("VW_COL_1"="PHO_H"."SYS_NC00007$") 20 - filter("SBQ_H"."PHONE_TYPE"='HOME') 21 - access("SBQ_H"."EMPLOYEE_ID"="PHO_H"."EMPLOYEE_ID") 22 - filter("PHO_M"."PHONE_TYPE"='MOBILE') 23 - access("PHO_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 24 - filter("PHO_W"."PHONE_TYPE"='WORK') 25 - access("PHO_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 26 - filter("VW_COL_1"="PHO_W"."SYS_NC00007$") 28 - filter("SBQ_W"."PHONE_TYPE"='WORK') 29 - access("SBQ_W"."EMPLOYEE_ID"="PHO_W"."EMPLOYEE_ID") 30 - filter("VW_COL_1"="PHO_M"."SYS_NC00007$") 32 - filter("SBQ_M"."PHONE_TYPE"='MOBILE') 33 - access("SBQ_M"."EMPLOYEE_ID"="PHO_M"."EMPLOYEE_ID")
Results
Wide Data SetTable of CPU TimesNote that the testing program iterates over the depth dimension within the width dimension loop, and stops the inner iteration once a CPU time of 300 seconds has been exceeded, hence the gaps.
Query W1 W2 W4 W8 W16 W32 W64 W128D1 1.35 1.73 2 2.15 2.34 4.01 7.3 13.53D2 1.29 1.94 2.71 3.49 5.37 10.26 20.02 34.59D4 1.51 2.39 4.02 8.99 11.63 21.29 34.43 67.66D8 1.89 4.28 7.47 24.91 15.56 28.7 154.86 298.62D16 2.95 7.83 163.68 320.74 608 85.89 878.53 324.23D32 5.98 16.54 55.29 213.4D64 14.59 59.48 159.65 292.08
Graph
document.doc Page 12 of 53
Deep Data Set (not tested as JNSQ clearly uncompetitive)Notes
The performance across the data set points is very erratic. For example, for D16, doubling the width dimension from W16 to W32 caused the CPU time to drop from 608s to 86s. This clearly indicates that the CBO had followed a highly sub-optimal execution plan for the former. This query is the most complicated of those tested, having 9 table instances, but it may still be considered surprising that performance (of the CBO) was so poor for what in the real world would not be an especially complex query. It is not part of the scope to analyse the execution plans in detail, but one may observe that the worse ones do three full scans of the detail table while the better ones do only two.
It might be worth noting that the best execution plan would most likely decompose into one subplan for obtaining the detail records and joining them to an employee-based record set, repeated four times, plus one to obtain the employees (which would drive). However, the CBO would not be likely to see it that way.
WJSQ: With, Subquery
Query TextWITH wit AS (SELECT pho.employee_id,
pho.phone_type,pho.phone_number
FROM phone_numbers pho WHERE TO_CHAR (pho.valid_from, 'J') || pho.phone_number = (
SELECT MAX (TO_CHAR (sbq.valid_from, 'J') || sbq.phone_number) FROM phone_numbers sbq WHERE sbq.employee_id = pho.employee_id AND sbq.phone_type = pho.phone_type)
)SELECT /* WJSQ*/'"' || emp.first_name || ' ' || emp.last_name || '","' || wit_h.phone_number || '","' || wit_w.phone_number || '","' || wit_m.phone_number || '","' || wit_f.phone_number || '4121"' FROM employees emp LEFT JOIN wit wit_h ON wit_h.employee_id = emp.employee_id AND wit_h.phone_type = 'HOME' LEFT JOIN wit wit_w ON wit_w.employee_id = emp.employee_id AND wit_w.phone_type = 'WORK' LEFT JOIN wit wit_m ON wit_m.employee_id = emp.employee_id AND wit_m.phone_type = 'MOBILE' LEFT JOIN wit wit_f ON wit_f.employee_id = emp.employee_id AND wit_f.phone_type = 'FAX' ORDER BY 1
document.doc Page 13 of 53
Query Diagram
Execution Plan Example (W128-D64)
The subquery factoring system table is actually named SYS_TEMP_0FD9D66F3_71C4A6, but I truncated it to avoid line wrapping below.
Notice how poor the cardinality estimate of 1 is on the subquery factor table. The actual number would be approximately four times the number of employees (13,696), i.e. 54,784, owing to the subquery correlation. On the other hand, the final cardinality estimate is about right.
-------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-------------------------------------------------------------------------------------------------------
document.doc Page 14 of 53
| 0 | SELECT STATEMENT | | | | | 28090 (100)| || 1 | TEMP TABLE TRANSFORMATION | | | | | | || 2 | LOAD AS SELECT | | | | | | ||* 3 | HASH JOIN | | 1 | 64 | 1592K| 27704 (2)| 00:05:33 || 4 | VIEW | VW_SQ_1 | 38739 | 1134K| | 14699 (2)| 00:02:57 || 5 | SORT GROUP BY | | 38739 | 983K| 121M| 14699 (2)| 00:02:57 ||* 6 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 87M| | 5285 (2)| 00:01:04 || 7 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 113M| | 5265 (1)| 00:01:04 || 8 | SORT ORDER BY | | 13583 | 1313K| 1520K| 386 (2)| 00:00:05 ||* 9 | HASH JOIN RIGHT OUTER | | 13583 | 1313K| | 78 (3)| 00:00:01 ||* 10 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 11 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 ||* 12 | HASH JOIN RIGHT OUTER | | 13583 | 1074K| | 76 (3)| 00:00:01 ||* 13 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 14 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 ||* 15 | HASH JOIN RIGHT OUTER | | 13583 | 835K| | 73 (2)| 00:00:01 ||* 16 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 17 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 ||* 18 | HASH JOIN RIGHT OUTER| | 13583 | 596K| | 71 (2)| 00:00:01 ||* 19 | VIEW | | 1 | 18 | | 2 (0)| 00:00:01 || 20 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66F| 1 | 19 | | 2 (0)| 00:00:01 || 21 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 |-------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
3 - access("VW_COL_1"=TO_CHAR(INTERNAL_FUNCTION("VALID_FROM"),'j')||"PHONE_NUMBER" AND "ITEM_0"="PHO"."EMPLOYEE_ID" AND "ITEM_1"="PHO"."PHONE_TYPE") 6 - filter(("SBQ"."PHONE_TYPE"='FAX' OR "SBQ"."PHONE_TYPE"='HOME' OR "SBQ"."PHONE_TYPE"='MOBILE' OR "SBQ"."PHONE_TYPE"='WORK')) 9 - access("WIT_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 10 - filter("WIT_F"."PHONE_TYPE"='FAX') 12 - access("WIT_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 13 - filter("WIT_M"."PHONE_TYPE"='MOBILE') 15 - access("WIT_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 16 - filter("WIT_W"."PHONE_TYPE"='WORK') 18 - access("WIT_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 19 - filter("WIT_H"."PHONE_TYPE"='HOME')
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.13 0.22 0.24 0.37 0.67 1.17 2.16 4.44D2 0.14 0.19 0.28 0.41 0.68 1.29 2.26 4.62D4 0.15 0.18 0.28 0.45 0.75 1.39 2.75 5.05D8 0.16 0.24 0.31 0.56 0.93 1.67 3.3 6.14D16 0.16 0.28 0.38 0.63 1.17 2.19 4.34 8.7D32 0.22 0.3 0.5 0.93 1.73 3.4 6.47 13.22D64 0.3 0.42 0.79 1.42 2.73 5.7 11.6 22.62
Graph
document.doc Page 15 of 53
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.19 0.21 0.23 0.38D2 0.19 0.23 0.25 0.41D4 0.14 0.2 0.26 0.42D8 0.15 0.21 0.29 0.48D16 0.17 0.22 0.39 0.61D32 0.22 0.3 0.54 0.88D64 0.28 0.47 0.73 1.42D128 0.44 0.67 1.28 2.5D256 0.62 1.19 2.38 4.59D512 1.11 2.15 4.45 9.54D1024 1.43 2.76 5.74 12.59
Graph
WJAN: With, Analytic
Query TextWITH wit AS (SELECT ilv.employee_id,
ilv.phone_type,ilv.pho_number
document.doc Page 16 of 53
FROM (SELECT pho.employee_id,
pho.phone_type,pho.phone_number,Row_Number() OVER (PARTITION BY PHO.employee_id, pho.phone_type ORDER BY pho.valid_from
DESC, pho.phone_number) ind FROM phone_numbers pho
) ilv WHERE ilv.ind = 1)SELECT /* WJAN*/'"' || emp.first_name || ' ' || emp.last_name || '","' || wit_h.phone_number || '","' || wit_w.phone_number || '","' || wit_m.phone_number || '","' || wit_f.phone_number || '3006"' FROM employees emp LEFT JOIN wit wit_h ON wit_h.employee_id = emp.employee_id AND wit_h.phone_type = 'HOME' LEFT JOIN wit wit_w ON wit_w.employee_id = emp.employee_id AND wit_w.phone_type = 'WORK' LEFT JOIN wit wit_m ON wit_m.employee_id = emp.employee_id AND wit_m.phone_type = 'MOBILE' LEFT JOIN wit wit_f ON wit_f.employee_id = emp.employee_id AND wit_f.phone_type = 'FAX' ORDER BY 1
document.doc Page 17 of 53
Query Diagram
Execution Plan Example (W128-D64)
The subquery factoring system table is actually named SYS_TEMP_0FD9D66F2_71C4A6, but I truncated it to avoid line wrapping below.
Notice how poor the cardinality estimate of 3,509,000 (about the number of phone numbers) is on the subquery factor table. The actual number would be approximately four times the number of employees (13,696), i.e. 54,784. Also, the final cardinality estimate is very poor, presumably just because of the initial mistake - 63T means, I think, 63,000,000,000,000, when the actual number would be the number of employees (13,696),
-----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 14G(100)| || 1 | TEMP TABLE TRANSFORMATION | | | | | | || 2 | LOAD AS SELECT | | | | | | ||* 3 | VIEW | | 3509K| 107M| | 31925 (1)| 00:06:24 ||* 4 | WINDOW SORT PUSHED RANK| | 3509K| 90M| 134M| 31925 (1)| 00:06:24 || 5 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 || 6 | SORT ORDER BY | | 63T| 5720T| 6574T| 14G (41)|999:59:59 ||* 7 | HASH JOIN RIGHT OUTER | | 63T| 5720T| 100M| 1494M (18)|999:59:59 ||* 8 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 9 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 |
document.doc Page 18 of 53
|* 10 | HASH JOIN RIGHT OUTER | | 242G| 17T| 100M| 4830K (21)| 16:06:11 ||* 11 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 12 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 ||* 13 | HASH JOIN RIGHT OUTER | | 928M| 54G| 100M| 30789 (13)| 00:06:10 ||* 14 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 15 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 ||* 16 | HASH JOIN OUTER | | 3552K| 152M| | 5146 (1)| 00:01:02 || 17 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 ||* 18 | VIEW | | 3509K| 60M| | 5063 (1)| 00:01:01 || 19 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6| 3509K| 63M| | 5063 (1)| 00:01:01 |-----------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
3 - filter("ILV"."IND"=1) 4 - filter(ROW_NUMBER() OVER ( PARTITION BY "PHO"."EMPLOYEE_ID","PHO"."PHONE_TYPE" ORDER BY INTERNAL_FUNCTION("PHO"."VALID_FROM") DESC ,"PHO"."PHONE_NUMBER")<=1) 7 - access("WIT_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 8 - filter("WIT_F"."PHONE_TYPE"='FAX') 10 - access("WIT_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 11 - filter("WIT_M"."PHONE_TYPE"='MOBILE') 13 - access("WIT_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 14 - filter("WIT_W"."PHONE_TYPE"='WORK') 16 - access("WIT_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 18 - filter("WIT_H"."PHONE_TYPE"='HOME')
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.14 0.14 0.24 0.34 0.64 1.06 2.11 4.31D2 0.14 0.16 0.27 0.44 0.58 1.11 2.25 4.39D4 0.14 0.19 0.28 0.45 0.67 1.25 2.39 4.97D8 0.14 0.2 0.28 0.47 0.83 1.53 3.07 6.19D16 0.16 0.23 0.39 0.69 1.13 2.13 4.42 8.72D32 0.17 0.32 0.52 0.97 1.74 3.48 7.05 12.49D64 0.34 0.45 0.85 1.65 3.41 6.71 10.42 21.31
Graph
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.15 0.2 0.27 0.32D2 0.14 0.19 0.26 0.39D4 0.14 0.19 0.27 0.42D8 0.14 0.19 0.27 0.44
document.doc Page 19 of 53
D16 0.17 0.23 0.32 0.61D32 0.18 0.25 0.47 0.89D64 0.24 0.44 0.86 1.58D128 0.42 0.78 1.62 3.09D256 0.8 1.56 3.18 6.44D512 1.56 3.17 6.68 9.89D1024 3.36 6.94 9.86 19.2
Graph
WJKP: With, Keep
Query TextWITH wit AS (SELECT pho.employee_id,
pho.phone_type, MAX (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) phone_number
FROM phone_numbers pho GROUP BY pho.employee_id, pho.phone_type)SELECT /* WJKP*/'"' || emp.first_name || ' ' || emp.last_name || '","' || wit_h.phone_number || '","' || wit_w.phone_number || '","' || wit_m.phone_number || '","' || wit_f.phone_number || '1970"' FROM employees emp LEFT JOIN wit wit_h ON wit_h.employee_id = emp.employee_id AND wit_h.phone_type = 'HOME' LEFT JOIN wit wit_w ON wit_w.employee_id = emp.employee_id AND wit_w.phone_type = 'WORK' LEFT JOIN wit wit_m ON wit_m.employee_id = emp.employee_id AND wit_m.phone_type = 'MOBILE' LEFT JOIN wit wit_f ON wit_f.employee_id = emp.employee_id AND wit_f.phone_type = 'FAX' ORDER BY 1
document.doc Page 20 of 53
Query Diagram
Execution Plan Example (W128-D64)
The subquery factoring system table is actually named SYS_TEMP_0FD9D66F3_71C4A6, but I truncated it to avoid line wrapping below.
Notice that the cardinality estimate of 38,739 on the subquery factor table is not too far off the actual number, that would be approximately four times the number of employees (13,696), i.e. 54,784. On the other hand, the final cardinality estimate of 943,000 is a long way off the actual number, 13,696.
------------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 38505 (100)| || 1 | TEMP TABLE TRANSFORMATION | | | | | | || 2 | LOAD AS SELECT | | | | | | || 3 | SORT GROUP BY | | 38739 | 1021K| 134M| 14919 (2)| 00:03:00 || 4 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 || 5 | SORT ORDER BY | | 943K| 89M| 102M| 23586 (1)| 00:04:44 ||* 6 | HASH JOIN RIGHT OUTER | | 943K| 89M| 1136K| 2351 (1)| 00:00:29 ||* 7 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 8 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 ||* 9 | HASH JOIN RIGHT OUTER | | 326K| 25M| 1136K| 812 (1)| 00:00:10 ||* 10 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 11 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 ||* 12 | HASH JOIN RIGHT OUTER | | 113K| 6963K| 1136K| 312 (1)| 00:00:04 ||* 13 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 14 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 ||* 15 | HASH JOIN OUTER | | 39210 | 1723K| | 109 (1)| 00:00:02 || 16 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 ||* 17 | VIEW | | 38739 | 680K| | 40 (0)| 00:00:01 || 18 | TABLE ACCESS FULL | SYS_TEMP_0FD9D66| 38739 | 643K| | 40 (0)| 00:00:01 |------------------------------------------------------------------------------------------------------
document.doc Page 21 of 53
Predicate Information (identified by operation id):---------------------------------------------------
6 - access("WIT_F"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 7 - filter("WIT_F"."PHONE_TYPE"='FAX') 9 - access("WIT_M"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 10 - filter("WIT_M"."PHONE_TYPE"='MOBILE') 12 - access("WIT_W"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 13 - filter("WIT_W"."PHONE_TYPE"='WORK') 15 - access("WIT_H"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 17 - filter("WIT_H"."PHONE_TYPE"='HOME')
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.15 0.18 0.23 0.36 0.62 1.09 2.07 4.16D2 0.15 0.17 0.26 0.32 0.66 1.08 2.09 4.04D4 0.14 0.17 0.25 0.34 0.69 1.14 2.28 4.4D8 0.14 0.17 0.25 0.38 0.72 1.28 2.45 4.93D16 0.15 0.2 0.28 0.44 0.81 1.52 3.07 6.07D32 0.15 0.21 0.34 0.56 1.05 2 4.17 8.08D64 0.2 0.3 0.48 0.78 1.53 3.04 6.05 12.29
Graph
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.14 0.14 0.21 0.38D2 0.14 0.16 0.23 0.36D4 0.14 0.17 0.25 0.36D8 0.16 0.17 0.25 0.39D16 0.14 0.19 0.28 0.5D32 0.15 0.22 0.34 0.53D64 0.2 0.3 0.41 0.8D128 0.23 0.33 0.64 1.21D256 0.36 0.53 1.02 2D512 0.48 0.9 1.86 4.18D1024 0.82 1.7 3.51 7.31
Graph
document.doc Page 22 of 53
PVAN: Pivot, Analytic
Query TextSELECT /* PVAN*/'"' || emp_name || '","' || h || '","' || w || '","' || m || '","' || f || '363"' FROM (
SELECT emp.first_name || ' ' || emp.last_name emp_name,pho.phone_type,pho.phone_number,Row_Number() OVER (PARTITION BY emp.employee_id, pho.phone_type ORDER BY pho.valid_from
DESC, pho.phone_number) ind FROM employees emp LEFT JOIN phone_numbers pho ON pho.employee_id = emp.employee_id
) PIVOT (MAX(phone_number) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) WHERE ind = 1 ORDER BY 1
Query Diagram
Execution Plan Example (W128-D64)
Notice that the final cardinality estimate of 3,480,000 is a long way off the actual number, 13,696. The CBO has clearly not allowed for pruning (or pivoting) affecting the cardinalities at all.
----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |----------------------------------------------------------------------------------------------------
document.doc Page 23 of 53
| 0 | SELECT STATEMENT | | | | | 51794 (100)| || 1 | SORT ORDER BY | | 3480K| 169M| | 51794 (1)| 00:10:22 || 2 | HASH GROUP BY PIVOT | | 3480K| 169M| | 51794 (1)| 00:10:22 ||* 3 | VIEW | | 3480K| 169M| | 51510 (1)| 00:10:19 ||* 4 | WINDOW SORT PUSHED RANK| | 3480K| 179M| 226M| 51510 (1)| 00:10:19 ||* 5 | HASH JOIN OUTER | | 3480K| 179M| | 5348 (2)| 00:01:05 || 6 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |----------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
3 - filter("from$_subquery$_001"."IND"=1) 4 - filter(ROW_NUMBER() OVER ( PARTITION BY "EMP"."EMPLOYEE_ID","PHO"."PHONE_TYPE" ORDER BY INTERNAL_FUNCTION("PHO"."VALID_FROM") DESC ,"PHO"."PHONE_NUMBER")<=1) 5 - access("PHO"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.15 0.15 0.19 0.32 0.56 1.08 2.07 3.9D2 0.11 0.16 0.22 0.35 0.62 1.17 2.29 4.28D4 0.13 0.14 0.21 0.39 0.69 1.31 2.53 4.98D8 0.14 0.17 0.28 0.44 0.86 1.67 3.26 6.52D16 0.15 0.2 0.33 0.64 1.26 2.44 4.92 9.41D32 0.19 0.3 0.55 1.04 2.08 4.13 7.71 15.79D64 0.29 0.47 0.97 1.9 3.84 6.88 13.67 29.13
Graph
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.14 0.14 0.19 0.33D2 0.12 0.13 0.21 0.34D4 0.11 0.14 0.23 0.37D8 0.12 0.17 0.28 0.47D16 0.15 0.2 0.34 0.67D32 0.17 0.29 0.53 1D64 0.27 0.49 0.97 1.94D128 0.46 0.9 1.85 3.65D256 0.9 1.88 3.91 6.03D512 1.89 3.84 6.26 11.58
document.doc Page 24 of 53
D1024 3.88 6.55 11.65 22.13Graph
PVANIV: Pivot, Analytic, View
Query TextWITH wit AS (SELECT emp.first_name,
emp.last_name,ilv.phone_type,ilv.phone_number
FROM employees emp LEFT JOIN (
SELECT pho.employee_id,pho.phone_type,pho.phone_number,Row_Number() OVER (PARTITION BY pho.employee_id, pho.phone_type ORDER BY pho.valid_from
DESC,pho.phone_number) ind FROM phone_numbers pho
) ilv ON ilv.employee_id = emp.employee_id WHERE ilv.ind = 1)SELECT /* PVANIV*/'"' || first_name || ' ' || last_name || '","' || h || '","' || w || '","' || m || '","' || f || '3212"' FROM wit PIVOT (MAX(phone_number) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f))ORDER BY 1
Query Diagram
document.doc Page 25 of 53
Execution Plan Example (W128-D64)
Notice that the final cardinality estimate is exactly right, although how the CBO gets there from id 3 to id 2 below is hard to understand.
-----------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |-----------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 66739 (100)| || 1 | SORT ORDER BY | | 13696 | 775K| 240M| 66739 (1)| 00:13:21 || 2 | HASH GROUP BY PIVOT | | 13696 | 775K| 240M| 66739 (1)| 00:13:21 ||* 3 | HASH JOIN | | 3480K| 192M| | 32009 (1)| 00:06:25 || 4 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 ||* 5 | VIEW | | 3509K| 103M| | 31925 (1)| 00:06:24 ||* 6 | WINDOW SORT PUSHED RANK| | 3509K| 90M| 134M| 31925 (1)| 00:06:24 || 7 | TABLE ACCESS FULL | PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |-----------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
3 - access("ILV"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID") 5 - filter("ILV"."IND"=1) 6 - filter(ROW_NUMBER() OVER ( PARTITION BY "PHO"."EMPLOYEE_ID","PHO"."PHONE_TYPE" ORDER BY INTERNAL_FUNCTION("PHO"."VALID_FROM") DESC ,"PHO"."PHONE_NUMBER")<=1)
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.11 0.12 0.21 0.33 0.61 1.06 2.02 3.81D2 0.11 0.15 0.23 0.39 0.61 1.12 2.04 4.17D4 0.11 0.16 0.22 0.39 0.67 1.23 2.35 4.7D8 0.13 0.19 0.28 0.45 0.78 1.49 2.86 5.91D16 0.14 0.19 0.33 0.62 1.12 2.19 4.11 8.19D32 0.21 0.26 0.45 0.91 1.82 3.56 6.96 13.03D64 0.23 0.46 0.78 1.63 3.2 6.8 10.69 22.97
Graph
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.11 0.14 0.19 0.33D2 0.11 0.14 0.22 0.36D4 0.11 0.15 0.21 0.36D8 0.14 0.16 0.25 0.42D16 0.12 0.18 0.31 0.54
document.doc Page 26 of 53
D32 0.18 0.26 0.46 0.78D64 0.3 0.42 0.78 1.57D128 0.39 0.78 1.54 3.04D256 0.77 1.53 3.17 6.3D512 1.5 3.18 6.66 10.01D1024 3.32 6.81 10.02 18.25
Graph
PVKP: Pivot, Keep
Query TextSELECT /* PVKP*/'"' || emp_name || '","' || h || '","' || w || '","' || m || '","' || f || '4895"' FROM (
SELECT emp.first_name || ' ' || emp.last_name emp_name,pho.phone_type,MAX (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) phone_number_last
FROM employees emp LEFT JOIN phone_numbers pho ON pho.employee_id = emp.employee_id GROUP BY emp.first_name || ' ' || emp.last_name, pho.phone_type
) PIVOT (MAX(phone_number_last) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1
Query Diagram
document.doc Page 27 of 53
Execution Plan Example (W128-D64)
Notice how poor the final cardinality estimate of 2,132,000 is, when the actual number would be the number of employees (13,696). In this case, it was found later that changing the form of the Group By to separate out the two name fields caused the cardinality estimate to improve to be about ¾ the correct value at id 4, although the execution plan did not change (see Analysis section later).
------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 40121 (100)| || 1 | SORT ORDER BY | | 2132K| 77M| | 40121 (1)| 00:08:02 || 2 | HASH GROUP BY PIVOT | | 2132K| 77M| | 40121 (1)| 00:08:02 || 3 | VIEW | | 2132K| 77M| | 39952 (1)| 00:08:00 || 4 | SORT GROUP BY | | 2132K| 109M| 226M| 39952 (1)| 00:08:00 ||* 5 | HASH JOIN OUTER | | 3480K| 179M| | 5348 (2)| 00:01:05 || 6 | TABLE ACCESS FULL| EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
5 - access("PHO"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.14 0.17 0.2 0.31 0.58 0.98 2 3.84D2 0.14 0.15 0.22 0.32 0.61 1.04 2.14 4.16D4 0.12 0.14 0.22 0.35 0.63 1.17 2.4 4.74D8 0.09 0.15 0.24 0.4 0.7 1.44 2.83 5.66D16 0.14 0.17 0.3 0.5 1 1.87 3.8 7.84D32 0.14 0.21 0.41 0.71 1.34 2.8 5.76 12D64 0.2 0.33 0.54 1.11 2.24 4.77 9.56 20.19
Graph
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.12 0.14 0.19 0.38D2 0.13 0.12 0.2 0.32D4 0.12 0.14 0.25 0.38D8 0.12 0.15 0.22 0.39D16 0.14 0.19 0.3 0.49
document.doc Page 28 of 53
D32 0.14 0.24 0.37 0.68D64 0.2 0.32 0.56 1.09D128 0.28 0.49 0.92 1.88D256 0.47 0.81 1.64 3.44D512 0.77 1.58 3.12 6.85D1024 1.45 2.86 6.2 13.06
Graph
PVKPIV: Pivot, Keep, View
Query TextSELECT /* PVKPIV*/'"' || first_name || ' ' || last_name || '","' || h || '","' || w || '","' || m || '","' || f || '599"' FROM employees emp LEFT JOIN (
SELECT pho.employee_id,pho.phone_type,MAX (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) phone_number_last
FROM phone_numbers pho GROUP BY pho.employee_id, pho.phone_type
) ilv ON ilv.employee_id = emp.employee_id PIVOT (MAX(phone_number_last) FOR phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1
Query Diagram
document.doc Page 29 of 53
Execution Plan Example (W128-D64)
Notice that the final cardinality estimate is almost four times the actual number, while the intermediate estimates from id 7 up to id 3 are about right. It seems that the CBO has not allowed for the pivoting operation reducing cardinality.
------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 15867 (100)| || 1 | SORT ORDER BY | | 38419 | 1688K| 2128K| 15867 (2)| 00:03:11 || 2 | HASH GROUP BY PIVOT | | 38419 | 1688K| 2128K| 15867 (2)| 00:03:11 ||* 3 | HASH JOIN OUTER | | 38419 | 1688K| | 14988 (2)| 00:03:00 || 4 | TABLE ACCESS FULL | EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 || 5 | VIEW | | 38739 | 680K| | 14919 (2)| 00:03:00 || 6 | SORT GROUP BY | | 38739 | 1021K| 134M| 14919 (2)| 00:03:00 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 3509K| 90M| | 5265 (1)| 00:01:04 |------------------------------------------------------------------------------------------------ Predicate Information (identified by operation id): --------------------------------------------------- 3 - access("ILV"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.13 0.16 0.21 0.35 0.57 1.01 1.95 3.83D2 0.14 0.16 0.22 0.33 0.6 1.03 2.01 3.97D4 0.11 0.15 0.21 0.35 0.62 1.07 2.14 4.23D8 0.14 0.17 0.23 0.37 0.65 1.21 2.42 4.79D16 0.13 0.19 0.27 0.49 0.76 1.43 2.87 5.86D32 0.1 0.17 0.35 0.6 1.01 1.97 3.94 7.79D64 0.16 0.24 0.4 0.73 1.47 3.14 5.97 11.99
Graph
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.11 0.14 0.18 0.32D2 0.11 0.15 0.2 0.33D4 0.11 0.18 0.2 0.34D8 0.12 0.15 0.22 0.39D16 0.13 0.18 0.25 0.42D32 0.12 0.19 0.29 0.53
document.doc Page 30 of 53
D64 0.18 0.25 0.42 0.73D128 0.19 0.32 0.61 1.17D256 0.29 0.52 1.01 2.05D512 0.45 0.9 1.84 4.04D1024 0.83 1.69 3.45 7.26
Graph
FNSC: Database Function
Query Text - MainSELECT /* FNSC*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||
Query_Test_Set.Last_Phone_Number (emp.employee_id, 'HOME') || '","'Query_Test_Set.Last_Phone_Number (emp.employee_id, 'WORK') || '","' ||Query_Test_Set.Last_Phone_Number (emp.employee_id, 'MOBILE') || '","' ||Query_Test_Set.Last_Phone_Number (emp.employee_id, 'FAX') || '4324"'
FROM employees empORDER BY 1
Query Text - FunctionSELECT /* FUNCTIONSQ */
phone_number FROM (SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = p_phone_type AND pho.employee_id = p_employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1
document.doc Page 31 of 53
Query/Function Diagram
Execution Plan Example - Main---------------------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | ---------------------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | | 175 (100)| | | 1 | SORT ORDER BY | | 13583 | 358K| 544K| 175 (2)| 00:00:03 | | 2 | TABLE ACCESS FULL| EMPLOYEES | 13583 | 358K| | 68 (0)| 00:00:01 | ----------------------------------------------------------------------------------------
Execution Plan Example – Function
It was not possible to obtain the execution plan for the function from the v$ tables, for some reason, so we did an Explain Plan on the query with typical bind values.
---------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |---------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 12 (100)| ||* 1 | COUNT STOPKEY | | | | | || 2 | VIEW | | 2 | 12 | 12 (0)| 00:00:01 ||* 3 | TABLE ACCESS BY INDEX ROWID | PHONE_NUMBERS | 1029 | 27783 | 12 (0)| 00:00:01 |
document.doc Page 32 of 53
|* 4 | INDEX RANGE SCAN DESCENDING| PHONE_NUMBERS_N1 | 8 | | 3 (0)| 00:00:01 |--------------------------------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter(ROWNUM=1) 3 - filter("PHO"."PHONE_TYPE"=:B2) 4 - access("PHO"."EMPLOYEE_ID"=:B1)
Results
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128D1 0.35 0.56 0.87 1.49 2.85 5.6 12.02 21.98D2 0.36 0.64 0.83 1.53 2.9 5.52 11.87 22.13D4 0.41 0.89 0.86 1.61 2.82 5.61 11.57 22.75D8 0.46 0.45 0.9 1.51 2.87 5.77 11.7 22.68D16 0.28 0.44 0.9 1.56 2.9 5.78 11.45 22.9D32 0.28 0.45 0.87 1.52 3.03 5.72 11.44 23.59D64 0.31 0.5 0.78 1.53 3.04 5.88 11.72 28.61
Graph
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8D1 0.3 0.58 0.81 1.44D2 0.34 0.58 0.77 1.48D4 0.38 0.78 0.84 1.46D8 0.47 0.44 0.83 1.45D16 0.28 0.47 0.83 1.46D32 0.27 0.48 0.78 1.47D64 0.25 0.44 0.84 1.48D128 0.25 0.42 0.78 1.45D256 0.25 0.43 0.76 1.47D512 0.26 0.42 0.81 1.74D1024 0.28 0.43 0.78 2.11
Graph
document.doc Page 33 of 53
SSKP: Select Scalar Subquery, KeepThis query was added after analysis of the main results, and only run for a single data set point, W1-D1024.
Query TextSELECT /* SSKP*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||
(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'HOME' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'WORK' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'MOBILE' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX (phone_number) KEEP (DENSE_RANK LAST ORDER BY valid_from) FROM phone_numbers WHERE phone_type = 'FAX' AND employee_id = emp.employee_id) ||'133"'
FROM employees emp ORDER BY 1
document.doc Page 34 of 53
Query Diagram
Execution Plan Example (W1-D1024)
Notice that the cardinalities look about right here, and the plan is the same as for SSSM.------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 69 (100)| || 1 | SORT AGGREGATE | | 1 | 26 | | ||* 2 | TABLE ACCESS FULL | PHONE_NUMBERS | 1028 | 26728 | 619 (1)| 00:00:08 || 3 | SORT AGGREGATE | | 1 | 26 | | ||* 4 | TABLE ACCESS FULL | PHONE_NUMBERS | 1023 | 26598 | 619 (1)| 00:00:08 || 5 | SORT AGGREGATE | | 1 | 26 | | ||* 6 | TABLE ACCESS FULL | PHONE_NUMBERS | 1037 | 26962 | 619 (1)| 00:00:08 || 7 | SORT AGGREGATE | | 1 | 26 | | ||* 8 | TABLE ACCESS FULL| PHONE_NUMBERS | 1028 | 26728 | 619 (1)| 00:00:08 || 9 | SORT ORDER BY | | 107 | 2033 | 69 (2)| 00:00:01 || 10 | TABLE ACCESS FULL | EMPLOYEES | 107 | 2033 | 68 (0)| 00:00:01 |------------------------------------------------------------------------------------------
Results
The query ran in 44 seconds CPU time. Compare with FNSC, which took 0.3 seconds for this data set point.
SSSM: Select Scalar Subquery, MaxThis query was added after analysis of the main results, and only run for a single data set point, W1-D1024.
Query TextSELECT /* SSKP*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||
(SELECT Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers WHERE phone_type = 'HOME' AND employee_id = emp.employee_id) || '","' ||(SELECT Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers WHERE phone_type = 'WORK' AND employee_id = emp.employee_id) || '","' ||(SELECT MAX Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers WHERE phone_type = 'MOBILE' AND employee_id = emp.employee_id) || '","' ||(SELECT Substr (Max (To_Char (valid_from,'j') || phone_number), 8) FROM phone_numbers
document.doc Page 35 of 53
WHERE phone_type = 'FAX' AND employee_id = emp.employee_id) ||'133"'
FROM employees emp ORDER BY 1
Query Diagram
Execution Plan Example (W1-D1024)
Notice that the cardinalities look about right here, and the plan is the same as for SSKP.------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 69 (100)| || 1 | SORT AGGREGATE | | 1 | 25 | | ||* 2 | TABLE ACCESS FULL | PHONE_NUMBERS | 1028 | 25700 | 619 (1)| 00:00:08 || 3 | SORT AGGREGATE | | 1 | 25 | | ||* 4 | TABLE ACCESS FULL | PHONE_NUMBERS | 1023 | 25575 | 619 (1)| 00:00:08 || 5 | SORT AGGREGATE | | 1 | 25 | | ||* 6 | TABLE ACCESS FULL | PHONE_NUMBERS | 1037 | 25925 | 619 (1)| 00:00:08 || 7 | SORT AGGREGATE | | 1 | 25 | | ||* 8 | TABLE ACCESS FULL| PHONE_NUMBERS | 1028 | 25700 | 619 (1)| 00:00:08 || 9 | SORT ORDER BY | | 107 | 2033 | 69 (2)| 00:00:01 || 10 | TABLE ACCESS FULL | EMPLOYEES | 107 | 2033 | 68 (0)| 00:00:01 |------------------------------------------------------------------------------------------
Results
The query ran in 46 seconds CPU time. Compare with FNSC, which took 0.3 seconds for this data set point (but 0.7s in the last run in Test_Phone_1-11.LST).
SSOB: Select Scalar Subquery, Order ByThis query was added after analysis of the main results, and is not valid in Oracle SQL due to a technical limitation whereby an alias can only be referenced at one level down within a scalar subquery. It is of interest here, because had it been valid it would probably be the most efficient method for narrow deep cases, from consideration of FNSC.
Query TextSELECT /* SSOB*/ '"' || emp.first_name || ' ' || emp.last_name || '","' ||
(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'HOME' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1
document.doc Page 36 of 53
) || '","' ||(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'WORK' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1) || '","' ||(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'MOBILE' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1) || '","' ||(SELECT pho.phone_number FROM(SELECT pho.phone_number FROM phone_numbers pho WHERE pho.phone_type = 'FAX' AND pho.employee_id = emp.employee_id ORDER BY pho.valid_from DESC) WHERE ROWNUM = 1) ||'999"'
FROM employees emp ORDER BY 1
Query Diagram
Results
The query fails with error:
ORA-00904: "EMP"."EMPLOYEE_ID": invalid identifier
document.doc Page 37 of 53
Analysis of Results
Slice AnalysisWe would like to like to compare the performances of the queries graphically, but as it would be difficult to do this using the 3-dimensional graphs above, we will instead consider the four slices obtained by fixing one of the width and depth dimensions at its minimum and maximum with the other displayed across its range. For each slice and data set the CPU times are tabulated across all queries and a line graph is given, then the relative performances are assessed in an analysis subsection.
Narrow Slice
Wide Data SetTable of CPU Times
Query D1 D2 D4 D8 D16 D32 D64FNSC 0.35 0.36 0.41 0.46 0.28 0.28 0.31PVAN 0.15 0.11 0.13 0.14 0.15 0.19 0.29PVANIV 0.11 0.11 0.11 0.13 0.14 0.21 0.23PVKP 0.14 0.14 0.12 0.09 0.14 0.14 0.2PVKPIV 0.13 0.14 0.11 0.14 0.13 0.1 0.16WJAN 0.14 0.14 0.14 0.14 0.16 0.17 0.34WJKP 0.15 0.15 0.14 0.14 0.15 0.15 0.2WJSQ 0.13 0.14 0.15 0.16 0.16 0.22 0.3JNSQ 1.35 1.29 1.51 1.89 2.95 5.98 14.59
Graph of CPU Times, excluding JNSQ
Deep Data SetTable of CPU Times
Query D1 D2 D4 D8 D16 D32 D64 D128 D256 D512 D1024FNSC 0.3 0.34 0.38 0.47 0.28 0.27 0.25 0.25 0.25 0.26 0.28PVAN 0.14 0.12 0.11 0.12 0.15 0.17 0.27 0.46 0.9 1.89 3.88PVANIV 0.11 0.11 0.11 0.14 0.12 0.18 0.3 0.39 0.77 1.5 3.32PVKP 0.12 0.13 0.12 0.12 0.14 0.14 0.2 0.28 0.47 0.77 1.45PVKPIV 0.11 0.11 0.11 0.12 0.13 0.12 0.18 0.19 0.29 0.45 0.83WJAN 0.15 0.14 0.14 0.14 0.17 0.18 0.24 0.42 0.8 1.56 3.36WJKP 0.14 0.14 0.14 0.16 0.14 0.15 0.2 0.23 0.36 0.48 0.82WJSQ 0.19 0.19 0.14 0.15 0.17 0.22 0.28 0.44 0.62 1.11 1.43
Graph of CPU Times
document.doc Page 38 of 53
AnalysisWide Data Set
This is a subset of the deep data set for the narrow slice (other than having wider date range), but was the only set that JNSQ was tested on (to save time).
JNSQ took 91 x best time at the maximum depth
Deep Data Set
FNSC is worst up to depth of 32 but best from 256, taking 34% of the next best time at maximum depth
FNSC shows essentially constant time; this is presumably because the main query scans the unchanging employees table, while the function query uses the index and is coded to stop after the first row and so may be able to obtain the rowid in the same small number of reads.
PVPKIV and WJKP show similar performance as next best at 2.9 x best
PVKP and WJSQ show similar performance as next best at 5.1 x best
PVANIV AND WJAN show similar performance as next best at 11.9 x best, with PVAN just behind at 13.9
Wide Slice
Wide Data Set – W128Table of CPU Times
Query D1 D2 D4 D8 D16 D32 D64FNSC 21.98 22.13 22.75 22.68 22.9 23.59 28.61PVAN 3.9 4.28 4.98 6.52 9.41 15.79 29.13PVANIV 3.81 4.17 4.7 5.91 8.19 13.03 22.97PVKP 3.84 4.16 4.74 5.66 7.84 12 20.19PVKPIV 3.83 3.97 4.23 4.79 5.86 7.79 11.99WJAN 4.31 4.39 4.97 6.19 8.72 12.49 21.31WJKP 4.16 4.04 4.4 4.93 6.07 8.08 12.29WJSQ 4.44 4.62 5.05 6.14 8.7 13.22 22.62JNSQ 13.53 34.59 67.66 298.62 324.23
Graph of CPU Times, excluding JNSQ
document.doc Page 39 of 53
Deep Data Set – W8Table of CPU Times
Query D1 D2 D4 D8 D16 D32 D64 D128 D256 D512 D1024FNSC 1.44 1.48 1.46 1.45 1.46 1.47 1.48 1.45 1.47 1.74 2.11PVAN 0.33 0.34 0.37 0.47 0.67 1 1.94 3.65 6.03 11.58 22.13PVANIV 0.33 0.36 0.36 0.42 0.54 0.78 1.57 3.04 6.3 10.01 18.25PVKP 0.38 0.32 0.38 0.39 0.49 0.68 1.09 1.88 3.44 6.85 13.06PVKPIV 0.32 0.33 0.34 0.39 0.42 0.53 0.73 1.17 2.05 4.04 7.26WJAN 0.32 0.39 0.42 0.44 0.61 0.89 1.58 3.09 6.44 9.89 19.2WJKP 0.38 0.36 0.36 0.39 0.5 0.53 0.8 1.21 2 4.18 7.31WJSQ 0.38 0.41 0.42 0.48 0.61 0.88 1.42 2.5 4.59 9.54 12.59
Graph of CPU Times
AnalysisWide Data Set
This is a subset of the deep data set for the wide slice (other than having wider date range), but was the only set that JNSQ was tested on (to save time).
JNSQ took 55 x best time at the maximum depth it was run at (D16)
Deep Data Set
FNSC is worst up to depth of 32 but best from 256, taking 29% of the next best time at maximum depth
document.doc Page 40 of 53
FNSC shows essentially constant time for the first 9 depths; this is presumably because the main query scans the unchanging employees table, while the function query uses the index and is coded to stop after the first row and so may be able to obtain the rowid in the same small number of reads. The last two depths show increases of 18% and a further 21%; possibly the greater sizes of the index have increased the index depth and hence the numbers of blocks read, but I have not verified this.
PVPKIV and WJKP show similar performance as next best at 3.5 x best at maximum depth
PVKP and WJSQ show similar performance as next best at 6.2 x best
PVANIV AND WJAN show similar performance as next best at 8.9 x best, with PVAN just behind at 10.5
Apart from FNSC, the relative performances remain largely similar after divergence appears from about D8
Shallow Slice
Wide Data SetTable of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128FNSC 0.35 0.56 0.87 1.49 2.85 5.6 12.02 21.98PVAN 0.15 0.15 0.19 0.32 0.56 1.08 2.07 3.9PVANIV 0.11 0.12 0.21 0.33 0.61 1.06 2.02 3.81PVKP 0.14 0.17 0.2 0.31 0.58 0.98 2 3.84PVKPIV 0.13 0.16 0.21 0.35 0.57 1.01 1.95 3.83WJAN 0.14 0.14 0.24 0.34 0.64 1.06 2.11 4.31WJKP 0.15 0.18 0.23 0.36 0.62 1.09 2.07 4.16WJSQ 0.13 0.22 0.24 0.37 0.67 1.17 2.16 4.44JNSQ 1.35 1.73 2 2.15 2.34 4.01 7.3 13.53
Graph of CPU Times
Deep Data SetTable of CPU Times
Query W1 W2 W4 W8FNSC 0.3 0.58 0.81 1.44PVAN 0.14 0.14 0.19 0.33PVANIV 0.11 0.14 0.19 0.33PVKP 0.12 0.14 0.19 0.38PVKPIV 0.11 0.14 0.18 0.32WJAN 0.15 0.2 0.27 0.32WJKP 0.14 0.14 0.21 0.38
document.doc Page 41 of 53
WJSQ 0.19 0.21 0.23 0.38Graph of CPU Times
AnalysisWide Data Set
JNSQ is worst up to depth of 8, then second worst to FNSC, with JNSQ and FNSC taking 3.6 and 5.8 times the best time at maximum width
For the rest, performances divide into two groups:
o PVAN, PVANIV, PVKP and PVPKIV are best, taking about 90% of the times of the next group at maximum width
o WJAN, WJKP and WJSQ take about 1.1 x the times of the best group
Deep Data Set
This is a subset of the wide data set for the shallow slice.
Deep Slice
Wide Data Set – D64Table of CPU Times
Query W1 W2 W4 W8 W16 W32 W64 W128FNSC 0.31 0.5 0.78 1.53 3.04 5.88 11.72 28.61PVAN 0.29 0.47 0.97 1.9 3.84 6.88 13.67 29.13PVANIV 0.23 0.46 0.78 1.63 3.2 6.8 10.69 22.97PVKP 0.2 0.33 0.54 1.11 2.24 4.77 9.56 20.19PVKPIV 0.16 0.24 0.4 0.73 1.47 3.14 5.97 11.99WJAN 0.34 0.45 0.85 1.65 3.41 6.71 10.42 21.31WJKP 0.2 0.3 0.48 0.78 1.53 3.04 6.05 12.29WJSQ 0.3 0.42 0.79 1.42 2.73 5.7 11.6 22.62JNSQ 14.59 59.48 159.65 292.08
Graph of CPU Times, excluding JNSQ
document.doc Page 42 of 53
Deep Data Set – D1024Table of CPU Times
Query W1 W2 W4 W8FNSC 0.28 0.43 0.78 2.11PVAN 3.88 6.55 11.65 22.13PVANIV 3.32 6.81 10.02 18.25PVKP 1.45 2.86 6.2 13.06PVKPIV 0.83 1.69 3.45 7.26WJAN 3.36 6.94 9.86 19.2WJKP 0.82 1.7 3.51 7.31WJSQ 1.43 2.76 5.74 12.59
Graph of CPU Times
AnalysisWide Data Set
JNSQ is worst in all cases where it was run, having 96 times the best time at the maximum width it was run at (W32)
For the rest, performances divide into three groups:
o PVPKIV and WJKP are best, taking about 57% of the times of the next group at maximum width
o PVANIV, PVKP, WJAN, and WJSQ take about 1.8 x the times of the best group
document.doc Page 43 of 53
o FNSC and PVAN take about 2.4 x the times of the best group
Deep Data Set
This is a subset of the wide data set for the shallow slice.
Summary AnalysisIn this section, the relative rankings and performance factors are tabulated across the query sets and slices for both data sets.
Tables and Graphs of Relative PerformanceTable of Rankings
DataSet> Wide DeepSlice -> Shallow Deep Narrow WideEnd -> Narrow Wide Narrow Wide Shallow Deep Shallow Deep
FNSC 8 9 6 7 8 1 8 1PVAN 6 1 5 7 4 8 1 8PVANIV 1 1 4 5 1 6 1 6PVKP 4 1 2 3 3 4 5 4PVKPIV 2 1 1 1 1 2 1 2WJAN 4 5 8 4 6 6 1 6WJKP 6 5 2 1 4 2 5 2WJSQ 2 6 6 5 7 4 5 4JNSQ 9 8 9 9 NA NA NA NA
Table of CPU Time Factors Relative to BestDataSet> Wide DeepSlice -> Shallow Deep Narrow WideEnd -> Narrow Wide Narrow Wide Shallow Deep Shallow Deep
FNSC 3.2 5.8 1.9 2.4 2.7 1 4.5 1PVAN 1.4 1 1.8 2.4 1.3 13.9 1 10.5PVANIV 1 1 1.4 1.9 1 11.9 1 8.6PVKP 1.3 1 1.3 1.7 1.1 5.2 1.2 6.2PVKPIV 1.2 1 1 1 1 3 1 3.4WJAN 1.3 1.1 2.1 1.8 1.4 12 1 9.1WJKP 1.4 1.1 1.3 1 1.3 2.9 1.2 3.5WJSQ 1.2 1.2 1.9 1.9 1.7 5.1 1.2 6JNSQ 12.3 3.6 91.2 0 NA NA NA NA
Graph of CPU Time Factors Relative to Best – Start (JNSQ excluded)
Graph of CPU Time Factors Relative to Best – End (JNSQ excluded)
document.doc Page 44 of 53
Which Query is Best?
PVKPIV is best on 5 of the 8 data set points considered in the summary analysis above, and is significantly worse than only one other query, FNSC, on the two deepest data set points, where it is about equal second with WJKP. A couple of further points may be noted:
It is not surprising that the query performing best overall (PVKPIV) uses Oracle’s native syntax for both pivoting and pruning, being better in particular than those pruning by use of the more general technique of analytic functions. It is very surprising though that the inline view modifier is needed to obtain the best performance, as is the result that using a database function outperforms all others in some scenarios. These points are discussed in the next section.
WJKP shows a very similar performance profile to PVKPIV, but slightly worse; it seems that the execution plans, although superficially different, may amount to much the same but with the overhead of writing to the temporary table likely explaining the differences.
Cost Based OptimizerThe results throw light on the performance of the CBO. In this section we consider what they imply for query construction in general, as well as for our test scenarios. It should be noted that our results are for a single class of queries, and it may be that the CBO happens to be particularly well or badly suited to that class compared with other classes.
CBO Cardinalities
In order for the CBO to compare candidate execution plans effectively, it is of course very important that its cardinality estimates be reasonably good. However, it can be seen from the sections on query execution plans above that the accuracy of the cardinality estimates is frequently very poor: It seems, in some cases, to have trouble with both the pivoting and pruning operations. For example, with PVKPIV it gets the pruning side right, but not the pivoting. It is worth noting that the use of the inline view seems to help it get its estimates right. For example, with PVAN, it does not reduce cardinalities at all for either operation, while with PVANIV it does much better.
The PVKP query revealed an interesting CBO problem with the Group By clause. In the query as shown, the Group By for the main view merely copies the grouping fields from the select list, as is usual:
GROUP BY emp.first_name || ' ' || emp.last_name, pho.phone_type
By experiment, I found that replacing the employee names expression with its components led to the CBO obtaining an accurate estimate.
GROUP BY emp.first_name, emp.last_name, pho.phone_type
Note that the two forms are equivalent, as Oracle clearly knows at the syntax-checking level, or it would raise the error: ‘ORA-00979: not a GROUP BY expression’
The two forms were tested on the W1-D1024 data set point, along with PVKPIV (see Test_Phone_1-11_3.LST for full listing), and the two explain plans were:
document.doc Page 45 of 53
Original------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 2573 (100)| || 1 | SORT ORDER BY | | 18565 | 688K| | 2573 (2)| 00:00:31 || 2 | HASH GROUP BY PIVOT | | 18565 | 688K| | 2573 (2)| 00:00:31 || 3 | VIEW | | 18565 | 688K| | 2570 (1)| 00:00:31 || 4 | SORT GROUP BY | | 18565 | 815K| 24M| 2570 (1)| 00:00:31 ||* 5 | HASH JOIN OUTER | | 433K| 18M| | 690 (2)| 00:00:09 || 6 | TABLE ACCESS FULL| EMPLOYEES | 107 | 2033 | | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 433K| 10M| | 619 (1)| 00:00:08 |------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
5 - access("PHO"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")
Modified----------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |----------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | 708 (100)| || 1 | SORT ORDER BY | | 303 | 11514 | 708 (4)| 00:00:09 || 2 | HASH GROUP BY PIVOT | | 303 | 11514 | 708 (4)| 00:00:09 || 3 | VIEW | | 303 | 11514 | 706 (4)| 00:00:09 || 4 | SORT GROUP BY | | 303 | 13635 | 706 (4)| 00:00:09 ||* 5 | HASH JOIN OUTER | | 433K| 18M| 690 (2)| 00:00:09 || 6 | TABLE ACCESS FULL| EMPLOYEES | 107 | 2033 | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 433K| 10M| 619 (1)| 00:00:08 |----------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
5 - access("PHO"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")
The timings were:
PVKP (original) 1.43
PVKPG (modified) 1.31
PVKPIV .84
Although the plans appear the same, the modified query appeared to be consistently slightly faster (Is the plan output hiding some differences ‘under the covers’? Incidentally, if the reader has not yet read Under the Net, the first, maybe best, book by the late English writer Iris Murdoch, it’s highly recommended). The change thus made little difference here, but in another query it might.
Inline View Modifier
Using an inline view to enclose the pruning operation is always at least as good as not, and up to 21% faster for the analytics strategy, and 59% faster for the keep strategy. The view guides the CBO to reduce the number of records being processed before joining them to the master records. This is extremely surprising because one would expect the CBO to work this out for itself, especially on our deliberately simple queries.
Subquery Factoring Modifier
Applying the subquery factor modifier proved extremely effective for the join subquery strategy, both in performance and in allowing outer joining.
It is interesting to note from the detailed timings in the output files (SQL Pivot and Prune Queries – Output) that, while Oracle normally defers execution of a query until the first fetch shows it’s needed, in all cases with a subquery factor execution occurs straight away on cursor opening.
Performance of JNSQ
One of the benefits of testing the queries across a 2 dimensional domain is that one can compare performances across data set points, and one sees immediately that the CBO has performed very poorly on some of these for JNSQ. In one case, doubling the number of master records reduced the CPU time by a factor of 7!
document.doc Page 46 of 53
Performance of Database Function and Scalar Subquery Strategies
Using database functions within queries, while quite common, is often regarded as bad practice both for performance and theoretical reasons, and usually correctly. The supposed modularity gains can generally be better obtained through other means, including high level design (can complex SQL be placed in a (possibly transactional) API rather than duplicated?) and, sometimes, views.
In our case, however, we found that the best performance obtained on narrow, deep cases was through a database function. This was because only by this means was the relevant index used to exclude almost all the detail records straight away; generally, the other strategies continued to favour full table scans. We tried to replicate this performance advantage through scalar subqueries, but the ones that were valid again neglected the obvious indexing strategy. This seems to be a case of CBO imperfection – the queries are very simple, bypassing any question of combinatorial complexity, the data distribution is random, not skewed, and statistics were gathered. It ought to have been possible for the CBO to obtain the best strategy by itself from the native syntax query (PVKP) telling it what was required, not how to do it. This type of query may be an area where the CBO is underdeveloped.
General Conclusions
Oracle’s concepts manual (REF-2) says:
In contrast to procedural languages such as C, which describe how things should be done, SQL is nonprocedural and describes what should be done
This is true for basic classes of SQL, but is not always true for more complex classes. In many cases it is necessary to use inline views together with analytic, and other, functions to build a structured query in essentially a procedural manner, in order to obtain a desired result. The ability to combine non-procedural components in this way to build very powerful queries is of course a big strength of SQL (and is why I place a high value on diagrammatic design techniques for SQL, here and in A Structured Approach to SQL Query Design).
In our relatively simple queries, we have found that using modifiers such as inline views and subquery factoring to tell the CBO how (at a high level) to process the queries, while not strictly necessary, is often beneficial to performance. This perhaps surprising finding shows that keeping an eye on performance is still important for queries in Oracle 11g.
document.doc Page 47 of 53
Query Testing ProgramThis section gives design information about the testing program. The modular design allows the generic package (Test Queries in the diagram below) and the specification of the Query Test Set package to be used for any set of queries provided in the body of Query Test Set. See REF-3 for the Oracle packages referenced.
Call Structure Diagram
document.doc Page 48 of 53
Data Flow Diagram
Table Structures
Generic Tables
RUN_CONTROL
Column Name Type Notesid* Number Sequence generated primary keydescription Char(500) Description of runstatus Char(1) S – success, F - failuremessage Char(4000) Error message if anypoint_wide_max Number Maximum width pointpoint_deep_max Number Maximum depth point
cpu_time Number Total CPU timeelapsed_time Number Total elapsed timecreation_date Date Creation date
RUN_STATISTICS
Column Name Type Notesrun_control_id* Number Foreign key to RUN_CONTROL tablerun_type* Char(60) Query codepoint_wide* Number Width pointpoint_deep* Number Depth pointcpu_time Number CPU timeelapsed_time Number Elapsed timecreation_date Date Creation datestatus Char(1) S – success, F - failure
document.doc Page 49 of 53
message Char(4000) Error message if any
OUTPUT_LOG
Column Name Type Notesline_ind Integer Line numberline_text Char(4000) Line textid Char(30) Identifier codecreation_date Date Creation date
Specific Tables for our HR Queries
EMPLOYEESThis is the Oracle HR demo table, with 107 seeded records that we copy with name changes.
Column Name Type Notesemployee_id* Number Sequence generated primary keyfirst_name Char(20) First name – copy number appended when copying last_name Char(25) Last nameemail Char(20) Email address – copy number appended when copying for
uniqueness, but not referenced in queriesphone_number Number
Populated as seeded but not referenced in queries
hire_date Datejob_id Numbersalary Numbercommission_pct Numbermanager_id Numberdepartment_id Number
Index EMP_EMP_ID_PK
employee_id
Indexes (various, not used, see Oracle’s HR demo schema)
PHONE_NUMBERSThis is a new table in the HR schema that stores phone numbers by employee and type in records, as in Oracle Applications modules, including the CRM modules.
Column Name Type Notesid* Number Sequence generated primary keyemployee_id Number Foreign key to EMPLOYEESphone_type Char(10) Phone number type (HOME, WORK, MOBILE, FAX)valid_from Date Valid from date (set randomly within a range)valid_to Date Valid to date (not set)phone_number Char(9) Phone number (set randomly within a range)
Index PHONE_NUMBERS_N1
employee_id
valid_from
Index PHONE_NUMBERS_N2
employee_id
To_Char (valid_from, 'J')
The idea here is to ensure that the queries have reasonable indexes to use if the CBO deems it advisable, but to avoid including all fields in an index, for the purpose of this testing. The function-based index is included to provide a usable index for the subquery strategies.
Program Logic Write header record (RUN_CONTROL table)
Get data set dimensions from Query_Test_Set package
document.doc Page 50 of 53
Loop over width points
o Loop over depth points
Setup test data, and gather CBO statistics, using Query_Test_Set package
Call interface
Open output file
Get query string, using Query_Test_Set package
Loop over cursor for query
o Write line to file
End loop
Write execution plan to log, using Utils.Write_Plan (OUTPUT_LOG table)
Write statistics record (RUN_STATISTICS table)
Close output file
End Call interface
o End loop
End loop
Update header record (RUN_CONTROL table)
Example OutputThe output is included in SQL Pivot and Prune Queries – Output. Here is an extract, with indentation removed, and the first 100 characters shown thereafter per line (the version of code included also writes the number of lines written, but that was added after the example below was run):
Setup data: 7-1024 Phone numbers truncated3424 numbers added, (iteration 1 of 1024)...3424 numbers added, (iteration 1024 of 1024)3506176 numbers added in total4096 numbers deleted for emp 1211024 numbers deleted for emp 196, type FAX79014 duplicate dates527835 duplicate numbers856 total employees3501056 total numbersTimings Elapsed (CPU)Delete/Add employees 0.00 (0.00)Add numbers 5,166.15 (637.35)Gather stats 55.51 (20.32)Count duplicate dates 20.94 (12.53)Count duplicate numbers 9.94 (6.27)Count totals 4.46 (0.98)Total times: 5,257.00 (677.45)(Other): 0.00 (0.00)PVKPSELECT /* PVKP*/ '"' || emp_name || '","' || h || '","' || w || '","' || m || '","' || f || '8943"'ame emp_name, pho.phone_type, Max (pho.phone_number) KEEP (DENSE_RANK LAST ORDER BY pho.valid_from) ne_numbers pho ON pho.employee_id = emp.employee_id GROUP BY emp.first_name || ' ' || emp.last_name,R phone_type IN ('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1SQL_ID 9xt2by4jcdg1x, child number 0-------------------------------------SELECT /* PVKP*/ '"' || emp_name || '","' || h || '","' || w || '","'|| m || '","' || f || '8943"' FROM ( SELECT emp.first_name || ' ' ||emp.last_name emp_name, pho.phone_type, Max (pho.phone_number) KEEP(DENSE_RANK LAST ORDER BY pho.valid_from) phone_number_last FROMemployees emp LEFT JOIN phone_numbers pho ON pho.employee_id =emp.employee_id GROUP BY emp.first_name || ' ' || emp.last_name,pho.phone_type ) PIVOT (Max(phone_number_last) FOR phone_type IN('HOME' AS h, 'WORK' AS w, 'MOBILE' AS m, 'FAX' AS f)) ORDER BY 1
Plan hash value: 3462645326
------------------------------------------------------------------------------------------------| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time |
document.doc Page 51 of 53
------------------------------------------------------------------------------------------------| 0 | SELECT STATEMENT | | | | | 22768 (100)| || 1 | SORT ORDER BY | | 148K| 5511K| | 22768 (1)| 00:04:34 || 2 | HASH GROUP BY PIVOT | | 148K| 5511K| | 22768 (1)| 00:04:34 || 3 | VIEW | | 148K| 5511K| | 22757 (1)| 00:04:34 || 4 | SORT GROUP BY | | 148K| 7686K| 227M| 22757 (1)| 00:04:34 ||* 5 | HASH JOIN OUTER | | 3495K| 176M| | 5348 (2)| 00:01:05 || 6 | TABLE ACCESS FULL| EMPLOYEES | 856 | 22256 | | 68 (0)| 00:00:01 || 7 | TABLE ACCESS FULL| PHONE_NUMBERS | 3495K| 90M| | 5265 (1)| 00:01:04 |------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):---------------------------------------------------
5 - access("PHO"."EMPLOYEE_ID"="EMP"."EMPLOYEE_ID")
Timings Elapsed (CPU)Open cursor 0.04 (0.03)First fetch 13.34 (12.78)Remaining fetches 0.10 (0.04)Write to file 0.08 (0.08)Write plan 0.12 (0.05)Total times: 13.71 (13.06)(Other): 0.03 (0.08)Summary for 8 * 107 = 856 employees with 1024 * 4 = 4096 numbers per employeeTimings Elapsed (CPU)Run_One - Total 13.72 (13.06)Total times: 13.72 (13.06)(Other): 0.00 (0.00)PVAN...
document.doc Page 52 of 53
ReferencesREF Document DetailsREF-1 Oracle® Database SQL Language Reference 11g Release
2 (11.2)http://www.oracle.com/pls/db112REF-2 Oracle® Database Concepts 11g Release 2 (11.2)
REF-3 Oracle® Database PL/SQL Packages and Types Reference 11g Release 2 (11.2)
REF-4 A Structured Approach to SQL Query Design BP Furey, May 2009REF-5 Code Timing and Object Orientation and Zombies BP Furey, November 2010REF-6 SQL Pivot and Prune Queries – Output BP Furey, May 2011
REF-7 Tom Kyte’s Oracle database forum I use it generally, and got the DBMS_XPlan reference there in particular
REF-8 Under the Net Iris Murdoch, 1954REF-9 Forming Range-Based Break Groups with Advanced SQL BP Furey, June 2011
document.doc Page 53 of 53