1

Oracle Database Advanced Querying

Zohar Elkayam CTO, Brillix

Zohar@Brillix.co.ilwww.realdbamagic.com

Twitter: @realmgic

2

Who am I?• Zohar Elkayam, CTO at Brillix

• Programmer, DBA, team leader, database trainer, public speaker, and a senior consultant for over18 years

• Oracle ACE Associate

• Part of ilOUG – Israel Oracle User Group

• Blogger – www.realdbamagic.com and www.ilDBA.co.il

3

About Brillix• We offer complete, integrated end-to-end solutions based on

best-of-breed innovations in database, security and big data technologies

• We provide complete end-to-end 24x7 expert remote database services

• We offer professional customized on-site trainings, delivered by our top-notch world recognized instructors

4

Some of Our Customers

5

Agenda• Aggregative and advanced grouping options

• Analytic functions, ranking and pagination

• Hierarchical and recursive queries

• Regular Expressions

• Oracle 12c new rows pattern matching

• XML and JSON handling with SQL

• Oracle 12c (12.1 + 12.2) new features

• SQL Developer Command Line tool (if time allows)

6

Our Goal Today• Learning new SQL techniques

• We will not expert everything

• Getting to know new features (12cR1 and 12cR2)

• This is a starting point – don’t be afraid to try

7

The REAL Agenda

יום הסמינר יישלח אליכם טופס משוב בהודעת בסיום •

SMS ,נשמח לקבל את חוות דעתכם .

!טאבלטבין ממלאי המשוב יוגרל מידי יום

הפסקה10:30-10:45

י הכנס בגן המלוןארוחת צהריים לכל משתתפ12:30-13:30

הפסקה מתוקה במתחם קבלת הפנים15:00-15:15

MySQL User Group Meetup-או ל)הולכים הביתה16:30(שיערך מיד אחרי הכנס כאן במלון

8

יכולות מתקדמות–Oracle SQLאודות

יכולות –Oracle SQL"הספר •

מדריך לשולף , מתקדמות

2011פורסם בשנת " המהיר

והיחידהראשון SQL-זה ספר ה•

שנכתב בעברית מתחילתו ועד

סופו

עמיאל דיוויס הספר נכתב על ידי •ועבר עריכה טכנית שלי

9

SQL טכניקות ויישומים פרקטיים–מתקדם

טכניקות –מתקדם SQL"הספר •

הוא ספר חדש " ויישומים פרקטיים

רם קדםשפורסם השנה על ידי

SQLבעיות 100-הספר מכיל כ•מורכבות ופתרונן

קיים גם בגרסת אונליין•

/http://ramkedem.com: לפרטים•

10

ANSI SQL• SQL was invented in 1970 by Dr. E. F. Codd

• Each vendor had its own flavor of SQL

• Standardized by ASNI since 1986

• Current stable standard is ANSI SQL:2011/2008

• Oracle 11g is compliant to SQL:2008

• Oracle 12c is fully compliant to CORE SQL:2011

11

Queries• In this seminar we will only talk about queries

Group Functions

More than just group by…

13

Group Function and SQL• Using SQL for aggregation:

– Group functions basics

– The CUBE and ROLLUP extensions to the GROUP BY clause

– The GROUPING functions

– The GROUPING SETS expression

• Working with composite columns

• Using concatenated groupings

14

Basics• Group functions will return a single row for each

group

• The group by clause groups rows together and allows group functions to be applied

• Common group functions: SUM, MIN, MAX, AVG, etc.

15

Group Functions Syntax

SELECT [column,] group_function(column). . .FROM table[WHERE condition][GROUP BY group_by_expression][ORDER BY column];

SELECT AVG(salary), STDDEV(salary),COUNT(commission_pct),MAX(hire_date)

FROM hr.employeesWHERE job_id LIKE 'SA%';

16

SELECT department_id, job_id, SUM(salary), COUNT(employee_id)

FROM hr.employeesGROUP BY department_id, job_idOrder by department_id;

The GROUP BY Clause

SELECT [column,] group_function(column)FROM table[WHERE condition][GROUP BY group_by_expression][ORDER BY column];

17

The HAVING Clause• Use the HAVING clause to specify which groups

are to be displayed

• You further restrict the groups on the basis of a limiting condition

SELECT [column,] group_function(column)... FROM table[WHERE condition][GROUP BY group_by_expression][HAVING having_expression] [ORDER BY column];

18

GROUP BY Using ROLLUP and CUBE

• Use ROLLUP or CUBE with GROUP BY to produce superaggregate rows by cross-referencing columns

• ROLLUP grouping produces a result set containing the regular grouped rows and the subtotal and grand total values

• CUBE grouping produces a result set containing the rows from ROLLUP and cross-tabulation rows

19

Using the ROLLUP Operator

• ROLLUP is an extension of the GROUP BY clause

• Use the ROLLUP operation to produce cumulative aggregates, such as subtotals

SELECT [column,] group_function(column). . .FROM table[WHERE condition][GROUP BY [ROLLUP] group_by_expression][HAVING having_expression];[ORDER BY column];

20

Using the ROLLUP Operator: Example

SELECT department_id, job_id, SUM(salary)FROM hr.employeesWHERE department_id < 60GROUP BY ROLLUP(department_id, job_id);

1

2

3

Total by DEPARTMENT_IDand JOB_ID

Total by DEPARTMENT_ID

Grand total

21

Using the CUBE Operator• CUBE is an extension of the GROUP BY clause

• You can use the CUBE operator to produce cross-tabulation values with a single SELECT statement

SELECT [column,] group_function(column)...FROM table[WHERE condition][GROUP BY [CUBE] group_by_expression][HAVING having_expression][ORDER BY column];

22

SELECT department_id, job_id, SUM(salary)FROM hr.employeesWHERE department_id < 60GROUP BY CUBE (department_id, job_id);

. . .

Using the CUBE Operator: Example

. . .

1

2

3

4

Grand total

Total by JOB_ID

Total by DEPARTMENT_IDand JOB_ID

Total by DEPARTMENT_ID

23

SELECT [column,] group_function(column) .. ,GROUPING(expr)

FROM table[WHERE condition][GROUP BY [ROLLUP][CUBE] group_by_expression][HAVING having_expression][ORDER BY column];

Working with the GROUPING Function• The GROUPING function:

– Is used with the CUBE or ROLLUP operator– Is used to find the groups forming the subtotal in a row– Is used to differentiate stored NULL values from NULL

values created by ROLLUP or CUBE– Returns 0 or 1

24

SELECT department_id DEPTID, job_id JOB, SUM(salary),GROUPING(department_id) GRP_DEPT,GROUPING(job_id) GRP_JOB

FROM hr.employeesWHERE department_id < 50GROUP BY ROLLUP(department_id, job_id);

Working with the GROUPING: Example

12

3

25

Working with GROUPING_ID Function• Extension to the GROUPING function

• GROUPING_ID returns a number corresponding to the GROUPING bit vector associated with a row

• Useful for understanding what level the row is aggregated at and filtering those rows

26

GROUPING_ID Function ExampleSELECT department_id DEPTID, job_id JOB,

SUM(salary),GROUPING_ID(department_id,job_id) GRP_ID

FROM hr.employeesWHERE department_id < 40GROUP BY CUBE(department_id, job_id);

DEPTID JOB SUM(SALARY) GRP_ID---------- ---------- ----------- ----------

48300 3

MK_MAN 13000 2MK_REP 6000 2PU_MAN 11000 2AD_ASST 4400 2PU_CLERK 13900 2

10 4400 110 AD_ASST 4400 020 19000 120 MK_MAN 13000 020 MK_REP 6000 030 24900 130 PU_MAN 11000 030 PU_CLERK 13900 0

27

Working with GROUP_ID Function• GROUP_ID distinguishes duplicate groups

resulting from a GROUP BY specification

• A Unique group will be assigned 0, the non unique will be assigned 1 to n-1 for n duplicate groups

• Useful in filtering out duplicate groupings from the query result

28

GROUP_ID Function ExampleSELECT department_id DEPTID, job_id JOB,

SUM(salary),GROUP_ID() UNIQ_GRP_ID

FROM hr.employeesWHERE department_id < 40GROUP BY department_id, CUBE(department_id, job_id);

DEPTID JOB SUM(SALARY) UNIQ_GRP_ID---------- ---------- ----------- -----------

10 AD_ASST 4400 020 MK_MAN 13000 020 MK_REP 6000 030 PU_MAN 11000 030 PU_CLERK 13900 010 AD_ASST 4400 120 MK_MAN 13000 120 MK_REP 6000 130 PU_MAN 11000 130 PU_CLERK 13900 110 4400 020 19000 030 24900 010 4400 120 19000 130 24900 1

29

GROUPING SETS

• The GROUPING SETS syntax is used to define multiple groupings in the same query.

• All groupings specified in the GROUPING SETSclause are computed and the results of individual groupings are combined with a UNION ALLoperation.

• Grouping set efficiency:– Only one pass over the base table is required.– There is no need to write complex UNION statements.– The more elements GROUPING SETS has, the greater the

performance benefit.

31

SELECT department_id, job_id, manager_id, AVG(salary)

FROM hr.employeesGROUP BY GROUPING SETS

((department_id,job_id), (job_id,manager_id));

GROUPING SETS: Example

. . .

1

2

33

Composite Columns• A composite column is a collection of columns that

are treated as a unit.ROLLUP (a,(b,c), d)

• Use parentheses within the GROUP BY clause to group columns, so that they are treated as a unit while computing ROLLUP or CUBE operators.

• When used with ROLLUP or CUBE, composite columns require skipping aggregation across certain levels.

35

SELECT department_id, job_id, manager_id,

SUM(salary)

FROM hr.employees

GROUP BY ROLLUP( department_id,(job_id, manager_id));

Composite Columns: Example

1

2

3

4

37

Concatenated Groupings• Concatenated groupings offer a concise way to

generate useful combinations of groupings.

• To specify concatenated grouping sets, you separate multiple grouping sets, ROLLUP, and CUBE operations with commas so that the Oracle server combines them into a single GROUP BYclause.

• The result is a cross-product of groupings from each GROUPING SET.

GROUP BY GROUPING SETS(a, b), GROUPING SETS(c, d)

38

SELECT department_id, job_id, manager_id, SUM(salary)

FROM hr.employeesGROUP BY department_id,

ROLLUP(job_id),CUBE(manager_id);

Concatenated Groupings: Example

…

…

…

1

3

4

5

6

2

7

…

…

Analytic Functions

Let’s analyze our data!

40

Overview of SQL for Analysis and Reporting• Oracle has enhanced SQL's analytical processing

capabilities by introducing a new family of analytic SQL functions.

• These analytic functions enable you to calculate and perform:– Rankings and percentiles– Pivoting operations– Moving window calculations– LAG/LEAD analysis– FIRST/LAST analysis– Linear regression statistics

41

Why Use Analytic Functions?• Ability to see one row from another row in the

results

• Avoid self-join queries

• Summary data in detail rows

• Slice and dice within the results

42

Using the Analytic Functions

Function type Used for

Ranking Calculating ranks, percentiles, and n-tiles of the values in a

result set

Windowing Calculating cumulative and moving aggregates, works with functions such as SUM, AVG, MIN, and so on

Reporting Calculating shares such as market share, works with functions such as SUM, AVG, MIN, MAX, COUNT, VARIANCE,

STDDEV, RATIO_TO_REPORT, and so on

LAG/LEAD Finding a value in a row or a specified number of rows

from a current row

FIRST/LAST First or last value in an ordered group

Linear Regression Calculating linear regression and other statistics

43

Concepts Used in Analytic Functions • Result set partitions: These are created and available to any

aggregate results such as sums and averages. The term “partitions” is unrelated to the table partitions feature.

• Window: For each row in a partition, you can define a sliding window of data, which determines the range of rows used to perform the calculations for the current row.

• Current row: Each calculation performed with an analytic function is based on a current row within a partition. It serves as the reference point determining the start and end of the window.

45

Reporting Functions• We can use aggregative/group functions as analytic

functions (i.e. SUM, AVG, MIN, MAX, COUNT etc.)

• Each row will get the aggregative value for a given partition without the need for group by clause so we can have multiple group by’s on the same row

• Getting the raw data along with the aggregated value

• Use Order By to get cumulative aggregations

46

Reporting Functions Examples

SELECT last_name, salary,ROUND(AVG(salary) OVER (PARTITION BY department_id),2),COUNT(*) OVER (PARTITION BY manager_id),SUM(salary) OVER (PARTITION BY department_id ORDER BY salary),MAX(salary) OVER ()

FROM hr.employees;

Ranking Functions

48

Using the Ranking Functions• A ranking function computes the rank of a record

compared to other records in the data set based on the values of a set of measures. The types of ranking function are:– RANK and DENSE_RANK functions

– PERCENT_RANK function

– ROW_NUMBER function

– NTILE function

– CUME_DIST function

49

Working with the RANK Function

• The RANK function calculates the rank of a value in a group of values, which is useful for top-N and bottom-N reporting.

• For example, you can use the RANK function to find the top ten products sold in Boston last year.

• When using the RANK function, ascending is the default sort order, which you can change to descending.

• Rows with equal values for the ranking criteria receive the same rank.

• Oracle Database then adds the number of tied rows to the tied rank to calculate the next rank.

RANK ( ) OVER ( [query_partition_clause] order_by_clause )

50

Using the RANK Function: Example

SELECT department_id, last_name, salary,RANK() OVER (PARTITION BY department_idORDER BY salary DESC) "Rank"

FROM employees WHERE department_id = 60ORDER BY department_id, "Rank", salary;

51

Per-Group Ranking• The RANK function can be made to operate within

groups - that is, the rank gets reset whenever the group changes

• This is accomplished with the PARTITION BY clause

• The group expressions in the PARTITION BY sub-clause divide the data set into groups within which RANK operates

• For example: to rank products within each channel by their dollar sales, you could issue a statement similar to the one in the next slide.

52

Per-Group Ranking: Example

SELECT channel_desc, calendar_month_desc, TO_CHAR(SUM(amount_sold),'9,999,999,999') SALES$, RANK() OVER (PARTITION BY channel_descORDER BY SUM(amount_sold) DESC) AS RANK_BY_CHANNELFROM sales, products, customers, times, channelsWHERE sales.prod_id = products.prod_idAND sales.cust_id = customers.cust_idAND sales.time_id = times.time_idAND sales.channel_Id = channels.channel_idAND times.calendar_month_desc IN ('2000-08', '2000-09', '2000-

10', '2000-11')AND channels.channel_desc IN ('Direct Sales', 'Internet')

GROUP BY channel_desc, calendar_month_desc;

53

RANK and DENSE_RANK Functions: Example

SELECT department_id, last_name, salary,RANK() OVER (PARTITION BY department_idORDER BY salary DESC) "Rank",

DENSE_RANK() over (partition by department_idORDER BY salary DESC) "Drank"

FROM employees WHERE department_id = 60ORDER BY department_id, last_name, salary DESC, "Rank" DESC;

DENSE_RANK ( ) OVER ([query_partition_clause] order_by_clause)

54

Per-Cube and Rollup Group RankingSELECT channel_desc, country_iso_code, TO_CHAR(SUM(amount_sold), '9,999,999,999')SALES$, RANK() OVER

(PARTITION BY GROUPING_ID(channel_desc, country_iso_code)ORDER BY SUM(amount_sold) DESC) AS RANK_PER_GROUP

FROM sales, customers, times, channels, countriesWHERE sales.time_id = times.time_id AND

sales.cust_id=customers.cust_id AND sales.channel_id = channels.channel_id AND channels.channel_desc IN ('Direct Sales', 'Internet') AND

times.calendar_month_desc='2000-09' AND country_iso_code IN ('GB', 'US', 'JP')

GROUP BY CUBE(channel_desc, country_iso_code);

55

Using the PERCENT_RANK Function• Uses rank values in its numerator and returns the percent rank of a

value relative to a group of values

• PERCENT_RANK of a row is calculated as follows:

• The range of values returned by PERCENT_RANK is 0 to 1, inclusive. The first row in any set has a PERCENT_RANK of 0. The return value is NUMBER. Its syntax is:

(rank of row in its partition - 1) / (number of rows in the partition - 1)

PERCENT_RANK () OVER ([query_partition_clause] order_by_clause)

56

Using the PERCENT_RANK Function: ExampleSELECT department_id, last_name, salary, PERCENT_RANK()

OVER (PARTITION BY department_id ORDER BY salary DESC) AS pr

FROM hr.employeesORDER BY department_id, pr, salary;

57

Working with the ROW_NUMBER Function

• The ROW_NUMBER function calculates a sequential number of a value in a group of values.

• When using the ROW_NUMBER function, ascending is the default sort order, which you can change to descending.

• Rows with equal values for the ranking criteria receive a different number.

ROW_NUMBER ( ) OVER ( [query_partition_clause] order_by_clause )

58

ROW_NUMBER VS. ROWNUM

• ROWNUM is a pseudo column, ROW_NUMBER is an actual function

• ROWNUM requires sorting of the entire dataset in order to return ordered list

• ROW_NUMBER will only sort the required rows thus giving better performance

59

Working With The NTILE Function

• Not really a rank function

• Divides an ordered data set into a number of buckets indicated by expr and assigns the appropriate bucket number to each row

• The buckets are numbered 1 through expr

NTILE ( expr ) OVER ([query_partition_clause] order_by_clause)

60

Summary of Ranking Functions• Different ranking functions may return different

results if the data has ties

SELECT last_name, salary, department_id,ROW_NUMBER() OVER (PARTITION BY department_id ORDER BY salary DESC) A,RANK() OVER (PARTITION BY department_id ORDER BY salary DESC) B,DENSE_RANK() OVER (PARTITION BY department_id ORDER BY salary DESC) C,PERCENT_RANK() OVER (PARTITION BY department_id ORDER BY salary DESC) D,NTILE(4) OVER (PARTITION BY department_id ORDER BY salary DESC) E

FROM hr.employees;

60

Inter-row Analytic Functions

62

Using the LAG and LEAD Analytic Functions

• LAG provides access to more than one row of a table at the same time without a self-join.

• Given a series of rows returned from a query and a position of the cursor, LAG provides access to a row at a given physical offset before that position.

• If you do not specify the offset, its default is 1. • If the offset goes beyond the scope of the window, the

optional default value is returned. If you do not specify the default, its value is NULL.

{LAG | LEAD}(value_expr [, offset ] [, default ])OVER ([ query_partition_clause ] order_by_clause)

63

Using the LAG and LEAD: Example

SELECT time_id, TO_CHAR(SUM(amount_sold),'9,999,999') AS SALES,

TO_CHAR(LAG(SUM(amount_sold),1) OVER (ORDER BY time_id),'9,999,999') AS LAG1,

TO_CHAR(LEAD(SUM(amount_sold),1) OVER (ORDER BY time_id),'9,999,999') AS LEAD1

FROM salesWHERE time_id >= TO_DATE('10-OCT-2000') AND

time_id <= TO_DATE('14-OCT-2000')GROUP BY time_id;

64

Using the LISTAGG Function

• For a specified measure, LISTAGG orders data within each group specified in the ORDER BY clause and then concatenates the values of the measure column

LISTAGG(measure_expr [, 'delimiter'])WITHIN GROUP (order_by_clause) [OVER query_partition_clause]

65

Using LISTAGG: ExampleSELECT department_id "Dept", hire_date

"Date", last_name "Name", LISTAGG(last_name, ', ') WITHIN GROUP

(ORDER BY hire_date, last_name)OVER (PARTITION BY department_id) as

"Emp_list"FROM hr.employeesWHERE hire_date < '01-SEP-2003'ORDER BY "Dept", "Date", "Name";

66

LISTAGG in Oracle 12c• Limited to output of 4000 chars or 32000 with

extended column sizes

• Oracle 12cR2 provides overflow handling:

• Example:

listagg (measure_expr, ','[ on overflow (truncate|error) ][ text ] [ (with|without) count ]

) within group (order by cols)

select listagg(table_name, ',' on overflow truncate) within group (order by table_name) table_names

from dba_tables

67

Using the FIRST and LAST Functions

• Both are aggregate and analytic functions

• Used to retrieve a value from the first or last row of a sorted group, but the needed value is not the sort key

• FIRST and LAST functions eliminate the need for self-joins or views and enable better performance

aggregate_function KEEP(DENSE_RANK FIRST ORDER BYexpr [ DESC | ASC ][ NULLS { FIRST | LAST } ][, expr [ DESC | ASC ] [ NULLS { FIRST | LAST } ]]...

)[ OVER query_partition_clause ]

68

FIRST and LAST Aggregate ExampleSELECT department_id,

MIN(salary) KEEP (DENSE_RANK FIRST ORDER BY commission_pct)

"Worst",

MAX(salary) KEEP (DENSE_RANK LAST ORDER BY commission_pct)

"Best"

FROM employees

GROUP BY department_id

ORDER BY department_id;

69

FIRST and LAST Analytic ExampleSELECT last_name, department_id, salary,

MIN(salary) KEEP (DENSE_RANK FIRST ORDER BY commission_pct)

OVER (PARTITION BY department_id) "Worst",

MAX(salary) KEEP (DENSE_RANK LAST ORDER BY commission_pct)

OVER (PARTITION BY department_id) "Best"

FROM employees

ORDER BY department_id, salary, last_name;

70

Using FIRST_VALUE Analytic Function

• Returns the first value in an ordered set of values

• If the first value in the set is null, then the function returns NULL unless you specify IGNORE NULLS. This setting is useful for data densification.

FIRST_VALUE (expr [ IGNORE NULLS ]) OVER (analytic_clause)

71

Using FIRST_VALUE: ExampleSELECT department_id, last_name, salary,

FIRST_VALUE(last_name) OVER (ORDER BY salary ASC ROWS

UNBOUNDED PRECEDING) AS lowest_sal

FROM (SELECT * FROM employees WHERE department_id = 30

ORDER BY employee_id)

ORDER BY department_id, last_name, salary, lowest_sal;

72

Using LAST_VALUE Analytic Function

• Returns the last value in an ordered set of values.

LAST_VALUE (expr [ IGNORE NULLS ]) OVER (analytic_clause)

73

Using NTH_VALUE Analytic Function

• Returns the N-th values in an ordered set of values

• Different default window: RANGE BETWEEN UNBOUNDED PRECEDING

AND CURRENT ROW

NTH_VALUE (measure_expr, n)[ FROM { FIRST | LAST } ][ { RESPECT | IGNORE } NULLS ] OVER (analytic_clause)

74

Using NTH_VALUE: Example

SELECT prod_id, channel_id, MIN(amount_sold),

NTH_VALUE ( MIN(amount_sold), 2) OVER (PARTITION BY

prod_id ORDER BY channel_id

ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED

FOLLOWING) nv

FROM sh.sales

WHERE prod_id BETWEEN 13 and 16

GROUP BY prod_id, channel_id;

75

Using NTH_VALUE: Example

SELECT prod_id, channel_id, MIN(amount_sold),

NTH_VALUE ( MIN(amount_sold), 2) OVER (PARTITION BY

prod_id ORDER BY channel_id

ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED

FOLLOWING) nv

FROM sh.sales

WHERE prod_id BETWEEN 13 and 16

GROUP BY prod_id, channel_id;

Window Functions

77

Window Functions• The windowing_clause gives some analytic

functions a further degree of control over this window within the current partition

• The windowing_clause can only be used if an order_by_clause is present

78

Windows Can Be By RANGE Or ROWS

Possible values for start_point and end_point

UNBOUNDED PRECEDING The window starts at the first row of the partition. Only available for start points.

UNBOUNDED FOLLOWING The window ends at the last row of the partition. Only available for end points.

CURRENT ROW The window starts or ends at the current row

value_expr PRECEDING A physical or logical offset before the current row.When used with RANGE, can also be an interval literal

value_expr FOLLOWING As above, but an offset after the current row

RANGE BETWEEN start_point AND end_point

ROWS BETWEEN start_point AND end_point

79

Shortcuts• Useful shortcuts for the windowing clause:

• The windows are limited to the current partition

• Generally, the default window is the entire work set unless said otherwise

ROWS UNBOUNDED PRECEDING ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW

ROWS 10 PRECEDING ROWS BETWEEN 10 PRECEDING AND CURRENT ROW

ROWS CURRENT ROW ROWS BETWEEN CURRENT ROW AND CURRENT ROW

80

Windowing Clause Useful Usages• Cumulative aggregation

• Sliding average over proceeding and/or following rows

• Using the RANGE parameter to filter aggregation records

Pivot and Unpivot

Turning things around!

82

PIVOT and UNPIVOT• You can use the PIVOT operator of the SELECT

statement to write cross-tabulation queries that rotate the column values into new columns, aggregating data in the process.

• You can use the UNPIVOT operator of the SELECT statement to rotate columns into values of a column.

PIVOT UNPIVOT

83

Pivoting on the QUARTERColumn: Conceptual Example

30,000

40,000

60,000

30,000

40,000

20,000

AMOUNT_SOLD

2,500Q1IUSAKids Jeans

2,000Q2CJapanKids Jeans

2,000Q3SUSAShorts

I

P

C

CHANNEL

Kids Jeans

Shorts

Shorts

PRODUCT

1,000Q2Germany

1,500Q4USA

Q2

QUARTER

2,500Poland

QUANTITY_SOLD

COUNTRY

2,000

Q3

Kids Jeans

Shorts

PRODUCT

3,500

2,000

Q2

1,5002,500

Q4Q1

84

Pivoting Before Oracle 11g• Pivoting the data before 11g was a complex query

which required the use of the CASE or DECODEfunctions

select product,sum(case when quarter = 'Q1' then amount_sold else null end) Q1,sum(case when quarter = 'Q2' then amount_sold else null end) Q2,sum(case when quarter = 'Q3' then amount_sold else null end) Q3,sum(case when quarter = 'Q4' then amount_sold else null end) Q4from sales

group by product;

85

PIVOT Clause Syntaxtable_reference PIVOT [ XML ]

( aggregate_function ( expr ) [[AS] alias ][, aggregate_function ( expr ) [[AS] alias ] ]...

pivot_for_clausepivot_in_clause )

-- Specify the column(s) to pivot whose values are to -- be pivoted into columns.pivot_for_clause = FOR { column |( column [, column]... ) }

-- Specify the pivot column values from the columns you -- specified in the pivot_for_clause.pivot_in_clause = IN ( { { { expr | ( expr [, expr]... ) } [ [ AS] alias] }...

| subquery | { ANY | ANY [, ANY]...} } )

87

Creating a New View: Example

CREATE OR REPLACE VIEW sales_view ASSELECTprod_name AS product, country_name AS country, channel_id AS channel, SUBSTR(calendar_quarter_desc, 6,2) AS quarter,SUM(amount_sold) AS amount_sold, SUM(quantity_sold) AS quantity_sold

FROM sales, times, customers, countries, productsWHERE sales.time_id = times.time_id AND

sales.prod_id = products.prod_id ANDsales.cust_id = customers.cust_id ANDcustomers.country_id = countries.country_id

GROUP BY prod_name, country_name, channel_id,SUBSTR(calendar_quarter_desc, 6, 2);

89

Selecting the SALES VIEW DataSELECT product, country, channel, quarter, quantity_soldFROM sales_view;

PRODUCT COUNTRY CHANNEL QUARTER QUANTITY_SOLD------------ ------------ ---------- -------- -------------Y Box Italy 4 01 21Y Box Italy 4 02 17Y Box Italy 4 03 20. . .Y Box Japan 2 01 35Y Box Japan 2 02 39Y Box Japan 2 03 36Y Box Japan 2 04 46Y Box Japan 3 01 65. . .Bounce Italy 2 01 34Bounce Italy 2 02 43. . .9502 rows selected.

90

Pivoting the QUARTER Column in the SH Schema: Example

SELECT *FROM

(SELECT product, quarter, quantity_soldFROM sales_view) PIVOT (sum(quantity_sold) FOR quarter IN ('01', '02', '03', '04'))

ORDER BY product DESC;

. . .

92

Unpivoting the QUARTER Column: Conceptual Example

2,000

Q3

Kids Jeans

Shorts

PRODUCT

3,500

2,000

Q2

1,5002,500

Q4Q1

2,500Q1Kids Jeans

2,000Q2Kids Jeans

3,500Q2Shorts

1,500Q4Kids Jeans

Q3

QUARTER

2,000Shorts

SUM_OF_QUANTITYPRODUCT

93

Unpivoting Before Oracle 11g• Univoting the data before 11g requires multiple

queries on the table using the UNION ALL operator

SELECT *FROM (

SELECT product, '01' AS quarter, Q1_value FROM salesUNION ALLSELECT product, '02' AS quarter, Q2_value FROM salesUNION ALLSELECT product, '03' AS quarter, Q3_value FROM salesUNION ALLSELECT product, '04' AS quarter, Q4_value FROM sales);

94

Using the UNPIVOT Operator• An UNPIVOT operation does not reverse a PIVOT

operation; instead, it rotates data found in multiple columns of a single row into multiple rows of a single column.

• If you are working with pivoted data, UNPIVOTcannot reverse any aggregations that have been made by PIVOT or any other means.

UNPIVOT

95

Using the UNPIVOT Clause• The UNPIVOT clause rotates columns from a

previously pivoted table or a regular table into rows. You specify:– The measure column or columns to be unpivoted

– The name or names for the columns that result from the UNPIVOT operation

– The columns that are unpivoted back into values of the column specified in pivot_for_clause

• You can use an alias to map the column name to another value.

96

UNPIVOT Clause Syntaxtable_reference UNPIVOT [{INCLUDE|EXCLUDE} NULLS]-- specify the measure column(s) to be unpivoted.( { column | ( column [, column]... ) }

unpivot_for_clauseunpivot_in_clause )

-- Specify one or more names for the columns that will-- result from the unpivot operation.

unpivot_for_clause = FOR { column | ( column [, column]... ) }

-- Specify the columns that will be unpivoted into values of -- the column specified in the unpivot_for_clause.

unpivot_in_clause = ( { column | ( column [, column]... ) }

[ AS { constant | ( constant [, constant]... ) } ][, { column | ( column [, column]... ) }[ AS { constant | ( constant [, constant]...) } ] ]...)

97

Creating a New Pivot Table: Example

. . .

CREATE TABLE pivotedtable ASSELECT *FROM

(SELECT product, quarter, quantity_soldFROM sales_view) PIVOT (sum(quantity_sold) FOR quarter IN ('01' AS Q1, '02' AS Q2,

'03' AS Q3, '04' AS Q4));

SELECT * FROM pivotedtable ORDER BY product DESC;

98

Unpivoting the QUARTER Column : Example• Unpivoting the QUARTER Column in the SH Schema:

SELECT *FROM pivotedtableUNPIVOT (quantity_sold For Quarter IN (Q1, Q2, Q3, Q4))ORDER BY product DESC, quarter;

. . .

99

More Examples…• More information and examples could be found on

my Blog:

http://www.realdbamagic.com/he/pivot-a-table/

Top-N and Paging QueriesIn Oracle 12c

101

Top-N Queries• A Top-N query is used to retrieve the top or

bottom N rows from an ordered set

• Combining two Top-N queries gives you the ability to page through an ordered set

• Oracle 12c has introduced the row limiting clause to simplify Top-N queries

102

Top-N in 12cR1

• This is ANSI syntax

• The default offset is 0

• Null values in offset, rowcount or percent will return no rows

[ OFFSET offset { ROW | ROWS } ][ FETCH { FIRST | NEXT } [ { rowcount | percent PERCENT } ]

{ ROW | ROWS } { ONLY | WITH TIES } ]

103

Top-N Examples

SELECT last_name, salaryFROM hr.employeesORDER BY salaryFETCH FIRST 4 ROWS ONLY;

SELECT last_name, salaryFROM hr.employeesORDER BY salaryFETCH FIRST 4 ROWS WITH TIES;

SELECT last_name, salaryFROM hr.employeesORDER BY salary DESCFETCH FIRST 10 PERCENT ROWS ONLY;

104

Paging Before 12c• Before 12c we had to use the rownum pseudo

column to filter out rows

• That will require sorting the entire rowset

SELECT valFROM (SELECT val, rownum AS rnum

FROM (SELECT valFROM rownum_order_testORDER BY val)

WHERE rownum <= 10)WHERE rnum >= 5;

105

Paging in Oracle 12c• After 12c we have a syntax improvement for

paging using the Top-N queries

• This will use ROW_NUMBER and RANK in the background – there is no real optimization improvements

SELECT valFROM rownum_order_testORDER BY valOFFSET 4 ROWS FETCH NEXT 5 ROWS ONLY;

106

More Examples• More information and examples could be found on

my blog:

http://www.realdbamagic.com/he/12c-top-n-query/

107

Analytic Functions and Performance• Analytic functions has positive impact on

performance for the most part

• Using analytic functions can reduce the number of table scans and reduce IO consumption

• The query might use more CPU and/or memory but it will usually run faster than the same result without analytic functions

• Top-N queries might struggle with cardinality evaluation when using the “With Ties” option

Common Table Expression and Subquery Factoring

109

Subquery Factoring• The WITH clause, or subquery factoring clause, is

part of the SQL-99 standard

• Introduced in Oracle 9.2

• The WITH produces a new inline view which we can query from

• Sometimes, the subquery is being cached (materialized) so it does not need to re-query the data again

110

Subquery ExampleSELECT e.LAST_NAME AS employee_name,

dc.dept_count AS emp_dept_countFROM employees e,

(SELECT DEPARTMENT_ID, COUNT(*) AS dept_countFROM employeesGROUP BY DEPARTMENT_ID) dc

WHERE e.DEPARTMENT_ID = dc.DEPARTMENT_ID;

WITH dept_count AS (SELECT DEPARTMENT_ID, COUNT(*) AS dept_count

FROM employeesGROUP BY DEPARTMENT_ID)

SELECT e.LAST_NAME AS employee_name,dc.dept_count AS emp_dept_count

FROM employees e,dept_count dc

WHERE e.DEPARTMENT_ID = dc.DEPARTMENT_ID;

111

Subquery ReuseWITH dept_count AS (SELECT DEPARTMENT_ID, COUNT(*) AS dept_count

FROM employeesGROUP BY DEPARTMENT_ID)

SELECT e1.LAST_NAME AS employee_name,e2.LAST_NAME as Manager_name,

dc1.dept_count AS emp_dept_count,dc2.dept_count as mgr_dept_count

FROM employees e1,employees e2,dept_count dc1,dept_count dc2

WHERE e1.DEPARTMENT_ID = dc1.DEPARTMENT_ID ande2.DEPARTMENT_ID = dc2.DEPARTMENT_ID and

e1.MANAGER_ID = e2.employee_id

112

Functions in the WITH Clause (12.1)

• Oracle 12c allows us the definition of anonymous function within the scope of a querywith

function sumascii (str in varchar2) return number isx number := 0;

beginfor i in 1..length (str)loop

x := x + ascii (substr (str, i, 1)) ;end loop;return x;

end;select /*+ WITH_PLSQL */ h.EMPLOYEE_ID, h.last_name,

sumascii (h.last_name) from hr.employees h

Hierarchical Queries andRecursive Queries

114

Using Hierarchical Queries• You can use hierarchical queries to retrieve data based on a

natural hierarchical relationship between rows in a table.

• A relational database does not store records in a hierarchical way; therefore, a hierarchical query is possible only when a relationship exists between rows in a table.

• However, where a hierarchical relationship exists between the rows of a single table, a process called “tree walking” enables the hierarchy to be constructed.

• A hierarchical query is a method of reporting, with the branches of a tree in a specific order.

115

Business Challenges• Getting all employees that report directly or

indirectly to a manager

• Managing documents and folders

• Managing privileges

• Aggregating levels on the same row

116

Using Hierarchical Queries: Example• Sample Data from the EMPLOYEES Table (HR schema)

• Kochhar, De Haan, and Hartstein report to the same manager (MANAGER_ID = 100)

• EMPLOYEE_ID = 100 is King

…

117

Natural Tree Structure

De Haan

HunoldWhalen

Kochhar

Higgins

Mourgos Zlotkey

Rajs Davies Matos

Gietz Ernst Lorentz

Hartstein

Fay

Abel Taylor Grant

Vargas

MANAGER_ID = 100 (Child)

EMPLOYEE_ID = 100 (Parent)

. . . . . .

. . .

. . .

. . .

King

118

Hierarchical Queries: Syntax• condition:

expr comparison_operator expr

SELECT [LEVEL], column, expr...FROM table

[WHERE condition(s)]

[START WITH condition(s)]

[CONNECT BY PRIOR condition(s)] ;

119

Walking the Tree: Specifying the Starting Point• Use the START WITH clause to specify the starting

point, that is, the row or rows to be used as the root of the tree:– Specifies the condition that must be met

– Accepts any condition that is valid in a WHERE clause

• For example, using the HR.EMPLOYEES table, start with the employee whose last name is Kochhar.. . .START WITH last_name = 'Kochhar'

START WITH column1 = value

120

Walking the Tree: Specifying the Direction• The direction of the query is determined by the

CONNECT BY PRIOR column placement.

• The PRIOR operator refers to the parent row.

CONNECT BY PRIOR column1 = column2

. . . CONNECT BY PRIOR employee_id = manager_id. . .

Parent key Child key

121

Hierarchical Query Example: Using the CONNECT BY Clause

SELECT employee_id, last_name, manager_idFROM hr.employees;

. . .

122

Specifying the Direction of the Query: From the Top Down

SELECT last_name||' reports to '|| PRIOR last_name "Walk Top Down"

FROM hr.employees

START WITH last_name = 'King'

CONNECT BY PRIOR employee_id = manager_id ;

. . .

123

Specifying the Direction of the Query: From the Bottom Up

SELECT employee_id, last_name, job_id, manager_id

FROM hr.employees

START WITH employee_id = 101

CONNECT BY PRIOR manager_id = employee_id ;

124

Using the LEVEL Pseudocolumn

Level 1root/

parent

Level 3parent/

child/leaf

Level 4leaf

De Haan

King

HunoldWhalen

Kochhar

Higgins

Mourgos Zlotkey

Rajs Davies Matos

Gietz Ernst Lorentz

Hartstein

Fay

Abel Taylor Grant

Vargas

Level 2parent/child

125

Using the LEVEL Pseudocolumn: ExampleSELECT employee_id, last_name, manager_id, LEVELFROM hr.employeesSTART WITH employee_id = 100CONNECT BY PRIOR employee_id = manager_idORDER siblings BY last_name;

. . .

126

Formatting Hierarchical Reports

• It is common to format Hierarchical reports using LEVEL and LPAD

– Create a report displaying company management levels beginning with the highest level and indenting each of the following levels.

SELECT LPAD(last_name, LENGTH(last_name)+(LEVEL*2)-2,'_') AS org_chart

FROM hr.employeesSTART WITH first_name = 'Steven' AND last_name = 'King' CONNECT BY PRIOR employee_id = manager_idORDER SIBLINGS BY last_name;

127

Result

128

Pruning Nodes and Branches• Use the WHERE clause to eliminate a node

• Use the CONNECT BY clause to eliminate a branch

Kochhar

Higgins

Gietz

Whalen

Kochhar

HigginsWhalen

Gietz

. . .WHERE last_name != 'Higgins'

. . .CONNECT BY PRIOR employee_id = manager_idAND last_name != 'Higgins'

1 2

129

Pruning Branches Example 1:Eliminating a Node

SELECT department_id, employee_id,last_name, job_id, salary

FROM hr.employeesWHERE last_name != 'Higgins'START WITH manager_id IS NULLCONNECT BY PRIOR employee_id = manager_id;

. . .

. . .

. . .

130

Pruning Branches Example 2:Eliminating a Branch

SELECT department_id, employee_id,last_name, job_id, salaryFROM hr.employeesSTART WITH manager_id IS NULLCONNECT BY PRIOR employee_id = manager_idAND last_name != 'Higgins';

. . .

131

Order of Precedence • Join happens before connect by

• Where is happening after connect by

• Regular order by will rearrange the returning rows

• Sibling order by will rearrange the returning rows for each level

132

Other Connect By Functions• CONNECT_BY_ISCYCLE

• CONNECT_BY_ISLEAF

• CONNECT_BY_ROOT

• SYS_CONNECT_BY_PATH

133

Recursive Subquery Factoring• ANSI SQL:2008 (Oracle 11g) introduced a new way

to run hierarchical queries: Recursive Subquery Factoring using Subquery Factoring

• That will mean that a query will query itself using the WITH clause, making queries easier to write

137

Recursive Subquery Factoring Example

with mytree(id, parent_id, "level")as (

select id, parent_id, 1 as "level"from temp_vwhere id = 1

union allselect temp_v.id, temp_v.parent_id,

mytree."level" + 1from temp_v, mytreewhere temp_v.parent_id = mytree.id

)Select * from mytree;

Stop Condition

Actual Recursion

139

Warning: Performance• Recursion and Hierarchies might have bad impact

on performance

• Watch out for mega-trees – it has CPU and memory impacts

• Using recursion might lead for multiple IO reads of the same blocks

Regular Expression

141

Regular Expression• Regular expression (regexp) is a sequence of

characters that define a search pattern

• Commonly used for smart “Search and Replace” of patterns and for input validations of text

• Widely introduced in Oracle 10g (and it even existed even before that)

142

Common REGEXP Functions and Operators

REGEXP_LIKE Perform regular expression matching

REGEXP_REPLACE Extends the functionality of the REPLACEfunction by using patterns

REGEXP_SUBSTR Extends the functionality of the SUBSTRfunction by using patterns

REGEXP_COUNT Count the number of matches of the pattern in a given string

REGEXP_INSTR Extends the functionality of the INSTRfunction by using patterns

143

Supported Regular Expression Patterns• Concatenation: No operator between elements.• Quantifiers:

– . Matches any character in the database character set– * 0 or more matches– + 1 or more matches– ? 0 or 1 match– {n} Exactly n matches– {n,} n or more matches– {n, m} Between n and m (inclusive) matches– {, m} Between 0 an m (inclusive) matches

• Alternation: [|]• Grouping: ()

144

Supported Regular Expression PatternsValue Description

^Matches the beginning of a string. If used with a match_parameter of 'm', it matches the start of a line anywhere within expression.

$Matches the end of a string. If used with a match_parameter of 'm', it matches the end of a line anywhere withinexpression.

\W Matches a nonword character.

\s Matches a whitespace character.

\S matches a non-whitespace character.

\AMatches the beginning of a string or matches at the end of a string before a newline character.

\Z Matches at the end of a string.

145

Character ClassesCharacter Class Description

[:alnum:] Alphanumeric characters

[:alpha:] Alphabetic characters

[:blank:] Blank Space Characters

[:cntrl:] Control characters (nonprinting)

[:digit:] Numeric digits

[:graph:] Any [:punct:], [:upper:], [:lower:], and [:digit:] chars

[:lower:] Lowercase alphabetic characters

[:print:] Printable characters

[:punct:] Punctuation characters

[:space:]Space characters (nonprinting), such as carriage return, newline, vertical tab, and form feed

[:upper:] Uppercase alphabetic characters

[:xdigit:] Hexidecimal characters

Regular Expression Demo

147

Pitfalls• Regular expressions might be slow when used on

large amount of data

• Writing regular expression can be very tricky –make sure your pattern is correct

• Oracle REGEXP syntax is not standard, regular expression might not work or partially work causing wrong results

• There can only be up to 9 placeholders in a given quantifier

Pattern Matching inOracle 12c

149

What is Pattern Matching• Identify and group rows with consecutive values

• Consecutive in this regards – row after row

• Uses regular expression like syntax to find patterns

150

Common Business Challenges• Finding sequences of events in security

applications

• Locating dropped calls in a CDR listing

• Financial price behaviors (V-shape, W-shape U-shape, etc.)

• Fraud detection and sensor data analysis

151

MATCH_RECOGNIZE Syntax

SELECTFROM [row pattern input table]MATCH_RECOGNIZE`( [ PARTITION BY <cols> ][ ORDER BY <cols> ][ MEASURES <cols> ][ ONE ROW PER MATCH | ALL ROWS PER MATCH ][ SKIP_TO_option]PATTERN ( <row pattern> )DEFINE <definition list>)

152

Basix Syntax Legend• PARTITION BY divides the data in to logical groups

• ORDER BY orders the data in each logical group

• MEASURES define the data measures of the pattern

• ONE/ALL ROW PER MATCH defines what to do with the pattern – return one row or all rows

• PATTERN says what the pattern actually is

• DEFINE gives us the condition that must be met for a row to map to the pattern variables

153

MATCH_RECOGNIZE Example

• Find Simple V-Shape with 1 row output per match

SELECT *FROM Ticker MATCH_RECOGNIZE (

PARTITION BY symbolORDER BY tstampMEASURES STRT.tstamp AS start_tstamp,

LAST(DOWN.tstamp) AS bottom_tstamp,LAST(UP.tstamp) AS end_tstamp

ONE ROW PER MATCHAFTER MATCH SKIP TO LAST UPPATTERN (STRT DOWN+ UP+)DEFINE

DOWN AS DOWN.price < PREV(DOWN.price),UP AS UP.price > PREV(UP.price)

) MRORDER BY MR.symbol, MR.start_tstamp;

154

What Will Be Matched?

155

Example: Sequential Employee IDs• Our goal: find groups of users with sequences IDs

• This can be useful for detecting missing employees in a table, or to locate “gaps” in a group

FIRSTEMP LASTEMP---------- ----------

7371 74987500 75207522 75657567 76537655 76977699 77817783 77877789 7838

156

Pattern Matching Example

SELECT *

FROM Emps

MATCH_RECOGNIZE (

ORDER BY emp_id

PATTERN (STRT B*)

DEFINE B AS emp_id = PREV(emp_id)+1

ONE ROW PER MATCH

MEASURES

STRT.emp_id firstemp,

LAST(emp_id) lastemp

AFTER MATCH SKIP PAST LAST ROW

);

1. Define input

2. Pattern Matching

3. Order input

4. Process pattern

5. Using defined conditions

6. Output: rows per match

7. Output: columns per row

8. Where to go after match?

Original concept by Stew Ashton

157

Pattern Matching Example (Actual Syntax)

SELECT *

FROM Emps

MATCH_RECOGNIZE (

ORDER BY emp_id

MEASURES

STRT.emp_id firstemp,

LAST(emp_id) lastemp

ONE ROW PER MATCH

AFTER MATCH SKIP PAST LAST ROW

PATTERN (STRT B*)

DEFINE B AS emp_id = PREV(emp_id)+1

);

1. Define input

2. Pattern Matching

3. Order input

4. Process pattern

5. Using defined conditions

6. Output: rows per match

7. Output: columns per row

8. Where to go after match?

158

Oracle 11g Analytic Function Solution

select firstemp, lastempFrom (select nvl (lag (r) over (order by r), minr) firstemp, q lastemp

from (select emp_id r,lag (emp_id) over (order by emp_id) q,min (emp_id) over () minr,max (emp_id) over () maxr

from emps e1)where r != q + 1 -- groups including lower endunionselect q,

nvl (lead (r) over (order by r), maxr)from ( select emp_id r,

lead (emp_id) over (order by emp_id) q,min (emp_id) over () minr,max (emp_id) over () maxr

from emps e1)where r + 1 != q -- groups including higher end

);

159

Supported Regular Expression Patterns• Concatenation: No operator between elements.• Quantifiers:

– * 0 or more matches.– + 1 or more matches– ? 0 or 1 match.– {n} Exactly n matches.– {n,} n or more matches.– {n, m} Between n and m (inclusive) matches.– {, m} Between 0 an m (inclusive) matches.

• Alternation: |• Grouping: ()

160

Functions• CLASSIFIER(): Which pattern variable applies to which row

• MATCH_NUMBER(): Which rows are members of which match

• PREV(): Access to a column/expression in a previous row

• NEXT(): Access to a column/expression in the next row

• LAST(): Last value within the pattern match

• FIRST(): First value within the pattern match

• COUNT(), AVG(), MAX(), MIN(), SUM()

161

Example: All Rows Per Match• Find suspicious transfers – a large transfer after 3 small

ones

SELECT userid, match_id, pattern_variable, time, amountFROM (SELECT * FROM event_logWHERE event = 'transfer')MATCH_RECOGNIZE(PARTITION BY userid ORDER BY timeMEASURESMATCH_NUMBER() match_id,CLASSIFIER() pattern_variableALL ROWS PER MATCHPATTERN ( x{3,} y)DEFINEx AS (amount < 2000 AND LAST(x.time) -FIRST(x.time) < 30),y AS (amount >= 1000000 AND y.time-LAST(x.time) < 10));

162

The Output• MATCH_ID shows current match sequence

• PATTERN_VARIABLE show which variable was applied

• USERID is the partition key

USERID MATCH_ID PATTERN_VA TIME AMOUNT-------- ---------- ---------- --------- ----------john 1 X 06-JAN-12 1000john 1 X 15-JAN-12 1500john 1 X 20-JAN-12 1500john 1 X 23-JAN-12 1000john 1 Y 26-JAN-12 1000000

163

Example: One Row Per Match• Same as before – show one row per match

SELECT userid, first_trx, last_trx, amountFROM (SELECT * FROM event_log WHERE event = 'transfer')MATCH_RECOGNIZE(PARTITION BY userid ORDER BY timeMEASURESFIRST(x.time) first_trx,y.time last_trx,y.amount amountONE ROW PER MATCHPATTERN ( x{3,} y )DEFINEx AS (amount < 2000 AND LAST(x.time) -FIRST(x.time) < 30),y AS (amount >= 1000000 AND y.time-LAST(x.time) < 10));

164

The Output• USERID is the partition key

• FIRST_TRX is a calculated measure

• AMOUNT and LAST_TRX are measures

USERID FIRST_TRX LAST_TRX AMOUNT-------- --------- --------- ----------john 06-JAN-12 26-JAN-12 1000000

165

Few Last Tips• Test all cases: pattern matching can be very tricky

• Don’t forget to test your data with no matches

• There is no LISTAGG and no DISTINCT when using match recognition

• Pattern variables cannot be used as bind variables

Using XML with SQL

167

What is XML• XML stand for eXtensible Markup Language

• Defines a set of rules for encoding documents in a format which is both human-readable and machine-readable

• Data is unstructured and can be transferred easily to other system

168

XML Terminology• Root

• Element

• Attribute

• Forest

• XML Fragment

• XML Document

169

What Does XML Look Like?

<?xml version="1.0"?><ROWSET><ROW><USERNAME>SYS</USERNAME><USER_ID>0</USER_ID><CREATED>28-JAN-08</CREATED>

</ROW><ROW>

<USERNAME>SYSTEM</USERNAME><USER_ID>5</USER_ID><CREATED>28-JAN-08</CREATED>

</ROW></ROWSET>

170

Generating XML From Oracle• Concatenating strings – building the XML

manually. This is highly not recommended

• Using DBMS_XMLGEN

• Using ANSI SQL:2003 XML functions

171

Using DBMS_XMLGEN• The DBMS_XMLGEN package converts the results

of a SQL query to a canonical XML format

• The package takes an arbitrary SQL query as input, converts it to XML format, and returns the result as a CLOB

• Using the DBMS_XMLGEN we can create contexts and use it to build XML documents

• Old package – exists since Oracle 9i

172

Example of Using DBMS_XMLGENselect dbms_xmlgen.getxml(q'{select column_name, data_typefrom all_tab_columnswhere table_name = 'EMPLOYEES' and owner = 'HR'}')from dual/

<?xml version="1.0"?><ROWSET><ROW><COLUMN_NAME>EMPLOYEE_ID</COLUMN_NAME><DATA_TYPE>NUMBER</DATA_TYPE>

</ROW><ROW><COLUMN_NAME>FIRST_NAME</COLUMN_NAME><DATA_TYPE>VARCHAR2</DATA_TYPE>

</ROW>[...]</ROWSET>

173

Why Not Use DBMS_XMLGEN• DBMS_XMLGEN is an old package (9.0 and 9i)

• Any context change requires complex PL/SQL

• There are improved ways to use XML in queries

• Use DBMS_XMLGEN for the “quick and dirty” solution only

174

Standard XML Functions• Introduced in ANSI SQL:2003 – Oracle 9iR2 and

10gR2

• Standard functions that can be integrated into queries

• Removes the need for PL/SQL code to create XML documents

175

XML FunctionsXMLELEMENT The basic unit for turning column data into XML

fragments

XMLATTRIBUTES Converts column data into attributes of the parent element

XMLFOREST Allows us to process multiple columns at once

XMLAGG Aggregate separate Fragments into a single fragment

XMLROOT Allows us to place an XML tag at the start of our XML document

176

XMLELEMENT

SELECT XMLELEMENT("name", e.last_name) AS employeeFROM employees eWHERE e.employee_id = 202;

EMPLOYEE------------------------------<name>Fay</name>

177

XMLELEMENT (2)

SELECT XMLELEMENT("employee",XMLELEMENT("works_number", e.employee_id),XMLELEMENT("name", e.last_name)

) AS employeeFROM employees eWHERE e.employee_id = 202;

EMPLOYEE----------------------------------------------------------<employee><works_number>202</works_number><name>Fay</name></employee>

178

XMLATTRIBUTES

SELECT XMLELEMENT("employee",XMLATTRIBUTES(e.employee_id AS "works_number",e.last_name AS "name")

) AS employeeFROM employees eWHERE e.employee_id = 202;

EMPLOYEE----------------------------------------------------------<employee works_number="202" name="Fay"></employee>

179

XMLFOREST

SELECT XMLELEMENT("employee",XMLFOREST(e.employee_id AS "works_number",e.last_name AS "name",e.phone_number AS "phone_number")

) AS employeeFROM employees eWHERE e.employee_id = 202;

EMPLOYEE----------------------------------------------------------<employee><works_number>202</works_number><name>Fay</name><phone_number>603.123.6666</phone_number></employee>

180

XMLFOREST Problem

SELECT XMLELEMENT("employee",XMLFOREST(e.employee_id AS "works_number",e.last_name AS "name",e.phone_number AS "phone_number")

) AS employeeFROM employees eWHERE e.employee_id in (202, 203);

EMPLOYEE----------------------------------------------------------<employee><works_number>202</works_number><name>Fay</name><phone_number>603.123.6666</phone_number></employee><employee><works_number>203</works_number><name>Mavris</name><phone_number>515.123.7777</phone_number></employee>

2 row selected.

181

XMLAGG

SELECT XMLAGG(XMLELEMENT("employee",

XMLFOREST(e.employee_id AS "works_number",e.last_name AS "name",e.phone_number AS "phone_number")

)) AS employeeFROM employees eWHERE e.employee_id in (202, 203);

EMPLOYEE----------------------------------------------------------<employee><works_number>202</works_number><name>Fay</name><phone_number>603.123.6666</phone_number></employee><employee><works_number>203</works_number><name>Mavris</name><phone_number>515.123.7777</phone_number></employee>

1 row selected.

182

XMLROOT

• Creating a well formed XML document

SELECT XMLROOT (XMLELEMENT("employees",XMLAGG(XMLELEMENT("employee",

XMLFOREST(e.employee_id AS "works_number",e.last_name AS "name",e.phone_number AS "phone_number")

))), VERSION '1.0') AS employeeFROM employees eWHERE e.employee_id in (202, 203);

183

XMLROOT

• Well formed, version bound, beatified XML:

EMPLOYEE------------------------------------------<?xml version="1.0"?><employees><employee><works_number>202</works_number><name>Fay</name><phone_number>603.123.6666</phone_number>

</employee><employee><works_number>203</works_number><name>Mavris</name><phone_number>515.123.7777</phone_number>

</employee></employees>

184

Using XQuery• Using the XQuery language we can create, read

and manipulate XML documents

• Two main functions: XMLQuery and XMLTable

• XQuery is about sequences - XQuery is a general sequence-manipulation language

• Each sequence can contain numbers, strings, Booleans, dates, or other XML fragments

185

Creating XML Document using XQuery

SELECT warehouse_name,EXTRACTVALUE(warehouse_spec, '/Warehouse/Area'),XMLQuery(

'for $i in /Warehousewhere $i/Area > 50000return <Details>

<Docks num="{$i/Docks}"/><Rail>

{if ($i/RailAccess = "Y") then "true" else

"false"}

</Rail></Details>' PASSING warehouse_spec RETURNING CONTENT)

"Big_warehouses"FROM warehouses;

186

Creating XML Document using XQuery

WAREHOUSE_ID Area Big_warehouses------------ --------- --------------------------------------------------------

1 250002 500003 85700 <Details><Docks></Docks><Rail>false</Rail></Details>4 103000 <Details><Docks num="3"></Docks><Rail>true</Rail></Details>

. . .

187

Example: Using XMLTable to Read XML

SELECT lines.lineitem, lines.description, lines.partid,lines.unitprice, lines.quantity

FROM purchaseorder,XMLTable('for $i in /PurchaseOrder/LineItems/LineItem

where $i/@ItemNumber >= 8and $i/Part/@UnitPrice > 50and $i/Part/@Quantity > 2

return $i'PASSING OBJECT_VALUECOLUMNS lineitem NUMBER PATH '@ItemNumber',

description VARCHAR2(30) PATH 'Description',partid NUMBER PATH 'Part/@Id',unitprice NUMBER PATH 'Part/@UnitPrice',quantity NUMBER PATH 'Part/@Quantity')

lines;

Oracle 12c JSON Support

189

What is JSON• JavaScript Object Notation

• Converts database tables to a readable document – just like XML but simpler

• Very common in NoSQL and Big Data solutions {"FirstName" : "Zohar","LastName" : "Elkayam","Age" : 36,"Connection" :[

{"Type" : “Email", "Value" : "zohar@DBAces.com"},{"Type" : “Twitter", "Value" : “@realmgic"},{"Type" : "Site", "Value" : "www.realdbamagic.com"},

]}

190

JSON Benefits• Ability to store data without requiring a Schema

– Store semi-structured data in its native (aggregated) form

• Ability to query data without knowledge of Schema

• Ability to index data with knowledge of Schema

191

Oracle JSON Support• Oracle supports JSON since version 12.1.0.2

• JSON documents stored in the database using existing data types: VARCHAR2, CLOB or BLOB

• External JSON data sources accessible through external tables including HDFS

• Data accessible via REST API

192

REST based API for JSON documents• Simple well understood model

• CRUD operations are mapped to HTTP Verbs

– Create / Update : PUT / POST

– Retrieve : GET

– Delete : DELETE

– QBE, Bulk Update, Utilitiy functions : POST

• Stateless

193

JSON Path Expression• Similar role to XPATH in XML

• Syntactically similar to Java Script (. and [ ])

• Compatible with Java Script

194

Common JSON SQL Functions• There are few common JSON Operators:

JSON_EXISTS Checks if a value exists in the JSON

JSON_VALUE Retrieve a scalar value from JSON

JSON_QUERY Query a string from JSON Document

JSON_TABLE Query data from JSON Document (like XMLTable)

195

JSON_QUERY• Extract JSON fragment from JSON document

select count(*)from J_PURCHASEORDER

where JSON_EXISTS(PO_DOCUMENT, '$.ShippingInstructions.Address.state‘)

/

196

Using JSON_TABLE

• Generate rows from a JSON Array

• Pivot properties / key values into columns

• Use Nested Path clause to process multi-level collections with a single JSON_TABLE operator.

197

Example: JSON_TABLE• 1 Row of output for each row in table

select M.*from J_PURCHASEORDER p,

JSON_TABLE(p.PO_DOCUMENT,'$'columnsPO_NUMBER NUMBER(10) path '$.PONumber',REFERENCE VARCHAR2(30 CHAR) path '$.Reference',REQUESTOR VARCHAR2(32 CHAR) path '$.Requestor',USERID VARCHAR2(10 CHAR) path '$.User',COSTCENTER VARCHAR2(16) path '$.CostCenter'

) Mwhere PO_NUMBER > 1600 and PO_Number < 1605/

198

Example: JSON_TABLE (2)• 1 row output for each member of LineItems array

select D.*from J_PURCHASEORDER p,

JSON_TABLE(p.PO_DOCUMENT,'$'columns(PO_NUMBER NUMBER(10) path '$.PONumber',NESTED PATH '$.LineItems[*]'columns(ITEMNO NUMBER(16) path '$.ItemNumber',UPCCODE VARCHAR2(14 CHAR) path '$.Part.UPCCode‘ ))

) Dwhere PO_NUMBER = 1600 or PO_NUMBER = 1601/

199

JSON Indexing• Known Query Patterns : JSON Path expression

– Functional indexes using JSON_VALUE and, JSON_EXISTS

– Materialized View using JSON_TABLE()

• Ad-hoc Query Strategy

– Based on Oracle’s full text index (Oracle Text)

– Support ad-hoc path, value and keyword query search using JSON Path expressions

200

JSON in 12.2.0.1• JSON in 12cR1 used to work with JSON documents

stored in the database

• 12cR2 brought the ability to create and modify JSON:– JSON_object

– JSON_objectagg

– JSON_array

– JSON_arrayagg

201

JSON Creation Exampleselect json_object (

'department' value d.department_name,'employees' value json_arrayagg (

json_object ('name' value first_name || ',' || last_name, 'job' value job_title )))

from hr.departments d, hr.employees e, hr.jobs jwhere d.department_id = e.department_idand e.job_id = j.job_idgroup by d.department_name;

Oracle 12c (12.1 and 12.2)New Features

203

Object Names Length (12.2)• Up to Oracle 12cR2, objects name length (tables,

columns, indexes, constraints etc.) were limited to 30 chars

• Starting Oracle 12cR2, length is now limited to 128 bytes

create table with_a_really_really_really_really_really_long_name (and_lots_and_lots_and_lots_and_lots_and_lots_of int,really_really_really_really_really_long_columns int

);

204

Verify Data Type Conversions (12.2)• If we try to validate using regular conversion we

might hit an error: ORA-01858: a non-numeric character was found where a numeric

was expected

• Use validate_conversion to validate the data without an error

select t.*from dodgy_dates twhere validate_conversion(is_this_a_date as date) = 1;

select t.*from dodgy_dates twhere validate_conversion(is_this_a_date as date, 'yyyymmdd') = 1;

205

Handle Casting Conversion Errors (12.2)• Let’s say we convert the value of a column using

cast. What happens if some of the values doesn’t fit?

• The cast function can now handle conversion errors:select cast (

'not a date' as datedefault date'0001-01-01' on conversion error

) dtfrom dual;

206

Approximate Query (12.1)• APPROX_COUNT_DISTINCT returns the

approximate number of rows that contain distinct values of expression

• This gives better performance but might not return the exact result

• Very good for large sets where exact values aren’t significant

• Adjustable using ERROR_RATE and CONFIDENCE parametes

207

Approximate Query Performance

208

Approximate Query Enhancements (12.2)• 12.2 introduced a parameter, approx_for_count_distinct

which automatically replace count distinct with APPROX_COUNT_DISTINCT

• New approximate function: approx_percentile

approx_percentile (<expression> [ deterministic ],[ ('ERROR_RATE' | 'CONFIDENCE') ]

) within group ( order by <expression>)

SQLcl Introduction

The Next Generation of SQL*Plus?

210

SQL*Plus• Introduced in Oracle 5 (1985)

• Looks very simple but has tight integration with other Oracle infrastructure and tools

• Very good for reporting, scripting, and automation

• Replaced old CLI tool called …

UFI (“User Friendly Interface”)

211

What’s Wrong With SQL*Plus?• Nothing really wrong with SQL*Plus – it is being

updated constantly but it is missing a lot of functionality

• SQL*Plus forces us to use GUI tools to complete some basic tasks

• Easy to understand, a bit hard to use

• Not easy for new users or developers

212

Using SQL Developer• SQL Developer is a free GUI tool to handle common

database operations• Comes with Oracle client installation starting Oracle

11g• Good for development and management of databases

– Developer mode– DBA mode– Modeling mode

• Has a Command Line interface (SDCLI) – but it’s not interactive

213

SQL Developer Command Line (SQLcl)• The SQL Developer Command Line (SQLcl, priv.

SDSQL) is a new command line interface (CLI) for SQL developers, report users, and DBAs

• It is part of the SQL Developer suite – developed by the same team: Oracle Database Development Tools Team

• Does (or will do) most of what SQL*Plus can do, and much more

• Main focus: making life easier for CLI users• Minimal installation, minimal requirements

214

Current Status (November 2016)• Production as of September 2016

– current version: 4.2.0.16.308.0750, November 3, 2016

• New version comes out every couple of months

– Adding support for existing SQL*Plus commands/syntax

– Adding new commands and functionality

• The team is accepting bug reports and enhancement requestsfrom the public

• Active community on OTN forums!

215

Prerequisites• Very small footprint: 16 MB

• Tool is Java based so it can run on Windows, Linux, and OS/X

• Java 7/8 JRE (runtime environment - no need for JDK)

• No need for installer or setup

• No need for any other additional software or special license

• No need for an Oracle Client

216

Installing• Download from: SQL Developer Command Line

OTN Page

• Unzip the file

• Run it

217

Running SQLcl

What Can It Do?

219

Connecting to the Database• When no Oracle Client - using thin connection:

EZConnect connect style out of the box

connect host:port/service

• Support TNS, Thick and LDAP connection when Oracle home detected

• Auto-complete connection strings from last connections AND tnsnames.ora

220

Object Completion and Easy Edit• Use the tab key to complete commands

• Can be used to list tables, views or other queriableobjects

• Can be used to replace the * with actual column names

• Use the arrow keys to move around the command

• Use CTRL+W and CTRL+S to jump to the beginning/end of commands

221

Command History• 100 command history buffer• Commands are persistent between sessions (watch out for

security!)• Use UP and DOWN arrow keys to access old commands• Usage:

historyhistory usageHistory scripthistory fullHistory clear [session?]

• Load from history into command buffer:history <number>

222

Describe, Information and Info+• Describe lists the column of the tables just like

SQL*Plus

• Information shows column names, default values, indexes and constraints.

• In 12c database information shows table statistics and In memory status

• Works for table, views, sequences, and code objects

• Info+ shows additional information regarding column statistics and column histograms

223

SHOW ALL and SHOW ALL+• The show all command is familiar from SQL*Plus –

it will show all the parameters for the SQL*Plus settings

• The show all+ command will show the show all command and some perks: available tns entries, list of pdbs, connection settings, instance settings, nls settings, and more!

224

Pretty Input• Using the SQL Developer formatting rules, it will

change our input into well formatted commands.

• Use the SQLFORMATPATH to point to the SQL Developer rule file (XML)

SQL> select * from dual;

D-X

SQL> format buffer;1 SELECT2 *3 FROM4* dual

225

SQL*Plus Output• SQL*Plus output is generated as text tables

• We can output the data as HTML but the will take over everything we do in SQL*Plus (i.e. describe command)

• We can’t use colors in our output

• We can’t generate other types of useful outputs (CSV is really hard for example)

226

Generating Pretty Output• Outputting query results becomes easier with the “set

sqlformat” command (also available in SQL Developer)

• We can create a query in the “regular” way and then switch between the different output styles:– ANSIConsole

– Fixed column size output

– XML or JSON output

– HTML output generates a built in search field and a responsive html output for the result only

227

Generating Other Useful Outputs• We can generate loader ready output (with “|” as

a delimiter)

• We can generate insert commands

• We can easily generate CSV output

• Usage:set sqlformat { csv,html,xml,json,ansiconsole,insert,loader,fixed,default}

228

Load Data From CSV File• Loads a comma separated value (csv) file into a

table

• The first row of the file must be a header row and the file must be encoded UTF8

• The load is processed with 50 rows per batch

• Usage:LOAD [schema.]table_name[@db_link] file_name

229

SCRIPT – Client Side Scripting• SQLcl exposes JavaScript scripting with nashorn to

make things very scriptable on the client side

• This means we can create our own commands inside SQLcl using JavaScript

• Kris Rice’s from the development team published multiple example on his blog http://krisrice.blogspot.com/ and in GitHub, for example the autocorrect example demo.

230

Summary• There is a lot in SQL than meets the eye

• Wise use of analytic queries can be good for readability and performance

• Recursive queries are good replacement for the old connect by prior but a little dangerous

• Oracle 12c features are really cool!

• Look out for SQLcl: it’s cool and it’s going places!

231

What Did We Not Talk About?• The Model clause

• Adding PL/SQL to our SQL (Oracle 12c)

• Hints and other tuning considerations

• The SQL reference book is 1906 pages long. We didn’t talk about most of it…

Q&AAny Questions? Now will be the time!

Zohar Elkayamtwitter: @realmgicZohar@Brillix.co.il

www.ilDBA.co.ilwww.realdbamagic.com

234