Still using Windows 3.1? So why stick with

SQL-92?

@ModernSQL - https://modern-sql.com/ @MarkusWinand

SQL:1999

WITH (Common Table Expressions)

Understand

this first

WITH (non-recursive) The ProblemNested queries are hard to read:

SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)

Then this...

WITH (non-recursive) The ProblemNested queries are hard to read:

SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)

Then this...

WITH (non-recursive) The ProblemNested queries are hard to read:

SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)

Finally the first line makes sense

WITH (non-recursive) The ProblemNested queries are hard to read:

SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)

CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

Keyword

WITH (non-recursive) Since SQL:1999

CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

Name of CTE and (here optional) column names

WITH (non-recursive) Since SQL:1999

CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

Definition

WITH (non-recursive) Since SQL:1999

CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

Introduces another CTE

Don't repeat WITH

WITH (non-recursive) Since SQL:1999

CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

WITH (non-recursive) Since SQL:1999

CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

May refer toprevious CTEs

WITH (non-recursive) Since SQL:1999

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)AS(SELECT…FROM…)

SELECT…FROMbJOINcON(…)

Third CTE

WITH (non-recursive) Since SQL:1999

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)AS(SELECT…FROM…)

SELECT…FROMbJOINcON(…)

No comma!

WITH (non-recursive) Since SQL:1999

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)AS(SELECT…FROM…)

SELECT…FROMbJOINcON(…)

Main query

WITH (non-recursive) Since SQL:1999

CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)AS(SELECT…FROM…)

SELECT…FROMbJOINcON(…)

Read top down

WITH (non-recursive) Since SQL:1999

‣ Literate SQL

Organize SQL code toimprove maintainability

‣ Assign column names

to tables produced by valuesor unnest.

‣ Overload tables (for testing)

with queries hide tablesof the same name.

Use-CasesWITH (non-recursive)

https://modern-sql.com/use-case/literate-sql

https://modern-sql.com/use-case/naming-unnamed-columns

https://modern-sql.com/use-case/unit-tests-on-transient-data

WITH are the "private methods" of SQL

WITH is a prefix to SELECT

WITH queries are only visible in the SELECT they precede

WITH in detail: https://modern-sql.com/feature/with

WITH (non-recursive) In a Nutshell

AvailabilityWITH (non-recursive)19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2 MariaDB8.0 MySQL

8.4 PostgreSQL3.8.3[0] SQLite

7.0 DB2 LUW9iR2 Oracle

2005[1] SQL Server[0]Only for top-level SELECT statements[1]Only allowed at the very begin of a statement. E.g. WITH...INSERT...SELECT.

WITHRECURSIVE (Common Table Expressions)

(This page is intentionally left blank)

WITHRECURSIVE The Problem

CREATETABLEt(idNUMERICNOTNULL,parent_idNUMERIC,…PRIMARYKEY(id))

Coping with hierarchies in the Adjacency List Model[0]

WITHRECURSIVE The Problem

[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”

SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)

Coping with hierarchies in the Adjacency List Model[0]

WITHRECURSIVE The Problem

WHEREd0.id=?

[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”

SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)

Coping with hierarchies in the Adjacency List Model[0]

WITHRECURSIVE The Problem

WHEREd0.id=?

[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”

SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)

Coping with hierarchies in the Adjacency List Model[0]

WITHRECURSIVE The Problem

WHEREd0.id=?

[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”

SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)

WITHRECURSIVE Since SQL:1999

WHEREd0.id=?

WITHRECURSIVEd(id,parent,…)AS(SELECTid,parent,…FROMtblWHEREid=?UNIONALLSELECTid,parent,…FROMdJOINtblON(tbl.parent=d.id))SELECT*FROMsubtree

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

Keyword

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

Column list mandatory here

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

Executed first

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

Result sent there

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

Result visible twice

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

n---123(3rows)

Once it becomes

part of the final result

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

n---123(3rows)

Second leg of UNION is executed

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

n---123(3rows)

Result sent there again

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

n---123(3rows)

It's a loop!

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

n---123(3rows)

It's a loop!

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

n---123(3rows)

It's a loop!

Since SQL:1999WITHRECURSIVE

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

n---123(3rows)

n=3 doesn't matchLoop terminates

Since SQL:1999WITHRECURSIVE

AvailabilityWITHRECURSIVE19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2 MariaDB8.0 MySQL

8.4 PostgreSQL3.8.3[0] SQLite

7.0 DB2 LUW11gR2 Oracle

2005 SQL Server[0]Only for top-level SELECT statements

GROUPINGSETS

Only one GROUPBY operation at a time:

GROUPINGSETS Before SQL:1999

SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

Monthly revenue Yearly revenue

SELECTyear,sum(revenue)FROMtblGROUPBYyear

GROUPINGSETS Before SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

SELECTyear,sum(revenue)FROMtblGROUPBYyear

GROUPINGSETS Before SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

SELECTyear,sum(revenue)FROMtblGROUPBYyear

UNIONALL

,null

GROUPINGSETS Since SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

SELECTyear,sum(revenue)FROMtblGROUPBYyear

UNIONALL

,null

SELECTyear,month,sum(revenue)FROMtblGROUPBYGROUPINGSETS((year,month),(year))

GROUPINGSETS are multiple GROUPBYs in one go

() (empty parenthesis) build a group over all rows

GROUPING (function) disambiguates the meaning of NULL(was the grouped data NULL or is this column not currently grouped?)

Permutations can be created using ROLLUP and CUBE(ROLLUP(a,b,c) = GROUPINGSETS((a,b,c),(a,b),(a),())

GROUPINGSETS In a Nutshell

GROUPINGSETS Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1[0] MariaDB5.0[1] MySQL

9.5 PostgreSQLSQLite

5 DB2 LUW9iR1 Oracle

2008 SQL Server[0]Only ROLLUP (properitery syntax).[1]Only ROLLUP (properitery syntax). GROUPING function since MySQL 8.0.

SQL:2003

FILTER

FILTER Before we start

In SQL, most aggregate functions* drop null arguments

prior to the aggregation.

*Exceptions: Aggregate functions that return structured data: array_agg, json_objectagg, json_arrayagg, xmlagg

See: http://modern-sql.com/concept/null#aggregates

SELECTYEAR,SUM(CASEWHENMONTH=1THENrevenueELSE0END)JAN,SUM(CASEWHENMONTH=2THENrevenueEND)FEB,…FROMsalesGROUPBYYEAR

FILTER The ProblemPivot table: Years on the Y axis, month on X:

SELECTYEAR,SUM(CASEWHENMONTH=1THENrevenueELSE0END)JAN,SUM(CASEWHENMONTH=2THENrevenueEND)FEB,…FROMsalesGROUPBYYEAR

FILTER The ProblemPivot table: Years on the Y axis, month on X:

Optional:ELSE null is default

SELECTYEAR,SUM(CASEWHENMONTH=1THENrevenueELSE0END)JAN,SUM(CASEWHENMONTH=2THENrevenueEND)FEB,…FROMsalesGROUPBYYEAR

FILTER The ProblemPivot table: Years on the Y axis, month on X:

SELECTYEAR,SUM(revenue)FILTER(WHEREMONTH=1)JAN,SUM(revenue)FILTER(WHEREMONTH=2)FEB,…FROMsalesGROUPBYYEAR;

FILTER Since SQL:2003SQL:2003 allows FILTER(WHERE…) after aggregates:

FILTER Since SQL:2003

Year

2016

2016

2016

2016

2016

Month

1

2

3

...

12

Revenue

1

23

345

...

1234

Year

2016

Jan

1

Feb

23

Mar

345

...

...

Dec

1234

SUM(…) FILTE

R(WHERE …)

SUM(…)

FILTER(WHER

E month=2)

SUM(rev

enue) FILTER(WHERE

month=3)

SUM(rev

enue) FILTER(WHERE mo

nth=…)

SUM(rev

enue) FILTER(WHERE month=

12)

Pivot in SQL1. Use GROUP BY

to combine rows2. Use FILTER to pick

rows per column

See: http://modern-sql.com/use-case/pivot

FILTER Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 MariaDBMySQL

9.4 PostgreSQLSQLite

DB2 LUWOracleSQL Server

BOOLEANTests

Before we start

SQL uses a three-valued logic. Boolean values are either

true, false or unknown(=null).

See: http://modern-sql.com/concept/three-valued-logic

BOOLEANAggregates

BOOLEANTests

Similar to isnull, there are tests for each Boolean value(of which there are three: true, false, unknown/null)

IS[NOT][TRUE|FALSE|UNKNOWN]

Since SQL:2003

CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);

WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE

BOOLEANTests Since SQL:2003

CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);

WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE

BOOLEANTests Since SQL:2003

UNKNOWN ifVALID_TO is NULL

CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);

WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE

BOOLEANTests Since SQL:2003

UNKNOWN ifVALID_TO is NULL Takes TRUE and

UNKNOWN

BOOLEANTests Since SQL:200319

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 MariaDB5.0.51a MySQL8.3 PostgreSQL3.5.7[0] SQLite

DB2 LUWOracleSQL Server

[0]No IS [NOT] UNKNOWN. Use IS [NOT] NULL instead.

OVER and

PARTITIONBY

OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:

‣Merge rows with the same key properties

‣ GROUPBY to specify key properties

‣ DISTINCT to use full row as key

‣ Aggregate data from related rows ‣ Requires GROUPBY to segregate the rows

‣ COUNT, SUM, AVG, MIN, MAX to aggregate grouped rows

SELECTc1,SUM(c2)totFROMtGROUPBYc1

OVER (PARTITION BY) The Problem

Yes ⇠

Mer

ge ro

ws ⇢

No

No ⇠ Aggregate ⇢ Yes

SELECTc1,c2FROMt

SELECTDISTINCTc1,c2FROMt

SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)

SELECTc1,SUM(c2)totFROMtGROUPBYc1

,tot

SELECTc1,SUM(c2)totFROMtGROUPBYc1

OVER (PARTITION BY) The Problem

Yes ⇠

Mer

ge ro

ws ⇢

No

No ⇠ Aggregate ⇢ Yes

SELECTc1,c2FROMt

SELECTDISTINCTc1,c2FROMt

SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)

SELECTc1,SUM(c2)totFROMtGROUPBYc1

,tot

SELECTc1,SUM(c2)totFROMtGROUPBYc1

OVER (PARTITION BY) Since SQL:2003

Yes ⇠

Mer

ge ro

ws ⇢

No

No ⇠ Aggregate ⇢ Yes

SELECTc1,c2FROMt

SELECTDISTINCTc1,c2FROMt

SELECTc1,c2FROMt

FROMt

,SUM(c2)OVER(PARTITIONBYc1)

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000

OVER (PARTITION BY) How it works

)PARTITIONBYdep

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000

OVER (PARTITION BY) How it works

)PARTITIONBYdep

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000

OVER (PARTITION BY) How it works

)PARTITIONBYdep

OVER and

ORDERBY(Framing & Ranking)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,FROMtransactionst

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,FROMtransactionst

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,

(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)

FROMtransactionst

Range segregation (<=)not possible with

GROUP BY orPARTITION BY

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYid

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDING

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)

acnt id value balance

1 1 +10 +10

22 2 +20 +20

22 3 -10 +10

333 4 +50 +50

333 5 -30 +20

333 6 -20 .0

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

PARTITIONBYacnt

OVER (ORDER BY) Since SQL:2003With OVER(ORDERBYn) a new type of functions make sense:

n ROW_NUMBER RANK DENSE_RANK PERCENT_RANK CUME_DIST1 1 1 1 0 0.252 2 2 2 0.33… 0.753 3 2 2 0.33… 0.754 4 4 3 1 1

‣ Aggregates without GROUPBY

‣ Running totals, moving averages

‣ Ranking‣ Top-N per Group

‣ Avoiding self-joins

[… many more …]

Use Cases

SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3

AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg

OVER (SQL:2003)

OVER may follow any aggregate function

OVER defines which rows are visible at each row

OVER() makes all rows visible at every row

OVER(PARTITIONBY …) segregates like GROUPBY

OVER(ORDERBY…BETWEEN) segregates using <, >

In a NutshellOVER (SQL:2003)

OVER (SQL:2003) Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2 MariaDB8.0 MySQL

8.4 PostgreSQLSQLite

7.0 DB2 LUW8i Oracle

2005 SQL Server

OVER (SQL:2003) Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2 MariaDB8.0 MySQL

8.4 PostgreSQLSQLite

7.0 DB2 LUW8i Oracle

2005 SQL Server

Hive

ImpalaSpark

NuoDB

SQL:2011

OVER

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn

prevbalance … rn

50 … 190 … 270 … 330 … 4

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn

prevbalance … rn

50 … 190 … 270 … 330 … 4

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn

prevbalance … rn

50 … 190 … 270 … 330 … 4

+50+40-20-40

SELECT*,balance-COALESCE(LAG(balance)OVER(ORDERBYx),0)FROMt

Available functions:LEAD/LAGFIRST_VALUE/LAST_VALUENTH_VALUE(col,n)FROMFIRST/LASTRESPECT/IGNORENULLS

OVER (SQL:2011) Since SQL:2011SQL:2011 introduced LEAD, LAG, NTH_VALUE, … for that:

OVER (LEAD, LAG, …) Since SQL:201119

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2[0] MariaDB8.0[0] MySQL

8.4[0] PostgreSQLSQLite

9.5[1] 11.1 DB2 LUW8i[1] 11gR2 Oracle

2012[1] SQL Server[0]No IGNORENULLS and FROMLAST[1]No NTH_VALUE

System Versioning (Time Traveling)

INSERTUPDATEDELETE

are DESTRUCTIVE

System Versioning The Problem

ID Data start_ts end_ts1 X 10:00:00

UPDATE...SETDATA='Y'...

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00

DELETE...WHEREID=1

INSERT...(ID,DATA)VALUES(1,'X')

System Versioning Since SQL:2011

ID Data start_ts end_ts1 X 10:00:00

UPDATE...SETDATA='Y'...

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00

DELETE...WHEREID=1

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00

INSERT...(ID,DATA)VALUES(1,'X')

System Versioning Since SQL:2011

Although multiple versions exist, only the “current” one is visible per default.

After 12:00:00, SELECT*FROMt doesn’t return anything anymore.

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00

System Versioning Since SQL:2011

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00

With FOR…ASOF you can query anything you like: SELECT*FROMtFORSYSTEM_TIMEASOFTIMESTAMP'2015-04-0210:30:00'

ID Data start_ts end_ts

1 X 10:00:00 11:00:00

System Versioning Since SQL:2011

1999

2001

2003

2005

2007

2009

2011

2013

2015

2017

5.1 MariaDB[0]

MySQLPostgreSQLSQLite

10.1[1] DB2 LUW10gR1[2] Oracle

2016 SQL Server[0]Available in MariaDB 10.3 beta.[1]Third column required (tx id), history table required.[2]Functionality available using Flashback

System Versioning Since SQL:2011

SQL:2016 (released: 2016-12-15)

New in SQL:2016 JSON

ROW PATTERN MATCHING https://www.slideshare.net/MarkusWinand/row-pattern-matching-in-sql2016

DATE FORMAT

POLYMORPHIC TABLE FUNCTIONS ➔ https://modern-sql.com/blog/2017-06/whats-new-in-sql-2016

SQL has evolved beyond the relational idea.

Modern SQL? @MarkusWinand

SQL has evolved beyond the relational idea.

If you are using SQL like 25 years ago,you are doing it wrong!

Modern SQL? @MarkusWinand

SQL has evolved beyond the relational idea.

If you are using SQL like 25 years ago,you are doing it wrong!

Don’t think SQL cannot do this or that.Act professionally: check it out before complaining.

Modern SQL? @MarkusWinand

Act professionally: check it out

SQL has evolved beyond the relational idea.

If you are using SQL like 25 years ago,you are doing it wrong!

Don’t think SQL cannot do this or that.Act professionally: check it out before complaining.

Modern SQL? @MarkusWinand

Act professionally: check it out

About @MarkusWinand

‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online

‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online

https://winand.at/

About @MarkusWinand€0,-

€10-30‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online

‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online

sql-performance-explained.com

About @MarkusWinand@ModernSQL

https://modern-sql.com‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online

‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online