Still using Windows 3.1?...‣Literate SQL Organize SQL code to improve maintainability ‣Assign...

Still using Windows 3.1? So why stick with

SQL-92?

@ModernSQL - https://modern-sql.com/ @MarkusWinand

SQL:1999

WITH (Common Table Expressions)

Understand

this first

WITH (non-recursive) The ProblemNested queries are hard to read:

SELECT…FROM(SELECT…FROMt1JOIN(SELECT…FROM…)aON(…))bJOIN(SELECT…FROM…)cON(…)

Then this...



Finally the first line makes sense



CTEs are statement-scoped views:

WITHa(c1,c2,c3)AS(SELECTc1,c2,c3FROM…),

b(c4,…)AS(SELECTc4,…FROMt1JOINaON(…)),

c(…)

Keyword

WITH (non-recursive) Since SQL:1999




c(…)

Name of CTE and (here optional) column names





c(…)

Definition





c(…)

Introduces another CTE

Don't repeat WITH





c(…)





c(…)

May refer toprevious CTEs




c(…)AS(SELECT…FROM…)

SELECT…FROMbJOINcON(…)

Third CTE






No comma!






Main query







Read top down


‣ Literate SQL

Organize SQL code toimprove maintainability

‣ Assign column names

to tables produced by valuesor unnest.

‣ Overload tables (for testing)

with queries hide tablesof the same name.

Use-CasesWITH (non-recursive)

https://modern-sql.com/use-case/literate-sql

https://modern-sql.com/use-case/naming-unnamed-columns

https://modern-sql.com/use-case/unit-tests-on-transient-data

http://modern-sql.com/use-case/literate-sql

http://modern-sql.com/use-case/naming-unnamed-columns

http://modern-sql.com/use-case/unit-tests-on-transient-data

WITH are the "private methods" of SQL

WITH is a prefix to SELECT

WITH queries are only visible in the SELECT they precede

WITH in detail: https://modern-sql.com/feature/with

WITH (non-recursive) In a Nutshell

http://modern-sql.com/feature/with

AvailabilityWITH (non-recursive)19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2 MariaDB8.0 MySQL

8.4 PostgreSQL3.8.3[0] SQLite

7.0 DB2 LUW9iR2 Oracle

2005[1] SQL Server[0]Only for top-level SELECT statements[1]Only allowed at the very begin of a statement. E.g. WITH...INSERT...SELECT.

WITHRECURSIVE (Common Table Expressions)

(This page is intentionally left blank)

WITHRECURSIVE The Problem

CREATETABLEt(idNUMERICNOTNULL,parent_idNUMERIC,…PRIMARYKEY(id))

Coping with hierarchies in the Adjacency List Model[0]


[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”

SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)

Coping with hierarchies in the Adjacency List Model[0]


WHEREd0.id=?

[0] Hierarchies implemented using a “parent id” — see “Joe Celko’s Trees and Hierarchies in SQL for Smarties”

SELECT*FROMtASd0LEFTJOINtASd1ON(d1.parent_id=d0.id)LEFTJOINtASd2ON(d2.parent_id=d1.id)

WITHRECURSIVE Since SQL:1999

WHEREd0.id=?

WITHRECURSIVEd(id,parent,…)AS(SELECTid,parent,…FROMtblWHEREid=?UNIONALLSELECTid,parent,…FROMdJOINtblON(tbl.parent=d.id))SELECT*FROMsubtree

Recursive common table expressions may refer to themselves in a leg of a UNION[ALL]:

WITHRECURSIVEcte(n)AS(SELECT1UNIONALLSELECTn+1FROMcteWHEREn<3)SELECT*FROMcte

Keyword

Since SQL:1999WITHRECURSIVE



Column list mandatory here




Executed first




Result sent there




Result visible twice




n---123(3rows)

Once it becomes

part of the final result




n---123(3rows)

Second leg of UNION is executed




n---123(3rows)

Result sent there again




n---123(3rows)

It's a loop!




n---123(3rows)

n=3 doesn't matchLoop terminates


AvailabilityWITHRECURSIVE19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17


8.4 PostgreSQL3.8.3[0] SQLite

7.0 DB2 LUW11gR2 Oracle

2005 SQL Server[0]Only for top-level SELECT statements

GROUPINGSETS

Only one GROUPBY operation at a time:

GROUPINGSETS Before SQL:1999

SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month

Monthly revenue Yearly revenue

SELECTyear,sum(revenue)FROMtblGROUPBYyear

GROUPINGSETS Before SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month


GROUPINGSETS Before SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month


UNIONALL

,null

GROUPINGSETS Since SQL:1999SELECTyear,month,sum(revenue)FROMtblGROUPBYyear,month


UNIONALL

,null

SELECTyear,month,sum(revenue)FROMtblGROUPBYGROUPINGSETS((year,month),(year))

GROUPINGSETS are multiple GROUPBYs in one go

() (empty parenthesis) build a group over all rows

GROUPING (function) disambiguates the meaning of NULL(was the grouped data NULL or is this column not currently grouped?)

Permutations can be created using ROLLUP and CUBE(ROLLUP(a,b,c) = GROUPINGSETS((a,b,c),(a,b),(a),())

GROUPINGSETS In a Nutshell

GROUPINGSETS Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1[0] MariaDB5.0[1] MySQL

9.5 PostgreSQLSQLite

5 DB2 LUW9iR1 Oracle

2008 SQL Server[0]Only ROLLUP (properitery syntax).[1]Only ROLLUP (properitery syntax). GROUPING function since MySQL 8.0.

SQL:2003

FILTER

FILTER Before we start

In SQL, most aggregate functions* drop null arguments

prior to the aggregation.

*Exceptions: Aggregate functions that return structured data: array_agg, json_objectagg, json_arrayagg, xmlagg

See: http://modern-sql.com/concept/null#aggregates

http://modern-sql.com/concept/null#aggregates

SELECTYEAR,SUM(CASEWHENMONTH=1THENrevenueELSE0END)JAN,SUM(CASEWHENMONTH=2THENrevenueEND)FEB,…FROMsalesGROUPBYYEAR

FILTER The ProblemPivot table: Years on the Y axis, month on X:



Optional:ELSE null is default

SELECTYEAR,SUM(revenue)FILTER(WHEREMONTH=1)JAN,SUM(revenue)FILTER(WHEREMONTH=2)FEB,…FROMsalesGROUPBYYEAR;

FILTER Since SQL:2003SQL:2003 allows FILTER(WHERE…) after aggregates:

FILTER Since SQL:2003

Year

2016

2016

2016

2016

2016

Month

1

2

3

...

12

Revenue

1

23

345

...

1234

Year

2016

Jan

1

Feb

23

Mar

345

...

...

Dec

1234

SUM(…) FILTE

R(WHERE …)

SUM(…)

FILTER(WHER

E month=2)

SUM(rev

enue) FILTER(WHERE

month=3)

SUM(rev

enue) FILTER(WHERE mo

nth=…)

SUM(rev

enue) FILTER(WHERE month=

12)

Pivot in SQL1. Use GROUP BY

to combine rows2. Use FILTER to pick

rows per column

See: http://modern-sql.com/use-case/pivot

http://modern-sql.com/use-case/pivot

FILTER Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 MariaDBMySQL


DB2 LUWOracleSQL Server

BOOLEANTests

Before we start

SQL uses a three-valued logic. Boolean values are either

true, false or unknown(=null).

See: http://modern-sql.com/concept/three-valued-logic

BOOLEANAggregates

http://modern-sql.com/concept/three-valued-logic

BOOLEANTests

Similar to isnull, there are tests for each Boolean value(of which there are three: true, false, unknown/null)

IS[NOT][TRUE|FALSE|UNKNOWN]

Since SQL:2003

CREATETABLEprices(…valid_fromDATENOTNULL,valid_toDATE,--null:openend…CHECK(valid_from<valid_to),);

WHEREvalid_from<CURRENT_DATEAND(valid_to<=CURRENT_DATE)ISNOTFALSE

BOOLEANTests Since SQL:2003




UNKNOWN ifVALID_TO is NULL




UNKNOWN ifVALID_TO is NULL Takes TRUE and

UNKNOWN


9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 MariaDB5.0.51a MySQL8.3 PostgreSQL3.5.7[0] SQLite

DB2 LUWOracleSQL Server

[0]No IS [NOT] UNKNOWN. Use IS [NOT] NULL instead.

OVER and

PARTITIONBY

OVER (PARTITION BY) The ProblemTwo distinct concepts could not be used independently:

‣Merge rows with the same key properties

‣ GROUPBY to specify key properties

‣ DISTINCT to use full row as key

‣ Aggregate data from related rows ‣ Requires GROUPBY to segregate the rows

‣ COUNT, SUM, AVG, MIN, MAX to aggregate grouped rows

SELECTc1,SUM(c2)totFROMtGROUPBYc1

OVER (PARTITION BY) The Problem

Yes ⇠

Mer

ge ro

ws ⇢

No

No ⇠ Aggregate ⇢ Yes

SELECTc1,c2FROMt

SELECTDISTINCTc1,c2FROMt

SELECTc1,c2FROMtJOIN()taON(t.c1=ta.c1)


,tot


OVER (PARTITION BY) Since SQL:2003

Yes ⇠

Mer

ge ro

ws ⇢

No

No ⇠ Aggregate ⇢ Yes

SELECTc1,c2FROMt

SELECTDISTINCTc1,c2FROMt

SELECTc1,c2FROMt

FROMt

,SUM(c2)OVER(PARTITIONBYc1)

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 600022 1000 600022 1000 6000333 1000 6000333 1000 6000333 1000 6000

OVER (PARTITION BY) How it works

SELECTdep,salary,SUM(salary)OVER()FROMemp

dep salary ts1 1000 100022 1000 200022 1000 2000333 1000 3000333 1000 3000333 1000 3000

OVER (PARTITION BY) How it works

)PARTITIONBYdep

OVER and

ORDERBY(Framing & Ranking)

acnt id value balance

1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,FROMtransactionst


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

OVER (ORDER BY) The Problem

SELECTid,value,

(SELECTSUM(value)FROMtransactionst2WHEREt2.id<=t.id)

FROMtransactionst

Range segregation (<=)not possible with

GROUP BY orPARTITION BY

OVER (ORDER BY) Since SQL:2003

SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYid


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDING


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +30

22 3 -10 +20

333 4 +50 +70

333 5 -30 +40

333 6 -20 +20

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW


SELECTid,value,

FROMtransactionst

SUM(value)OVER(

)


1 1 +10 +10

22 2 +20 +20

22 3 -10 +10

333 4 +50 +50

333 5 -30 +20

333 6 -20 .0

ORDERBYidROWSBETWEENUNBOUNDEDPRECEDINGANDCURRENTROW

PARTITIONBYacnt

OVER (ORDER BY) Since SQL:2003With OVER(ORDERBYn) a new type of functions make sense:

n ROW_NUMBER RANK DENSE_RANK PERCENT_RANK CUME_DIST1 1 1 1 0 0.252 2 2 2 0.33… 0.753 3 2 2 0.33… 0.754 4 4 3 1 1

‣ Aggregates without GROUPBY

‣ Running totals, moving averages

‣ Ranking‣ Top-N per Group

‣ Avoiding self-joins

[… many more …]

Use Cases

SELECT*FROM(SELECTROW_NUMBER()OVER(PARTITIONBY…ORDERBY…)rn,t.*FROMt)numbered_tWHERErn<=3

AVG(…)OVER(ORDERBY…ROWSBETWEEN3PRECEDINGAND3FOLLOWING)moving_avg

OVER (SQL:2003)

OVER may follow any aggregate function

OVER defines which rows are visible at each row

OVER() makes all rows visible at every row

OVER(PARTITIONBY …) segregates like GROUPBY

OVER(ORDERBY…BETWEEN) segregates using <, >

In a NutshellOVER (SQL:2003)

OVER (SQL:2003) Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17



7.0 DB2 LUW8i Oracle

2005 SQL Server

OVER (SQL:2003) Availability19

9920

0120

0320

0520

0720

0920

1120

1320

1520

17



7.0 DB2 LUW8i Oracle

2005 SQL Server

Hive

ImpalaSpark

NuoDB

SQL:2011

WITHnumbered_tAS(SELECT*)

SELECTcurr.*,curr.balance-COALESCE(prev.balance,0)FROMnumbered_tcurrLEFTJOINnumbered_tprevON(curr.rn=prev.rn+1)

OVER (SQL:2011) The ProblemDirect access of other rows of the same window is not possible.

(E.g., calculate the difference to the previous rows)

currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt





currbalance … rn

50 … 190 … 270 … 330 … 4

FROMt,ROW_NUMBER()OVER(ORDERBYx)rn





currbalance … rn

50 … 190 … 270 … 330 … 4


prevbalance … rn

50 … 190 … 270 … 330 … 4





currbalance … rn

50 … 190 … 270 … 330 … 4


prevbalance … rn

50 … 190 … 270 … 330 … 4

+50+40-20-40

SELECT*,balance-COALESCE(LAG(balance)OVER(ORDERBYx),0)FROMt

Available functions:LEAD/LAGFIRST_VALUE/LAST_VALUENTH_VALUE(col,n)FROMFIRST/LASTRESPECT/IGNORENULLS

OVER (SQL:2011) Since SQL:2011SQL:2011 introduced LEAD, LAG, NTH_VALUE, … for that:

OVER (LEAD, LAG, …) Since SQL:201119

9920

0120

0320

0520

0720

0920

1120

1320

1520

17

5.1 10.2[0] MariaDB8.0[0] MySQL

8.4[0] PostgreSQLSQLite

9.5[1] 11.1 DB2 LUW8i[1] 11gR2 Oracle

2012[1] SQL Server[0]No IGNORENULLS and FROMLAST[1]No NTH_VALUE

System Versioning (Time Traveling)

INSERTUPDATEDELETE

are DESTRUCTIVE

System Versioning The Problem

ID Data start_ts end_ts1 X 10:00:00

UPDATE...SETDATA='Y'...

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00

DELETE...WHEREID=1

INSERT...(ID,DATA)VALUES(1,'X')

System Versioning Since SQL:2011

ID Data start_ts end_ts1 X 10:00:00

UPDATE...SETDATA='Y'...

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00

DELETE...WHEREID=1

ID Data start_ts end_ts1 X 10:00:00 11:00:001 Y 11:00:00 12:00:00

INSERT...(ID,DATA)VALUES(1,'X')


Although multiple versions exist, only the “current” one is visible per default.

After 12:00:00, SELECT*FROMt doesn’t return anything anymore.




With FOR…ASOF you can query anything you like: SELECT*FROMtFORSYSTEM_TIMEASOFTIMESTAMP'2015-04-0210:30:00'

ID Data start_ts end_ts

1 X 10:00:00 11:00:00


1999

2001

2003

2005

2007

2009

2011

2013

2015

2017

5.1 MariaDB[0]

MySQLPostgreSQLSQLite

10.1[1] DB2 LUW10gR1[2] Oracle

2016 SQL Server[0]Available in MariaDB 10.3 beta.[1]Third column required (tx id), history table required.[2]Functionality available using Flashback


SQL:2016 (released: 2016-12-15)

New in SQL:2016 JSON

ROW PATTERN MATCHING https://www.slideshare.net/MarkusWinand/row-pattern-matching-in-sql2016

DATE FORMAT

POLYMORPHIC TABLE FUNCTIONS ➔ https://modern-sql.com/blog/2017-06/whats-new-in-sql-2016

SQL has evolved beyond the relational idea.

Modern SQL? @MarkusWinand


If you are using SQL like 25 years ago,you are doing it wrong!



If you are using SQL like 25 years ago,you are doing it wrong!

Don’t think SQL cannot do this or that.Act professionally: check it out before complaining.


Act professionally: check it out

About @MarkusWinand

‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online

‣SQL Tuning ‣ Index-Redesign ‣ Query Improvements ‣ On-Site or Online

https://winand.at/

About @MarkusWinand€0,-

€10-30‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online


sql-performance-explained.com

About @MarkusWinand@ModernSQL

https://modern-sql.com‣Training for Developers ‣ SQL Performance (Indexing) ‣ Modern SQL ‣ On-Site or Online


Still using Windows 3.1?...‣Literate SQL Organize SQL code to improve maintainability ‣Assign...

Documents

Transcript of Still using Windows 3.1?...‣Literate SQL Organize SQL code to improve maintainability ‣Assign...