Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out...
Transcript of Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out...
![Page 1: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/1.jpg)
Common Table Expressions in MariaDB 10.2
Sergei PetruniaGalina Shalygina
![Page 2: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/2.jpg)
2
● A standard SQL feature
● Two kinds of CTEs
•Recursive
•Non-recursive
● Supported by Oracle, MS SQL Server, PostgreSQL, SQLite, …
● MySQL world:
•MariaDB: 10.2 Beta (Sept, 27th)
•MySQL: MySQL-8.0.0-labs-optimizer tree
Common Table Expressions
![Page 3: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/3.jpg)
3
Plan
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
![Page 4: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/4.jpg)
4
Plan
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
![Page 5: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/5.jpg)
5
CTE SyntaxCTE name
CTE Body
CTE Usage
● Similar to DERIVED tables● “Query-local VIEWs”
with engineers as ( select * from employees where dept='Engineering')select * from engineers where ...
WITH
![Page 6: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/6.jpg)
6
select *from( select * from employees where dept='Engineering') as engineerswhere...
with engineers as ( select * from employees where dept='Engineering')select * from engineers where ...
CTEs are like derived tables
● Similar to DERIVED tables● “Query-local VIEWs”
![Page 7: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/7.jpg)
7
Use case #1: CTEs refer to CTEs
● More readable than nested FROM(SELECT …)
with engineers as ( select * from employees where dept in ('Development','Support')),eu_engineers as ( select * from engineers where country IN ('NL',...))select ...from eu_engineers;
![Page 8: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/8.jpg)
8
with engineers as ( select * from employees where dept in ('Development','Support')),select * from engineers E1 where not exists (select 1 from engineers E2 where E2.country=E1.country and E2.name <> E1.name);
Use case #2: Multiple use of CTE● Anti-self-join
![Page 9: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/9.jpg)
9
select * from sales_product_year CUR, sales_product_year PREV,where CUR.product=PREV.product and CUR.year=PREV.year + 1 and CUR.total_amt > PREV.total_amt
with sales_product_year as ( select product, year(ship_date) as year, sum(price) as total_amt from item_sales group by product, year)
Use case #2: example 2● Year-over-year comparisons
![Page 10: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/10.jpg)
10
select * from sales_product_year S1where total_amt > (select 0.1*sum(total_amt) from sales_product_year S2 where S2.year=S1.year)
with sales_product_year as ( select product, year(ship_date) as year, sum(price) as total_amt from item_sales group by product, year)
Use case #2: example 3● Compare individuals against their group
![Page 11: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/11.jpg)
11
● Non-recursive CTES are “Query-local VIEWs”
● One CTE can refer to another
•Better than nested FROM (SELECT …)
● Can refer to a CTE from multiple places
•Better than copy-pasting FROM(SELECT …)
● CTE adoption
•TPC-H (1999) - no CTEs
•TPC-DS (2011) - 38 of 100 queries use CTEs.
Conclusions so far
![Page 12: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/12.jpg)
12
Plan
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
![Page 13: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/13.jpg)
13
Base algorithm: materialize in a temp.table
● Always works● Often not optimal
with engineers as ( select * from employees where dept='Engineering' or dept='Support')select ...from engineers, other_table, ...
![Page 14: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/14.jpg)
14
Optimization #1: CTE Mergingwith engineers as ( select * from employees where dept='Development')select ...from engineers E, support_cases SCwhere E.name=SC.assignee and SC.created='2016-09-30' and E.location='Amsterdam'
select ...from employees E, support_cases SCwhere E.dept='Development' and E.name=SC.assignee and SC.created='2016-09-30' and E.location=’Amsterdam’
● for each support case•lookup employee by name
![Page 15: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/15.jpg)
15
Optimization #1: CTE Merging (2)
● Requirement
•CTE is a JOIN : no GROUP BY, DISTINCT, etc
● Output
•CTE is merged into parent’s join
•Optimizer can pick the best query plan
● This is the same as ALGORITHM=MERGE for VIEWs
![Page 16: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/16.jpg)
16
with sales_per_year as ( select year(order.date) as year sum(order.amount) as sales from order group by year)select * from sales_per_year where year in ('2015','2016')
with sales_per_year as ( select year(order.date) as year sum(order.amount) as sales from order where year in ('2015','2016') group by year)select * from sales_per_year
Optimization #2: condition pushdown
![Page 17: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/17.jpg)
17
Optimization #2: condition pushdown (2)Summary
● Used when merging is not possible
•CTE has a GROUP BY
● Makes temp. table smaller
● Allows to filter out whole groups
● Besides CTEs, works for derived tables and VIEWs
● MariaDB 10.2, Galina’s GSOC 2016 project:
“Pushing conditions into non-mergeable views and derived tables in
MariaDB”
![Page 18: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/18.jpg)
18
with product_sales as ( select product_name, year(sale_date), count(*) as count from product_sales group by product, year)select *from product_sales P1, product_sales P2where P1.year = 2010 AND P2.year = 2011 AND ...
Optimization #3: CTE reuse
● Idea•Fill the CTE once•Then use multiple times
● Doesn’t work together with condition pushdown or merging
![Page 19: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/19.jpg)
19
CTE Optimizations summary
● Merge and condition pushdown are most important
•MariaDB supports them, like MS SQL.
● PostgreSQL’s approach is *weird*
•“CTEs are optimization barriers”
● MySQL’s labs tree: “try merging, otherwise reuse”
CTE Merge Condition pushdown CTE reuse
MariaDB 10.2 ✔ ✔ ✘
MS SQL Server ✔ ✔ ✘
PostgreSQL ✘ ✘ ✔
MySQL 8.0.0-labs-optimizer ✔ ✘ ✔*
![Page 20: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/20.jpg)
20
Recursive CTEs
![Page 21: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/21.jpg)
21
Plan
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
![Page 22: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/22.jpg)
22
Recursive CTEs
● First attempt: Oracle’s CONNECT BY syntax (80’s)● Superseded by Recursive CTEs
•SQL:1999, implementations in 2007-2009
● SQL is poor at “recursive” data structures/algorithms
wheel
boltcapnut
tire valve
rimtirespokes- Trees - Graphs
Chicago
Nashville Atlanta
Orlando
![Page 23: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/23.jpg)
23
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
Plan
![Page 24: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/24.jpg)
24
Recursive CTE syntax
Recursive part
Anchor partRecursive use of CTE
“recursive”
with recursive ancestors as ( select * from folks where name = 'Alex' union [all] select f.* from folks as f, ancestors AS a where f.id = a.father or f.id = a.mother)select * from ancestors;
![Page 25: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/25.jpg)
25
Sister AmyAlex
Mom Dad
Grandpa Bill
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL || 98 | Sister Amy | 20 | 30 |+------+--------------+--------+--------+
Recursive CTE computation
![Page 26: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/26.jpg)
26
with recursive ancestors as ( select * from folks where name = 'Alex' union select f.* from folks as f, ancestors AS a where f.id = a.father or f.id = a.mother)select * from ancestors;
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 |+------+--------------+--------+--------+
Computation Result table
Step #1:execution of the anchor part
![Page 27: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/27.jpg)
27
with recursive ancestors as ( select * from folks where name = 'Alex' union select f.* from folks as f, ancestors AS a where f.id = a.father or f.id = a.mother)select * from ancestors;
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 |+------+--------------+--------+--------+
Computation Result table
+------+--------------+--------+--------+| id | name | father | moher |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL || 98 | Sister Amy | 20 | 30 |+------+--------------+--------+--------+
Step #2:execution of the recursive part
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL || 98 | Sister Amy | 20 | 30 |+------+--------------+--------+--------+
![Page 28: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/28.jpg)
28
with recursive ancestors as ( select * from folks where name = 'Alex' union select f.* from folks as f, ancestors AS a where f.id = a.father or f.id = a.mother)select * from ancestors;
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL |+------+--------------+--------+--------+
Computation Result table
Step #2:execution of the recursive part
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL || 98 | Sister Amy | 20 | 30 |+------+--------------+--------+--------+
![Page 29: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/29.jpg)
29
with recursive ancestors as ( select * from folks where name = 'Alex' union select f.* from folks as f, ancestors AS a where f.id = a.father or f.id = a.mother)select * from ancestors;
Computation Result table
Step #3:execution of the recursive part
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL |+------+--------------+--------+--------+
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL || 98 | Sister Amy | 20 | 30 |+------+--------------+--------+--------+
![Page 30: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/30.jpg)
30
with recursive ancestors as ( select * from folks where name = 'Alex' union select f.* from folks as f, ancestors AS a where f.id = a.father or f.id = a.mother)select * from ancestors;
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL |+------+--------------+--------+--------+
Computation Result table
Step #3:execution of the recursive part
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL || 98 | Sister Amy | 20 | 30 |+------+--------------+--------+--------+
![Page 31: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/31.jpg)
31
with recursive ancestors as ( select * from folks where name = 'Alex' union select f.* from folks as f, ancestors AS a where f.id = a.father or f.id = a.mother)select * from ancestors;
Computation Result table
Step #4:execution of the recursive part
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL || 98 | Sister Amy | 20 | 30 |+------+--------------+--------+--------+
+------+--------------+--------+--------+| id | name | father | mother |+------+--------------+--------+--------+| 100 | Alex | 20 | 30 || 20 | Dad | 10 | NULL || 30 | Mom | NULL | NULL || 10 | Grandpa Bill | NULL | NULL |+------+--------------+--------+--------+
No results!
![Page 32: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/32.jpg)
32
Summary so far
1. Compute anchor_data
2. Compute recursive_part to
get the new data
3. if (new data is non-empty)
goto 2;
with recursive R as ( select anchor_data union [all] select recursive_part from R, …)select …
![Page 33: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/33.jpg)
33
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
Plan
![Page 34: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/34.jpg)
34
Transitive closure
bus_routes+------------+------------+ | origin | dst | +------------+------------+| New York | Boston || Boston | New York || New York | Washington || Washington | Boston || Washington | Raleigh |+------------+------------+
New York
Boston Washington
Raleigh
![Page 35: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/35.jpg)
35
with recursive bus_dst as ( select origin as dst from bus_routes where origin='New York' union select bus_routes.dst from bus_routes, bus_dst where bus_dst.dst= bus_routes.origin)select * from bus_dst
● Start from New York
New York
Boston Washington
Raleigh
Transitive closure
![Page 36: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/36.jpg)
36
with recursive bus_dst as ( select origin as dst from bus_routes where origin='New York' union select bus_routes.dst from bus_routes, bus_dst where bus_dst.dst= bus_routes.origin)select * from bus_dst
● Step #1: add Boston and Washington
Transitive closure +------------+ | dst | +------------+| New York |+------------+
New York
Boston Washington
Raleigh
![Page 37: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/37.jpg)
37
with recursive bus_dst as ( select origin as dst from bus_routes where origin='New York' union select bus_routes.dst from bus_routes, bus_dst where bus_dst.dst= bus_routes.origin)select * from bus_dst
● Step #1: add Boston and Washington
New York
Boston Washington
Raleigh
Transitive closure +------------+ | dst | +------------+| New York || Boston || Washington |+------------+
![Page 38: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/38.jpg)
38
with recursive bus_dst as ( select origin as dst from bus_routes where origin='New York' union select bus_routes.dst from bus_routes, bus_dst where bus_dst.dst= bus_routes.origin)select * from bus_dst
● Step#2:•Add Raleigh•UNION excludes nodes that are already present.
New York
Boston Washington
Raleigh
Transitive closure +------------+ | dst | +------------+| New York || Boston || Washington || Raleigh |+------------+
![Page 39: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/39.jpg)
39
UNION and MySQL-8.0-labs● UNION is useful for “transitive closure” queries
● Supported in all databases
● MySQL 8.0.0-labs
ERROR 42000: This version of MySQL doesn't yet support
'UNION DISTINCT in recursive Common Table Expression (use
UNION ALL)'
•No idea why they have this limitation.
![Page 40: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/40.jpg)
40
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
Plan
![Page 41: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/41.jpg)
41
Computing “Paths”
bus_routes+------------+------------+ | origin | dst | +------------+------------+| New York | Boston || Boston | New York || New York | Washington || Washington | Boston || Washington | Raleigh |+------------+------------+
New York
Boston Washington
Raleigh
● Want paths like “New York -> Washington -> Raleigh”
![Page 42: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/42.jpg)
42
with recursive paths (cur_path, cur_dest) as ( select origin, origin from bus_routes where origin='New York' union select concat(paths.cur_path, ',', bus_routes.dest), bus_routes.dest from paths, bus_routes where paths.cur_dest= bus_routes.origin and locate(bus_routes.dest, paths.cur_path)=0)select * from paths
New York
Boston Washington
Raleigh
Computing “Paths”
![Page 43: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/43.jpg)
43
New York
Boston Washington
Raleigh
select concat(paths.cur_path, ',', bus_routes.dest), bus_routes.dest from paths, bus_routes where paths.cur_dest= bus_routes.origin and locate(bus_routes.dest, paths.cur_path)=0
+-----------------------------+------------+| cur_path | cur_dest |+-----------------------------+------------+| New York | New York || New York,Boston | Boston || New York,Washington | Washington || New York,Washington,Boston | Boston || New York,Washington,Raleigh | Raleigh |+-----------------------------+------------+
Computing “Paths”
![Page 44: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/44.jpg)
44
Recursion stopping summary● Tree or Directed Acyclic Graph walking
•Execution is guaranteed to stop
● Computing transitive closure
•Use UNION
● Computing “Paths” over graph with loops
•Put condition into WHERE to stop loops/growth
● Safety measure: @@max_recursive_iterations
•Like in SQL Server
•MySQL-8.0: @@max_execution_time ?
![Page 45: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/45.jpg)
45
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
Plan
![Page 46: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/46.jpg)
46
[Non-]linear recursion
with recursive R as ( select anchor_data
union [all]
select recursive_part from R, …)select …
The SQL standard requires that
recursion is linear:
● recursive_part must refer to R only
once
•No self-joins
•No subqueries
● Not from inner side of an outer join
● …
![Page 47: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/47.jpg)
47
Linearity of SELECT statements● Recursive CTE is a “function”: R(x)=a
R
● Linear “function”: R(x+y)=a+b
•added rows in source -> added rows in the result
x a
+y+b
![Page 48: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/48.jpg)
48
Linear recursion
with recursive R as ( select anchor_data
union [all]
select recursive_part from R, …)select …
A
![Page 49: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/49.jpg)
49
with recursive R as ( select anchor_data
union [all]
select recursive_part from R, …)select …
Linear recursion
● New data is generated by “wave-front” elements
● Contents of R are always growing
![Page 50: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/50.jpg)
50
Non-linear recursion● Next version of R is computed from the
entire R (not just “wavefront”)
● R may grow, shrink, or change
● It’s possible to screw things up
•R may flip-flop
•R may be non-deterministic
● Still, MariaDB supports it:
set @@standard_compliant_cte=0
•Be sure to know what you’re doing.
![Page 51: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/51.jpg)
51
● Non-recursive CTEs•Use cases•Optimizations
● Recursive CTEs•Basics•Transitive closure•Paths•(Non-)linear recursion•Mutual recursion
Plan
![Page 52: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/52.jpg)
52
● Multiple CTEs refer to each other
● Useful for “bipartite” graphs
•emp->dept->emp
● MariaDB supports it
● No other database does
with recursive C1 as ( select … from anchor_table union select … from C2), C2 as ( select … from C1)select ...
Mutual recursion
![Page 53: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/53.jpg)
53
Conclusions● MariaDB 10.2 has Common Table Expressions
● Both Recursive and Non-recursive are supported
● Non-recursive
•“Query-local VIEWs”
•Competitive set of query optimizations
● Recursive
•Useful for tree/graph-walking queries
•Mutual and non-linear recursion is supported.
![Page 54: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/54.jpg)
54
Rate My Session!
![Page 55: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/55.jpg)
55
● Bill of materials
Tree-walking
Which part of bicycle is nut situated in?
![Page 56: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/56.jpg)
56
+------+------------+-------+---------+| id | name | count | part_of |+------+------------+-------+---------+| 10 | nut | 1 | 7 |+------+------------+-------+---------+
Bill of materials
wheel
boltcapnut
tire valve
rimtirespokes
with recursive nut as ( select * from bicycle_parts where name = 'nut' union select b.* from bicycle_parts as b, nut AS n where b.id = n.part_of )select * from nutwhere part_of = NULL;
![Page 57: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/57.jpg)
57
+------+------------+-------+---------+| id | name | count | part_of |+------+------------+-------+---------+| 10 | nut | 1 | 7 || 7 | tire valve | 2 | 4 |+------+------------+-------+---------+
Bill of materials
wheel
boltcapnut
tire valve
rimtirespokes
with recursive nut as ( select * from bicycle_parts where name = 'nut' union select b.* from bicycle_parts as b, nut AS n where b.id = n.part_of )select * from nutwhere part_of = NULL;
![Page 58: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/58.jpg)
58
+------+------------+-------+---------+| id | name | count | part_of |+------+------------+-------+---------+| 10 | nut | 1 | 7 || 7 | tire valve | 2 | 4 || 4 | tire | 2 | 1 |+------+------------+-------+---------+
Bill of materials
wheel
boltcapnut
tire valve
rimtirespokes
with recursive nut as ( select * from bicycle_parts where name = 'nut' union select b.* from bicycle_parts as b, nut AS n where b.id = n.part_of )select * from nutwhere part_of = NULL;
![Page 59: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/59.jpg)
59
+------+------------+-------+---------+| id | name | count | part_of |+------+------------+-------+---------+| 10 | nut | 1 | 7 || 7 | tire valve | 2 | 4 || 4 | tire | 2 | 1 || 1 | wheel | 2 | NULL |+------+------------+-------+---------+
wheel
boltcapnut
tire valve
rimtirespokes
Bill of materials
with recursive nut as ( select * from bicycle_parts where name = 'nut' union select b.* from bicycle_parts as b, nut AS n where b.id = n.part_of )select * from nutwhere part_of = NULL;
![Page 60: Galina Shalygina Sergei Petrunia - percona.com · Makes temp. table smaller Allows to filter out whole groups Besides CTEs, works for derived tables and VIEWs MariaDB 10.2, Galina’s](https://reader033.fdocuments.in/reader033/viewer/2022043011/5fa5e994ab994a3450372542/html5/thumbnails/60.jpg)
60
with recursive nut as ( select * from bicycle_parts where name = 'nut' union select b.* from bicycle_parts as b, nut AS n where b.id = n.part_of )select * from nutwhere part_of = NULL;
It is a part of wheel!
Bill of materials
+------+------------+-------+---------+| id | name | count | part_of |+------+------------+-------+---------+| 1 | wheel | 2 | NULL |+------+------------+-------+---------+